Explore Docker registries and manipulate Docker images!

Overview

CircleCI Go Report Card Coverage Status

L/S tags

Utility and API to manipulate (analyze, synchronize and aggregate) images across different Docker registries.

Example invocation

$ lstags alpine~/^3\\./
<STATE>      <DIGEST>                                   <(local) ID>    <Created At>          <TAG>
ABSENT       sha256:9363d03ef12c8c25a2def8551e609f146   n/a             2017-09-13T16:32:00   alpine:3.1
CHANGED      sha256:9866438860a1b28cd9f0c944e42d3f6cd   39be345c901f    2017-09-13T16:32:05   alpine:3.2
ABSENT       sha256:ae4d16d132e3c93dd09aec45e4c13e9d7   n/a             2017-09-13T16:32:10   alpine:3.3
CHANGED      sha256:0d82f2f4b464452aac758c77debfff138   f64255f97787    2017-09-13T16:32:15   alpine:3.4
PRESENT      sha256:129a7f8c0fae8c3251a8df9370577d9d6   074d602a59d7    2017-09-13T16:32:20   alpine:3.5
PRESENT      sha256:f006ecbb824d87947d0b51ab8488634bf   76da55c8019d    2017-09-13T16:32:26   alpine:3.6

NB! You can specify many images to operate on, e.g: lstags nginx~/^1\\.13/ mesosphere/chronos alpine~/^3\\./

Why would someone use this?

You could use lstags, if you ...

  • ... aggregate images from different external registries into your own registry for speed and locality reasons.
  • ... compare images present locally with the registry ones (e.g.: know if image tagged "latest" was re-pushed).
  • ... continuously pull Docker images from some public or private registry to speed-up Docker run on your system.

How?

... pull Ubuntu 14.04 & 16.04, all the Alpine images and Debian "stretch" to have the latest software to play with:

lstags --pull ubuntu~/^1[46]\\.04$/ alpine debian~/stretch/

... pull and re-push CoreOS-related images from quay.io to your own registry (in case these hipsters will break everything):

lstags -P /quay -r registry.company.io quay.io/coreos/hyperkube quay.io/coreos/flannel

NB! In case you use private registry with authentication, make sure your Docker client knows how to authenticate against it! lstags will reuse credentials saved by Docker client in its config.json file, one usually found at ~/.docker/config.json

Possible image states

lstags distinguishes five states of Docker image:

  • ABSENT - present in registry, but absent locally
  • PRESENT - present in registry, present locally, with local and remote digests being equal
  • CHANGED - present in registry, present locally, but with different local and remote digests
  • LOCAL_ONLY - present locally, absent in registry
  • NOT_FOUND - absent in registry, absent locally, probably does not exist at all

Authentication

You can either:

  • rely on lstags discovering credentials "automagically" 🎩
  • load credentials from any Docker JSON config file specified

Assume tags

Sometimes registry may contain tags not exposed to any kind of search though still existing. lstags is unable to discover these tags, but if you need to pull or push them, you may "assume" they exist and make lstags blindly try to pull these tags from the registry. To inject assumed tags into the registry query you need to extend repository specification with a = followed by a comma-separated list of tags you want to assume.

e.g. we assume tags v1.6.1 and v1.7.0 exist like this: lstags quay.io/calico/cni=v1.6.1,v1.7.0

Repository specification

Full repository specification looks like this:

[REGISTRY[:PORT]/]REPOSITORY[~/FILTER_REGEXP/][=TAG1,TAG2,TAGn]

You may provide infinite number of repository specifications to lstags

Push prefix

When you [re]push images to your "push" registry, you can control the destination repository path prefix:

  • by default, repository path prefix will be auto-generated from the source registry hostname, e.g.:
    • alpine ▶️ /registry/hub/docker/com/
    • localhost:5000/nginx ▶️ /localhost/
    • registry.company.com/hype/kubernetes ▶️ /registry/company/com/
  • passing --push-prefix=/ will push images "as is", with no additional repository path prefix
  • passing --push-prefix=/my/prefix/ will push images appending /my/prefix/ to the repository path
  • specifying /my/prefix without trailing slash is OK, as long as path would still be formatted correctly by API
  • passing --push-prefix="" would trigger "default" behavior with prefix being auto-generated

To fail or not to fail?

By default application exits after encountering any errors. To make it more tolerant to subsequent failures, you may use CLI option -N, --do-not-fail or set environment variable DO_NOT_FAIL=true before running application. HINT: Option -d, --daemon-mode always implies activation of --do-not-fail.

YAML

💡 You can load repositories from the YAML file just like you do it from the command line arguments:

lstags -f file.yaml

A valid YAML file looks like this (mandatory lstags root key is here to be able to use "shared" YAMLs):

lstags:
  repositories:
    - busybox
    - nginx:stable
    - mesosphere/marathon-lb~/^v1/
    - quay.io/coreos/awscli=master,latest,edge
    - gcr.io/google-containers/hyperkube~/^v1\.(9|10)\./

NB! lstags can load repositories from YAML or from CLI args, but not from both at the same time!

Install: Binaries

https://github.com/ivanilves/lstags/releases

Install: Wrapper

git clone [email protected]:ivanilves/lstags.git
cd lstags
sudo make wrapper
lstags -h

A special wrapper script will be installed to manage lstags invocation and updates. 😎

Install: From source

git clone [email protected]:ivanilves/lstags.git
cd lstags
dep ensure
go build
./lstags -h

NB! I assume you have current versions of Go & dep installed and also have set up GOPATH correctly.

Using it with Docker

docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock ivanilves/lstags
Usage:
  lstags [OPTIONS] REPO1 REPO2 REPOn...

Application Options:
  -j, --docker-json=          JSON file with credentials (default:
                              ~/.docker/config.json) [$DOCKER_JSON]
  -p, --pull                  Pull Docker images matched by filter (will use
                              local Docker deamon) [$PULL]
  -P, --push                  Push Docker images matched by filter to some
                              registry (See 'push-registry') [$PUSH]
  -r, --push-registry=        [Re]Push pulled images to a specified remote
                              registry [$PUSH_REGISTRY]

--- OUTPUT WAS CUT HERE TO SAVE SPACE ---

Analyze an image

docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock ivanilves/lstags alpine~/^3\\./
ANALYZE alpine
FETCHED alpine
-
<STATE>   <DIGEST>                                  <(local) ID>    <Created At>            <TAG>
CHANGED   sha256:b40e202395eaec699f2d0c5e01e6d6cb8  76da55c8019d    2017-10-25T23:19:51Z    alpine:3.6
ABSENT    sha256:d95da16498d5d6fb4b907cbe013f95032  n/a             2017-10-25T23:20:18Z    alpine:3.1
ABSENT    sha256:cb275b62f789b211114f28b391fca3cc2  n/a             2017-10-25T23:20:32Z    alpine:3.2
ABSENT    sha256:27af7da847283a947c008592f2b2cd6d2  n/a             2017-10-25T23:20:45Z    alpine:3.3
CHANGED   sha256:246bbbaa81b28837b64cb9dfc574de958  1a19a71e5d38    2017-10-25T23:20:59Z    alpine:3.4
CHANGED   sha256:aa96c8dc3815c44d4aceaf1ee7903ce58  37c7be7a096b    2017-10-25T23:21:13Z    alpine:3.5
-

Development

You are very welcome to open pull requests to this repository! 😉

⚠️ CI build will fail, if your commit messages are not semantic!

To maximize our collaboration efficiency we would humbly ask you to follow these recommendations:

  • Please add reasonable description (what?/why?/etc) to your pull request
  • Your code should pass CI (CircleCI) and a [pretty liberal] code review 🔍
  • If code adds or changes some logic, it should be covered by a unit test :neckbeard:
  • Please, put meaningful and semantic messages on your commits 🙏

NB! Not a requirement, but a GIF included in PR description would make our world a happier place!

'NORELEASE' branches and commits

We have automatic release system. Every PR merge will create a new application release with a changelog generated from PR branch commits. For the most cases it is OK. However, if you work with things that do not need to be released (e.g. non user-facing changes), you have following options:

  • If you don't want to create release from your PR, make it from branch containing "NORELEASE" keyword in its name.
  • If you want to prevent single commit from appearing in a changelog, please start commit message with "NORELEASE".

⚠️ We don't build RPMs/DEBs/etc, as we see no need for it. We ship lstags as a single binary or as a Docker container.

API

You may use lstags either as a standalone CLI or as a Golang package inside your own application.

Set up and build PoC application with our v1 API:

make poc-app APP_PATH=../lstags-api
cd ../lstags-api
go build
# run "./lstags-api" binary to see PoC in action (examine main.go first to ensure no "rm -rf /" is there)
  • This installs all necessary dependencies and sets up PoC application at the path ../lstags-api/
  • We assume you already have recent Golang version installed on your system https://golang.org/dl/

GoDoc

NB! Far more complete API usage example could be found in main.go 😉

Issues
  • Bad response status: 401 Unauthorized

    Bad response status: 401 Unauthorized

    Hello, When I do a docker pull from our repos using docker pull everything works fine. When using lstags docker the pull doesnt work. Please help.

    [email protected] config]# docker pull gcr.io/anyvision-training/templates Using default tag: latest Trying to pull repository gcr.io/anyvision-training/templates ... latest: Pulling from gcr.io/anyvision-training/templates Digest: sha256:7fc087838e4da276debc2b20ba57c865d25192a39597b7dcc572c9fd396f97f9 Status: Image is up to date for gcr.io/anyvision-training/templates:latest

    [email protected] config]# docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock ivanilves/lstags -p gcr.io/anyvision-training/templates INFO[0000] ANALYZE gcr.io/anyvision-training/templates
    Bad response status: 401 Unauthorized >> https://gcr.io/v2/anyvision-training/templates/tags/list

    [email protected] lstags]# docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock ivanilves/lstags -j /root/.gcp/docker-registry-ro-training.json -p gcr.io/anyvision-training/templates open /root/.gcp/docker-registry-ro-training.json: no such file or directory

    [[email protected] lstags]# ls -ltr /root/.gcp/docker-registry-ro-training.json -rw-r--r-- 1 root root 2327 Oct 2 06:13 /root/.gcp/docker-registry-ro-training.json

    The xml file is good, it is used when using the plain docker command.

    Thanks,

    Meir

    bug critical 
    opened by meirhazonAnyVision 20
  • goroutine fatal when pushing big amount of images

    goroutine fatal when pushing big amount of images

    in similar context as https://github.com/ivanilves/lstags/issues/153, but different scenario, got goroutine fatal errors when pushing ~400 images to registry.

    the registry:5000 is insecure registry confured via env: INSECURE_REGISTRY_EX=registry:5000

    worked fine with 1 image with same configuration.

    $ lstags '--docker-json=/root/.docker/config.json' '--yaml-config=lstags.yml' --push '--push-prefix=/' '--push-registry=registry:5000'
    INFO[0000] BATCH 1 of 13
    fatal error: concurrent map writes
    
    goroutine 1346 [running]:
    runtime.throw(0x7f16c1, 0x15)
        /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/panic.go:619 +0x81 fp=0xc4203d5d60 sp=0xc4203d5d40 pc=0x429951
    runtime.mapassign_faststr(0x77a200, 0xc42008b650, 0xc42030a1b0, 0x2f, 0xc420f91d40)
        /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/hashmap_fast.go:703 +0x3e9 fp=0xc4203d5dd0 sp=0xc4203d5d60 pc=0x40bb39
    github.com/ivanilves/lstags/api/v1.(*API).CollectTags.func1(0xc4202c6320, 0xc42008b650, 0xc420d9a9c0, 0xc420e339e0)
        /home/travis/gopath/src/github.com/ivanilves/lstags/api/v1/v1.go:143 +0x609 fp=0xc4203d5fc0 sp=0xc4203d5dd0 pc=0x6f0709
    runtime.goexit()
        /home/travis/.gimme/versions/go1.10.linux.amd64/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4203d5fc8 sp=0xc4203d5fc0 pc=0x455791
    created by github.com/ivanilves/lstags/api/v1.(*API).CollectTags
        /home/travis/gopath/src/github.com/ivanilves/lstags/api/v1/v1.go:120 +0x24f
    

    More complete output: push_b_1a.log

    bug critical handicap 
    opened by glensc 17
  • Problem with great amount of entries.

    Problem with great amount of entries.

    We have an other problem that happens with 1.2.7 till 1.2.11. We mirror a lot of images from Docker Hub and the yaml file with the images/tags is large. The problem was, that lstags stopped working after a specific amount of images/tags. We enabled trace logging but it does not show anything useful. The "batches" all work fine and the "state" is shown. But during "[PULL/PUSH] PUSHING" there is a moment when the time to the next "[PULL/PUSH] PUSHING" increases heavily from image to image until some image where lstags waits forever. We looked at the process and it is not crashed but it does not run anymore.

    We found a workaround by splitting the yaml file from Docker Hub but it seems that there is a problem when the amount of images, tags or size reaches some limit after that lstags stops working.

    We also found out that the lstags process does not terminate when run from gitlab-ci and terminated through pipeline timeout. We found hanging lstags processes on the runner. But sending them a regular sigterm works to to terminate them.

    help wanted 
    opened by nudgegoonies 14
  • handicapped pull

    handicapped pull

    there's no progress update what is lstags doing, even with verbosity set to max.

    and there's bug that when one of the images specified in yaml/commandline is not pullable. the process never finishes. and i have no indication has it finished and idling due the bug, or actually doing the hard work of pulling.

    # ./lstags --version
    VERSION: v1.0.61
    # ./lstags --pull alpine:2  -vvv
    DEBU[0000] [New()] API config: {DockerJSONConfigFile:~/.docker/config.json ConcurrentRequests:32 TraceRequests:false RetryRequests:2 RetryDelay:30s InsecureRegistryEx: VerboseLogging:true}
    INFO[0000] BATCH 1 of 1
    DEBU[0000] [CollectTags()] references: [alpine:2]
    DEBU[0000] [CollectTags()] repository: &{ref:alpine:2 registry:registry.hub.docker.com fullRepo:registry.hub.docker.com/alpine repoTags:[2] filterRE:<nil> isSecure:true isSingle:true}
    INFO[0000] ANALYZE alpine:2
    DEBU[0001] [func1():alpine:2] remote tags: map[]
    DEBU[0001] [func1():alpine:2] local tags: map[]
    DEBU[0001] [func1():alpine:2] joined tags: map[2:0xc4200b4900]
    INFO[0001] FETCHED alpine:2
    DEBU[0001] [CollectTags()] tags: map[alpine:2:[0xc4200b4900]]
    -
    <STATE>      <DIGEST>                                      <(local) ID>    <Created At>              <IMAGE>:<TAG>
    ASSUMED      n/a                                           n/a             1970-01-01T03:00:00       alpine:2
    -
    DEBU[0001] [PullTags()] collection: &{refs:[alpine:2] repos:map[alpine:2:0xc4200b4ae0] tags:map[alpine:2:[0xc4200b4900]]} (1 repos / 1 tags)
    DEBU[0001] [PullTags()] repository: &{ref:alpine:2 registry:registry.hub.docker.com fullRepo:registry.hub.docker.com/alpine repoTags:[2] filterRE:<nil> isSecure:true isSingle:true}
    DEBU[0001] [PullTags()] tag: &{name:2 digest:n/a imageID:n/a state:ASSUMED created:0 containerID:}
    INFO[0001] PULLING alpine:2
    INFO[0001] PULLING alpine:2
    ^C
    
    # docker pull alpine:2
    Error response from daemon: manifest for alpine:2 not found
    
    critical handicap 
    opened by glensc 10
  • Unable to clear

    Unable to clear "push-prefix".

    i want to copy images from one registry to other registry without altering their prefix. i seem not to succeed on that.

    # lstags.yaml
    lstags:
      repositories:
        - gitlab.example.net:4567/ed/php~/^(5\.[36]|7\.[12])-/
    

    with --push-prefix=/ it creates url with double slash and gets auth failure.

    $ ./lstags --docker-json=$HOME/.docker/config.json --yaml-config=lstags.yaml  --push-registry=registry.gitlab.test.example.net --push-prefix=/
    INFO[0000] ANALYZE gitlab.example.net:4567/ed/php~/^(5\.[36]|7\.[12])-/
    INFO[0001] FETCHED gitlab.example.net:4567/ed/php~/^(5\.[36]|7\.[12])-/
    -
    <STATE>      <DIGEST>                                      <(local) ID>    <Created At>              <IMAGE>:<TAG>
    PRESENT      sha256:4c90cc845b6774236878e81a86099270d      518f80c5b6eb    2018-04-14T04:11:40       gitlab.example.net:4567/ed/php:5.3-cli
    PRESENT      sha256:f0f4fd471768435a9542bf70c2b015e51      2a6486ae26ab    2018-04-14T04:11:40       gitlab.example.net:4567/ed/php:5.6-cli
    PRESENT      sha256:cf95b3b716aea34d7121ac9b499b1d4ac      4770d7ee458d    2018-04-14T04:11:42       gitlab.example.net:4567/ed/php:7.1-cli
    PRESENT      sha256:a416582b3bd748ce8ecfcf289afbc9332      728c2d641db5    2018-04-14T04:13:01       gitlab.example.net:4567/ed/php:7.2-cli
    PRESENT      sha256:d97cd6f5603bfda42c00e6eb7b25cce6d      a71b22877f5c    2018-04-14T04:13:57       gitlab.example.net:4567/ed/php:5.3-fpm
    PRESENT      sha256:c523a8822c4207aa45c30d1f21bbb8738      a62dd9c0f1a8    2018-04-14T04:13:59       gitlab.example.net:4567/ed/php:5.6-fpm
    PRESENT      sha256:6e42a119faab447e8ca15e985e4c3f62a      036e59ff29da    2018-04-14T04:13:59       gitlab.example.net:4567/ed/php:7.1-fpm
    PRESENT      sha256:8613ce851ab4a2c25b3c6601e5c4f5ac6      575bb1c83b10    2018-04-14T04:14:41       gitlab.example.net:4567/ed/php:7.2-fpm
    PRESENT      sha256:ea4a8fb5f10eaffe016fb57b38016a9df      1708314e97dd    2018-04-14T04:15:48       gitlab.example.net:4567/ed/php:5.3-apache
    PRESENT      sha256:7a69db8b42b8f3c9e870fd63678d164e1      0ff1700cc718    2018-04-14T04:15:48       gitlab.example.net:4567/ed/php:5.6-apache
    PRESENT      sha256:c3c958966f6f11fca4ebe2123de1d59cd      b11ee7c8dc3e    2018-04-14T04:15:49       gitlab.example.net:4567/ed/php:7.1-apache
    PRESENT      sha256:59dd8d1a58f5946199113bf3714f2ae10      cdcffea8acad    2018-04-14T04:16:35       gitlab.example.net:4567/ed/php:7.2-apache
    -
    INFO[0001] [PULL/PUSH] ANALYZE gitlab.example.net:4567/ed/php~/^(5\.[36]|7\.[12])-/ => registry.gitlab.test.example.net//ed/php~/.*/
    Bad response status: 401 Unauthorized >> https://registry.gitlab.test.example.net/v2//ed/php/tags/list
    

    with --push-prefix= it seems to ignore argument, defaulting to some "dynamic prefix from pushed image" aka the default behavior:

    $ ./lstags --docker-json=$HOME/.docker/config.json --yaml-config=lstags.yaml  --push-registry=registry.gitlab.test.example.net --push-prefix=
    INFO[0000] ANALYZE gitlab.example.net:4567/ed/php~/^(5\.[36]|7\.[12])-/
    INFO[0001] FETCHED gitlab.example.net:4567/ed/php~/^(5\.[36]|7\.[12])-/
    -
    <STATE>      <DIGEST>                                      <(local) ID>    <Created At>              <IMAGE>:<TAG>
    PRESENT      sha256:4c90cc845b6774236878e81a86099270d      518f80c5b6eb    2018-04-14T04:11:40       gitlab.example.net:4567/ed/php:5.3-cli
    PRESENT      sha256:f0f4fd471768435a9542bf70c2b015e51      2a6486ae26ab    2018-04-14T04:11:40       gitlab.example.net:4567/ed/php:5.6-cli
    PRESENT      sha256:cf95b3b716aea34d7121ac9b499b1d4ac      4770d7ee458d    2018-04-14T04:11:42       gitlab.example.net:4567/ed/php:7.1-cli
    PRESENT      sha256:a416582b3bd748ce8ecfcf289afbc9332      728c2d641db5    2018-04-14T04:13:01       gitlab.example.net:4567/ed/php:7.2-cli
    PRESENT      sha256:d97cd6f5603bfda42c00e6eb7b25cce6d      a71b22877f5c    2018-04-14T04:13:57       gitlab.example.net:4567/ed/php:5.3-fpm
    PRESENT      sha256:c523a8822c4207aa45c30d1f21bbb8738      a62dd9c0f1a8    2018-04-14T04:13:59       gitlab.example.net:4567/ed/php:5.6-fpm
    PRESENT      sha256:6e42a119faab447e8ca15e985e4c3f62a      036e59ff29da    2018-04-14T04:13:59       gitlab.example.net:4567/ed/php:7.1-fpm
    PRESENT      sha256:8613ce851ab4a2c25b3c6601e5c4f5ac6      575bb1c83b10    2018-04-14T04:14:41       gitlab.example.net:4567/ed/php:7.2-fpm
    PRESENT      sha256:ea4a8fb5f10eaffe016fb57b38016a9df      1708314e97dd    2018-04-14T04:15:48       gitlab.example.net:4567/ed/php:5.3-apache
    PRESENT      sha256:7a69db8b42b8f3c9e870fd63678d164e1      0ff1700cc718    2018-04-14T04:15:48       gitlab.example.net:4567/ed/php:5.6-apache
    PRESENT      sha256:c3c958966f6f11fca4ebe2123de1d59cd      b11ee7c8dc3e    2018-04-14T04:15:49       gitlab.example.net:4567/ed/php:7.1-apache
    PRESENT      sha256:59dd8d1a58f5946199113bf3714f2ae10      cdcffea8acad    2018-04-14T04:16:35       gitlab.example.net:4567/ed/php:7.2-apache
    -
    INFO[0001] [PULL/PUSH] ANALYZE gitlab.example.net:4567/ed/php~/^(5\.[36]|7\.[12])-/ => registry.gitlab.test.example.net/gitlab/example.net/ed/php~/.*/
    Bad response status: 401 Unauthorized >> https://registry.gitlab.test.example.net/v2/gitlab/example.net/ed/php/tags/list
    
    bug handicap 
    opened by glensc 9
  • invalid image in spec skips any pulling

    invalid image in spec skips any pulling

    invalid, aka inexistent image. should not abort whole process. just do as much work as possible, report errors summary if needed.

    [email protected] ~/scm/docker# time ./lstags --pull alpine:2 alpine:3.3
    INFO[0000] BATCH 1 of 1
    INFO[0000] ANALYZE alpine:3.3
    INFO[0000] ANALYZE alpine:2
    INFO[0001] FETCHED alpine:2
    INFO[0002] FETCHED alpine:3.3
    -
    <STATE>      <DIGEST>                                      <(local) ID>    <Created At>              <IMAGE>:<TAG>
    ASSUMED      n/a                                           n/a             1970-01-01T03:00:00       alpine:2
    ABSENT       sha256:36b22b4ceb2128310ed800fdb28c52bf8      n/a             2018-01-09T23:12:20       alpine:3.3
    -
    INFO[0002] PULLING alpine:3.3
    INFO[0002] PULLING alpine:2
    Error response from daemon: manifest for alpine:2 not found
    
    real    0m3,681s
    user    0m0,137s
    sys     0m0,030s
    
    [email protected] ~/scm/docker# ./lstags --version
    VERSION: v1.0.63
    
    bug critical handicap 
    opened by glensc 7
  • index out of range error

    index out of range error

    Just trying to get things up and running–

    $ lstags --version
    VERSION: v30-d15b77b
    $ lstags alpine~/^3\\./
    panic: runtime error: index out of range
    
    goroutine 1 [running]:
    github.com/ivanilves/lstags/docker/config.Load(0x1360289, 0x15, 0x0, 0x0, 0x0)
    	/home/travis/gopath/src/github.com/ivanilves/lstags/docker/config/config.go:79 +0x4fe
    main.main()
    	/home/travis/gopath/src/github.com/ivanilves/lstags/main.go:98 +0x8d
    
    possible bug 
    opened by ryanking 7
  • Better world: First refactor of the lstags

    Better world: First refactor of the lstags

    We have reached the state when we need to:

    • stop adding new functionality for a while.
    • work more on automated testing, code quality and various fixes.
    • add performance, benefit more of concurrency Golang gives us [almost] for free.

    TODO

    • [x] Add black-box shell tests for "image pull" functionality.
    • [x] Fix lame ("double"?) help messages.
    • [x] Make registry and operations event more concurrent.
    • [x] Fix authenticated pull.

    GIF

    enhancement critical 
    opened by ivanilves 7
  • panic: value method ... bearer.Token.Method called using nil *Token pointer

    panic: value method ... bearer.Token.Method called using nil *Token pointer

    • I am using latest release 1.2.20 (built from sources on macOS 11.4 using go version go1.16.5 darwin/amd64)
    • However I saw the same happening with 1.2.18 on macOS and 1.2.20 downloaded from GitHub on Linux
    • I try to check the attached file but get the following (this does not happen all the times but on Linux it never succeeds):
    $ YAML_CONFIG=registry.gitlab.com.txt CONCURRENT_REQUESTS=16 WAIT_BETWEEN=0s RETRY_DELAY=1s RETRY_REQUESTS=2
    time="2021-07-09T15:23:37Z" level=info msg="BATCH 1 of 1"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/spotbugs=2,latest"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/flawfinder=2,latest"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/mobsf=latest"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/kubesec=2,latest"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/bandit=2,latest"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/phpcs-security-audit=2,latest"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/eslint=2,latest"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/brakeman=2,latest"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan=2,latest"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/semgrep=2,latest"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/security-code-scan=2,latest"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/sobelow=2,latest"
    time="2021-07-09T15:23:37Z" level=info msg="ANALYZE registry.gitlab.com/gitlab-org/security-products/analyzers/pmd-apex=2,latest"
    panic: value method github.com/ivanilves/lstags/api/v1/registry/client/auth/bearer.Token.Method called using nil *Token pointer
    
    goroutine 254 [running]:
    github.com/ivanilves/lstags/api/v1/registry/client/auth/bearer.(*Token).Method(0x0, 0xc000790bc0, 0x4170de)
    	<autogenerated>:1 +0x49
    github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).repoToken(0xc00035fc70, 0xc0000be0f0, 0x2f, 0xc000790cb8, 0x40bc76, 0xc00038a660, 0x60)
    	/go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:179 +0x47b
    github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).v1TagOptions(0xc00035fc70, 0xc0000be0f0, 0x2f, 0xc000142a94, 0x1, 0xc000142a94, 0x1, 0xc00038a600)
    	/go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:333 +0x5d
    github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).Tag(0xc00035fc70, 0xc0000be0f0, 0x2f, 0xc000142a94, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    	/go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:380 +0x130
    github.com/ivanilves/lstags/tag/remote.FetchTags.func1(0xc00035fc70, 0xc00024ff80, 0xc000142a94, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    	/go/src/github.com/ivanilves/lstags/tag/remote/remote.go:104 +0x8b
    created by github.com/ivanilves/lstags/tag/remote.FetchTags
    	/go/src/github.com/ivanilves/lstags/tag/remote/remote.go:98 +0x46f
    panic: value method github.com/ivanilves/lstags/api/v1/registry/client/auth/bearer.Token.Method called using nil *Token pointer
    
    goroutine 278 [running]:
    github.com/ivanilves/lstags/api/v1/registry/client/auth/bearer.(*Token).Method(0x0, 0xce5d20, 0x9d3da0)
    	<autogenerated>:1 +0x49
    github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).repoToken(0xc00035fc70, 0xc0000be0f0, 0x2f, 0xc000001680, 0xc00036cec0, 0x4051aa, 0xc00039c660)
    	/go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:179 +0x47b
    github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).tagDigest(0xc00035fc70, 0xc0000be0f0, 0x2f, 0xc000142a94, 0x1, 0xc000412180, 0xc000318aa8, 0xc00036cfb0, 0x6c46c2)
    	/go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:274 +0x5d
    github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).Tag.func1(0xc00035fc70, 0xc0000be0f0, 0x2f, 0xc000142a94, 0x1, 0xc00038a600, 0xc00038a660)
    	/go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:371 +0x70
    created by github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).Tag
    	/go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:370 +0xe0
    

    registry.gitlab.com.txt

    bug critical 
    opened by mfriedenhagen 5
  • Calls list API with page size of 1 causing rate limiting with quay.io

    Calls list API with page size of 1 causing rate limiting with quay.io

    Sorry for opening so many tickets. I'm just raising them as I find them and will try and contribute back to the project as soon as I can.

    It is taking me many attempts to pull repositories with a lot of tags. It seems to be because (at least with quay.io) the page size is set to 1, which means that if a repository has 100 tags it's got to do 100 API calls. Using a bigger page size for the calls, or not calling the list API when a specific tag is requested (see #222) could help solve this.

    For maximum pain try pulling this one :wink: https://quay.io/repository/jetstack/cert-manager-controller

    opened by cablespaghetti 5
  • Support paging when receiving tags

    Support paging when receiving tags

    The problem with quay.io seems to be the amount of tags. They only deliver the 50 first tags with the first request:

    $ CONCURRENT_REQUESTS=1 lstags -T -v 'quay.io/coreos/etcd:v3.3'
    DEBU[0000] [New()] API config: {DockerJSONConfigFile:~/.docker/config.json ConcurrentRequests:1 WaitBetween:0s TraceRequests:true RetryRequests:2 RetryDelay:30s InsecureRegistryEx: VerboseLogging:true} 
    INFO[0000] BATCH 1 of 1                                 
    DEBU[0000] [CollectTags()] references: [quay.io/coreos/etcd:v3.3] 
    DEBU[0000] [CollectTags()] repository: &{ref:quay.io/coreos/etcd:v3.3 registry:quay.io fullRepo:quay.io/coreos/etcd repoTags:[v3.3] filterRE:<nil> isSecure:true isSingle:true} 
    INFO[0000] ANALYZE quay.io/coreos/etcd:v3.3             
    DEBU[0000] Try to login with less permissions (repository:catalog:*) 
    d6951b3|@URL: https://quay.io/v2/coreos/etcd/tags/list
    d6951b3|@HEADER: Content-Type                             = [application/json]
    d6951b3|@HEADER: Content-Length                           = [589]
    d6951b3|@HEADER: Link                                     = [</v2/coreos/etcd/tags/list?next_page=gAAAAABcywP0Zuqt0GQh2CEVObDgUzCsvtYqiDOIfgC-JhU6i4O6u-CMA-_aLBthbWbL1KiFicqs-irXCgCuxBocP9vpfISiIfJc19qPsGZqvqT8LGwxSME%3D&n=50>; rel="next"]
    d6951b3|@HEADER: X-Frame-Options                          = [DENY]
    d6951b3|@HEADER: Strict-Transport-Security                = [max-age=63072000; preload]
    d6951b3|@HEADER: Server                                   = [nginx/1.14.2]
    d6951b3|@HEADER: Date                                     = [Thu, 02 May 2019 14:51:32 GMT]
    d6951b3|--- BODY BEGIN ---
    d6951b3|{"name":"coreos/etcd","tags":["3.1","3.2","latest","test","v0.4.6","v0.4.8","v0.5.0_alpha.0","v0.5.0_alpha.1","v0.5.0_alpha.2","v0.5.0_alpha.3","v0.5.0_alpha.4","v0.5.0_alpha.5","v2.0.0","v2.0.0_rc.1","v2.0.10","v2.0.11","v2.0.12","v2.0.13","v2.0.3","v2.0.4","v2.0.5","v2.0.6","v2.0.7","v2.0.8","v2.0.9","v2.1.0-alpha.0","v2.1.0-alpha.1","v2.1.0-rc.0","v2.1.1","v2.1.2","v2.1.3","v2.2.0","v2.2.0-alpha.0","v2.2.0-alpha.1","v2.2.0-rc.0","v2.2.1","v2.2.2","v2.2.3","v2.2.4","v2.2.5","v2.3.0","v2.3.0-alpha.0","v2.3.0-alpha.1","v2.3.1","v2.3.2","v2.3.3","v2.3.4","v2.3.5","v2.3.6","v2.3.7"]}
    d6951b3|
    d6951b3|--- BODY END ---
    DEBU[0001] [func1():quay.io/coreos/etcd:v3.3] remote tags: map[] 
    DEBU[0001] [func1():quay.io/coreos/etcd:v3.3] local tags: map[] 
    DEBU[0001] [func1():quay.io/coreos/etcd:v3.3] sending joined tags: map[v3.3:0xc4203e1db0] 
    INFO[0001] FETCHED quay.io/coreos/etcd:v3.3             
    DEBU[0001] [quay.io/coreos/etcd:v3.3] receiving tags: [0xc4203e1db0] 
    DEBU[0001] [CollectTags()] tags: map[quay.io/coreos/etcd:v3.3:[0xc4203e1db0]] 
    -
    <STATE>      <DIGEST>                                      <(local) ID>    <Created At>              <IMAGE>:<TAG>
    ASSUMED      n/a                                           n/a             1970-01-01T01:00:00       quay.io/coreos/etcd:v3.3
    -
    

    See curl output:

    $ curl -i https://quay.io/v2/coreos/etcd/tags/list                                                                                                                                           
    HTTP/2 200 
    server: nginx/1.14.2
    date: Thu, 02 May 2019 14:56:05 GMT
    content-type: application/json
    content-length: 589
    link: </v2/coreos/etcd/tags/list?next_page=gAAAAABcywUFNIm76e9bBn-An3WMgoBwdT7UnbaMDZXY7aw1BmMAh9GiY3eI7mXuyll7yUQdXNXLv1g_LadGXhD3LndANIhaIAtQHPWO7AWSYyol-pe9Yo4%3D&n=50>; rel="next"
    x-frame-options: DENY
    strict-transport-security: max-age=63072000; preload
    
    {"name":"coreos/etcd","tags":["3.1","3.2","latest","test","v0.4.6","v0.4.8","v0.5.0_alpha.0","v0.5.0_alpha.1","v0.5.0_alpha.2","v0.5.0_alpha.3","v0.5.0_alpha.4","v0.5.0_alpha.5","v2.0.0","v2.0.0_rc.1","v2.0.10","v2.0.11","v2.0.12","v2.0.13","v2.0.3","v2.0.4","v2.0.5","v2.0.6","v2.0.7","v2.0.8","v2.0.9","v2.1.0-alpha.0","v2.1.0-alpha.1","v2.1.0-rc.0","v2.1.1","v2.1.2","v2.1.3","v2.2.0","v2.2.0-alpha.0","v2.2.0-alpha.1","v2.2.0-rc.0","v2.2.1","v2.2.2","v2.2.3","v2.2.4","v2.2.5","v2.3.0","v2.3.0-alpha.0","v2.3.0-alpha.1","v2.3.1","v2.3.2","v2.3.3","v2.3.4","v2.3.5","v2.3.6","v2.3.7"]}
    

    And then whith following the link:

    $ curl -i 'https://quay.io/v2/coreos/etcd/tags/list?next_page=gAAAAABcywO9mhMhjsLLMxdC4UXKURJ4KZx9ZGxAK0AcXYp3O1uvqIPO1pv8BXp2VbUn2t-qcVpi4U7dx-_HtP_q1gi_WjbnX5QE9n1-sulVWdiZBGnvnL4%3D&n=50'
    HTTP/2 200 
    server: nginx/1.14.2
    date: Thu, 02 May 2019 14:56:39 GMT
    content-type: application/json
    content-length: 544
    link: </v2/coreos/etcd/tags/list?next_page=gAAAAABcywUn7SnuY-WUOxzJ_sJsya0-OJM7shadXBHJ2tfDrIkX5RJ8XhjvfHLsCS0uOaC6OLdYGkovvTW-nc8QnuNHl-pTRklk5Bw7W5n2MCyIeowfh_8%3D&n=50>; rel="next"
    x-frame-options: DENY
    strict-transport-security: max-age=63072000; preload
    
    {"name":"coreos/etcd","tags":["v2.3.8","v3.0","v3.0.0","v3.0.0-beta.0","v3.0.1","v3.0.10","v3.0.11","v3.0.12","v3.0.13","v3.0.14","v3.0.15","v3.0.16","v3.0.17","v3.0.2","v3.0.3","v3.0.4","v3.0.5","v3.0.6","v3.0.7","v3.0.8","v3.0.9","v3.1","v3.1.0","v3.1.0-alpha.0","v3.1.0-alpha.1","v3.1.0-rc.0","v3.1.0-rc.1","v3.1.1","v3.1.10","v3.1.11","v3.1.12","v3.1.13","v3.1.14","v3.1.14-arm64","v3.1.14-ppc64le","v3.1.15","v3.1.16","v3.1.17","v3.1.18","v3.1.19","v3.1.2","v3.1.24","v3.1.3","v3.1.4","v3.1.5","v3.1.6","v3.1.7","v3.1.8","v3.1.9","v3.2"]}
    

    And many more tags with the next link and the next link. By the way, the paging does not support more than 50.

    critical handicap 
    opened by nudgegoonies 5
  • Crash when accessing large repository

    Crash when accessing large repository

    The repository has 4020 tags, and most of them are with broken digest. I'm trying to list the tags so I could push over and delete them, but lstags crashes...

    $ lstags --version
    VERSION: v1.2.21
    $ lstags gitlab.example.net:4567/blah/builds | awk '$2 == "this.image.is.bad.it.has.no.digest.fuuu!" {print $5}'
    INFO[0000] BATCH 1 of 1
    INFO[0000] ANALYZE gitlab.example.net:4567/blah/builds
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x7f4238]
    
    goroutine 10387 [running]:
    github.com/ivanilves/lstags/api/v1/registry/client/request.Perform(0xc001ed7c70, 0x66, 0xc0048ce500, 0x492, 0x927fa5, 0x2, 0x400, 0x2, 0x1dcd65000, 0xc001ed7c70, ...)
            /go/src/github.com/ivanilves/lstags/api/v1/registry/client/request/request.go:116 +0x3f8
    github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).v1TagOptions(0xc0002e1ce0, 0xc002046ea0, 0x22, 0xc00019c060, 0x18, 0xc00019c060, 0x18, 0xc003e68ea0)
            /go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:339 +0x2a5
    github.com/ivanilves/lstags/api/v1/registry/client.(*RegistryClient).Tag(0xc0002e1ce0, 0xc002046ea0, 0x22, 0xc00019c060, 0x18, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
            /go/src/github.com/ivanilves/lstags/api/v1/registry/client/client.go:381 +0x130
    github.com/ivanilves/lstags/tag/remote.FetchTags.func1(0xc0002e1ce0, 0xc00021ed80, 0xc00019c060, 0x18, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
            /go/src/github.com/ivanilves/lstags/tag/remote/remote.go:104 +0x8b
    created by github.com/ivanilves/lstags/tag/remote.FetchTags
            /go/src/github.com/ivanilves/lstags/tag/remote/remote.go:98 +0x46f
    
    opened by glensc 2
  • bearer response should also allow for access_token

    bearer response should also allow for access_token

    via https://docs.docker.com/registry/spec/auth/token/#requesting-a-token

    token: An opaque Bearer token that clients should supply to subsequent requests in the Authorization header.

    access_token: For compatibility with OAuth 2.0, we will also accept token under the name access_token. At least one of these fields must be specified, but both may also appear (for compatibility with older clients). When both are specified, they should be equivalent; if they differ the client's choice is undefined.

    diff --git a/api/v1/registry/client/auth/bearer/bearer.go b/api/v1/registry/client/auth/bearer/bearer.go
    index 3a99170..fc1ea5b 100644
    --- a/api/v1/registry/client/auth/bearer/bearer.go
    +++ b/api/v1/registry/client/auth/bearer/bearer.go
    @@ -9,7 +9,8 @@ import (
     
     // Token implementation for Bearer authentication
     type Token struct {
    -       T string `json:"token"`
    +       T string `json:"token,omitempty"`
    +       A string `json:"access_token,omitempty"`
            E int    `json:"expires_in"`
     }
     
    @@ -20,7 +21,10 @@ func (tk Token) Method() string {
     
     // String form of Bearer token
     func (tk Token) String() string {
    -       return tk.T
    +       if tk.A == "" {
    +               return tk.T
    +       }
    +       return tk.A
     }
    

    bit dirty but it does the trick @ivanilves

    enhancement 
    opened by Deshke 1
  • 401 error while pushing since 1.2.11

    401 error while pushing since 1.2.11

    Since our update to lstags 1.2.11 we get 401 errors on pushing when images have changed.

    I turned on tracing and verbosity with 1.2.10 and 1.2.11 and copied the relevant sections for the image/tag that fails with 1.2.11. This is the working 1.2.10:

    b6874ea|@URL: https://docker.elastic.co/v2/kibana/kibana/tags/list
    b6874ea|@HEADER: Date                                     = [Thu, 09 Jan 2020 08:09:08 GMT]
    b6874ea|@HEADER: Docker-Distribution-Api-Version          = [registry/2.0]
    b6874ea|@HEADER: X-Content-Type-Options                   = [nosniff]
    b6874ea|@HEADER: Content-Length                           = [2278]
    b6874ea|@HEADER: Connection                               = [keep-alive]
    b6874ea|@HEADER: Content-Type                             = [application/json; charset=utf-8]
    b6874ea|--- BODY BEGIN ---
    b6874ea|{"name":"kibana/kibana","tags":["5.0.0-731e78df","5.0.0-86a0b164","5.0.0-alpha5","5.0.0-beta1","5.0.0-ccd69424","5.0.0-rc1","5.0.0","5.0.1-6d75f4de","5.0.1-c068af18","5.0.1","5.0.2","5.1.1-71988514","5.1.1","5.1.2","5.2.0-11b63c8d","5.2.0-6ee3d9ba","5.2.0-a9f63967","5.2.0","5.2.1","5.2.2","5.3.0-d5b30bd7","5.3.0","5.3.1","5.3.2","5.3.3","5.4.0","5.4.1","5.4.2","5.4.3","5.5.0-e210c18b","5.5.0","5.5.1","5.5.2","5.5.3","5.6.0","5.6.1","5.6.10","5.6.11","5.6.12","5.6.13","5.6.14","5.6.15","5.6.16","5.6.2","5.6.3","5.6.4-1fdad85","5.6.4","5.6.5","5.6.6","5.6.7","5.6.8","5.6.9","6.0.0-alpha1","6.0.0-alpha2","6.0.0-beta1","6.0.0-beta2-d2816397","6.0.0-beta2","6.0.0-rc1","6.0.0-rc2","6.0.0","6.0.1","6.1.0","6.1.1","6.1.2","6.1.3","6.1.4","6.2.0","6.2.1","6.2.2","6.2.3","6.2.4","6.3.0","6.3.1","6.3.2","6.3.3-SNAPSHOT","6.4-SNAPSHOT","6.4.0-SNAPSHOT","6.4.0","6.4.1-SNAPSHOT","6.4.1","6.4.2-SNAPSHOT","6.4.2","6.4.3-SNAPSHOT","6.4.3","6.4.4-SNAPSHOT","6.5-SNAPSHOT","6.5.0-SNAPSHOT","6.5.0","6.5.1-109","6.5.1-SNAPSHOT","6.5.1","6.5.2-SNAPSHOT","6.5.2","6.5.3-SNAPSHOT","6.5.3","6.5.4-SNAPSHOT","6.5.4","6.5.5-SNAPSHOT","6.6-SNAPSHOT","6.6.0-SNAPSHOT","6.6.0","6.6.1-SNAPSHOT","6.6.1","6.6.2-SNAPSHOT","6.6.2","6.6.3-SNAPSHOT","6.7-SNAPSHOT","6.7.0-SNAPSHOT","6.7.0","6.7.1-SNAPSHOT","6.7.1","6.7.2-SNAPSHOT","6.7.2","6.7.3-SNAPSHOT","6.8-SNAPSHOT","6.8.0-SNAPSHOT","6.8.0","6.8.1-SNAPSHOT","6.8.1-e43faf64","6.8.1","6.8.2-SNAPSHOT","6.8.2","6.8.3-SNAPSHOT","6.8.3","6.8.4-SNAPSHOT","6.8.4","6.8.5-SNAPSHOT","6.8.5","6.8.6-SNAPSHOT","6.8.6","6.8.7-SNAPSHOT","6.x-SNAPSHOT","7.0-SNAPSHOT","7.0.0-SNAPSHOT","7.0.0-alpha1-SNAPSHOT","7.0.0-alpha1","7.0.0-alpha2","7.0.0-beta1","7.0.0-rc1","7.0.0-rc2","7.0.0","7.0.1-SNAPSHOT","7.0.1","7.0.2-SNAPSHOT","7.1-SNAPSHOT","7.1.0-SNAPSHOT","7.1.0","7.1.1-SNAPSHOT","7.1.1","7.1.2-SNAPSHOT","7.2-SNAPSHOT","7.2.0-SNAPSHOT","7.2.0","7.2.1-SNAPSHOT","7.2.1","7.2.2-SNAPSHOT","7.3-SNAPSHOT","7.3.0-SNAPSHOT","7.3.0","7.3.1-SNAPSHOT","7.3.1","7.3.2-SNAPSHOT","7.3.2","7.3.3-SNAPSHOT","7.4-SNAPSHOT","7.4.0-SNAPSHOT","7.4.0","7.4.1-SNAPSHOT","7.4.1","7.4.2-SNAPSHOT","7.4.2","7.4.3-SNAPSHOT","7.5-SNAPSHOT","7.5.0-SNAPSHOT","7.5.0","7.5.1-SNAPSHOT","7.5.1","7.5.2-SNAPSHOT","7.6.0-SNAPSHOT","7.x-SNAPSHOT","8.0.0-SNAPSHOT","master-SNAPSHOT"]}
    b6874ea|
    b6874ea|--- BODY END ---
    [...]
    12c6fdb|@URL: https://$ARTIFACTORY/v2/kibana/kibana/tags/list
    12c6fdb|@HEADER: Connection                               = [keep-alive]
    12c6fdb|@HEADER: Server                                   = [Artifactory/6.16.0]
    12c6fdb|@HEADER: X-Artifactory-Id                         = [$ID]
    12c6fdb|@HEADER: Docker-Distribution-Api-Version          = [registry/2.0]
    12c6fdb|@HEADER: Date                                     = [Thu, 09 Jan 2020 08:09:15 GMT]
    12c6fdb|@HEADER: Content-Type                             = [application/json]
    

    This is the failing 1.2.11:

    d55957a|@URL: https://docker.elastic.co/v2/kibana/kibana/tags/list
    d55957a|@HEADER: Content-Type                             = [application/json; charset=utf-8]
    d55957a|@HEADER: Date                                     = [Thu, 09 Jan 2020 08:03:50 GMT]
    d55957a|@HEADER: Docker-Distribution-Api-Version          = [registry/2.0]
    d55957a|@HEADER: X-Content-Type-Options                   = [nosniff]
    d55957a|@HEADER: Content-Length                           = [2278]
    d55957a|@HEADER: Connection                               = [keep-alive]
    d55957a|--- BODY BEGIN ---
    d55957a|{"name":"kibana/kibana","tags":["5.0.0-731e78df","5.0.0-86a0b164","5.0.0-alpha5","5.0.0-beta1","5.0.0-ccd69424","5.0.0-rc1","5.0.0","5.0.1-6d75f4de","5.0.1-c068af18","5.0.1","5.0.2","5.1.1-71988514","5.1.1","5.1.2","5.2.0-11b63c8d","5.2.0-6ee3d9ba","5.2.0-a9f63967","5.2.0","5.2.1","5.2.2","5.3.0-d5b30bd7","5.3.0","5.3.1","5.3.2","5.3.3","5.4.0","5.4.1","5.4.2","5.4.3","5.5.0-e210c18b","5.5.0","5.5.1","5.5.2","5.5.3","5.6.0","5.6.1","5.6.10","5.6.11","5.6.12","5.6.13","5.6.14","5.6.15","5.6.16","5.6.2","5.6.3","5.6.4-1fdad85","5.6.4","5.6.5","5.6.6","5.6.7","5.6.8","5.6.9","6.0.0-alpha1","6.0.0-alpha2","6.0.0-beta1","6.0.0-beta2-d2816397","6.0.0-beta2","6.0.0-rc1","6.0.0-rc2","6.0.0","6.0.1","6.1.0","6.1.1","6.1.2","6.1.3","6.1.4","6.2.0","6.2.1","6.2.2","6.2.3","6.2.4","6.3.0","6.3.1","6.3.2","6.3.3-SNAPSHOT","6.4-SNAPSHOT","6.4.0-SNAPSHOT","6.4.0","6.4.1-SNAPSHOT","6.4.1","6.4.2-SNAPSHOT","6.4.2","6.4.3-SNAPSHOT","6.4.3","6.4.4-SNAPSHOT","6.5-SNAPSHOT","6.5.0-SNAPSHOT","6.5.0","6.5.1-109","6.5.1-SNAPSHOT","6.5.1","6.5.2-SNAPSHOT","6.5.2","6.5.3-SNAPSHOT","6.5.3","6.5.4-SNAPSHOT","6.5.4","6.5.5-SNAPSHOT","6.6-SNAPSHOT","6.6.0-SNAPSHOT","6.6.0","6.6.1-SNAPSHOT","6.6.1","6.6.2-SNAPSHOT","6.6.2","6.6.3-SNAPSHOT","6.7-SNAPSHOT","6.7.0-SNAPSHOT","6.7.0","6.7.1-SNAPSHOT","6.7.1","6.7.2-SNAPSHOT","6.7.2","6.7.3-SNAPSHOT","6.8-SNAPSHOT","6.8.0-SNAPSHOT","6.8.0","6.8.1-SNAPSHOT","6.8.1-e43faf64","6.8.1","6.8.2-SNAPSHOT","6.8.2","6.8.3-SNAPSHOT","6.8.3","6.8.4-SNAPSHOT","6.8.4","6.8.5-SNAPSHOT","6.8.5","6.8.6-SNAPSHOT","6.8.6","6.8.7-SNAPSHOT","6.x-SNAPSHOT","7.0-SNAPSHOT","7.0.0-SNAPSHOT","7.0.0-alpha1-SNAPSHOT","7.0.0-alpha1","7.0.0-alpha2","7.0.0-beta1","7.0.0-rc1","7.0.0-rc2","7.0.0","7.0.1-SNAPSHOT","7.0.1","7.0.2-SNAPSHOT","7.1-SNAPSHOT","7.1.0-SNAPSHOT","7.1.0","7.1.1-SNAPSHOT","7.1.1","7.1.2-SNAPSHOT","7.2-SNAPSHOT","7.2.0-SNAPSHOT","7.2.0","7.2.1-SNAPSHOT","7.2.1","7.2.2-SNAPSHOT","7.3-SNAPSHOT","7.3.0-SNAPSHOT","7.3.0","7.3.1-SNAPSHOT","7.3.1","7.3.2-SNAPSHOT","7.3.2","7.3.3-SNAPSHOT","7.4-SNAPSHOT","7.4.0-SNAPSHOT","7.4.0","7.4.1-SNAPSHOT","7.4.1","7.4.2-SNAPSHOT","7.4.2","7.4.3-SNAPSHOT","7.5-SNAPSHOT","7.5.0-SNAPSHOT","7.5.0","7.5.1-SNAPSHOT","7.5.1","7.5.2-SNAPSHOT","7.6.0-SNAPSHOT","7.x-SNAPSHOT","8.0.0-SNAPSHOT","master-SNAPSHOT"]}
    d55957a|
    d55957a|--- BODY END ---
    
    Bad response status: 401 Unauthorized >> https://$ARTIFACTORY/v2/kibana/kibana/tags/list
    

    Because this error comes from authentication there is no useful log from Artifactory. The nginx log shows these:

    [09/Jan/2020:08:03:15 +0000] "GET /artifactory/api/docker/docker-external-local/v2/kibana/kibana/tags/list HTTP/1.1" 200 119 "-" "Go-http-client/1.1"
    [09/Jan/2020:08:09:56 +0000] "GET /artifactory/api/docker/docker-external-local/v2/kibana/kibana/tags/list HTTP/1.1" 401 101 "-" "Go-http-client/1.1"
    

    I suspect this has to do with the new token handling that was merged into 1.2.11.

    possible bug 
    opened by nudgegoonies 7
  • Allow synchronization with sha syntax

    Allow synchronization with sha syntax

    For images that only have a latest tag it would be useful to sync an image via SHA. This is an example image that we stumbled on where we mirror the SHA manually:

    solsson/[email protected]:6f82e2b0464f50da8104acd7363fb9b995001ddff77d248379f8788e78946143
    
    feature 
    opened by nudgegoonies 1
  • Multi-platform images do not show

    Multi-platform images do not show

    Hello,

    I noticed an error with multi-platform images (at least in my configuration w/ Ubuntu 18.04 out of the box install).

    When mirroring https://quay.io/repository/prometheus/alertmanager the images sync perfectly, until image v0.17.0, where the images seem to download. But upon re-run, the images never get to "PRESENT" but remain in "CHANGED" state each run.

    To reproduce:

    lstags -v quay.io/prometheus/alertmanager:v0.18.0

    CHANGED sha256:3c0f0ea52a662342282fc7236ce058ab4 ce3c87f17369 2019-07-08T08:07:19-07:00 quay.io/prometheus/alertmanager:v0.18.0

    This above command never succeeds in updating v0.18.0, despite docker being tasked with a new download.

    docker images --digests | grep v0.18.0

    quay.io/prometheus/alertmanager v0.18.0 sha256:3d52ab7ef3a2e23e26a73cfd0ce13189e3796fb276b9af3e79664e4f743d8a1d ce3c87f17369 2 months ago 51.9MB

    I believe this issue is due to Docker using the top level checksum, while lstags uses the container specific sha256. Looking at https://quay.io/repository/prometheus/alertmanager?tag=latest&tab=tags I noticed the lstags checksum matches the child manifest checksum.

    Furthermore, this behavior for prometheus/alertmanager started with v0.17.0, the same time that alertmanager shows multi-platform containers added.

    Thank you for any suggestions, or if I can provide more info or test, I would be happy to.

    Matt

    bug handicap 
    opened by mattsimonsen 1
Releases(v1.2.23)
Owner
Ivan Ilves
Seasoned Linux system admin and DevOps engineer. Sceptic, realistic, but still in love with the industry. ;)
Ivan Ilves
Container-Explorer is a tool to explore containerd installation on a mounted image.

Container-Explorer Container-Explorer is a tool to explore containerd installation on a mounted image. Container-Explorer attempts to provide the simi

Google 22 Aug 3, 2022
Vilicus is an open source tool that orchestrates security scans of container images(docker/oci) and centralizes all results into a database for further analysis and metrics.

Vilicus Table of Contents Overview How does it work? Architecture Development Run deployment manually Usage Example of analysis Overview Vilicus is an

Ederson Brilhante 77 Jun 21, 2022
ghcr images - Fetched from docker-library

ghcri ghcri is the repo for Github Container Registry Images. Just like docker-library for Docker Registry. Usage Replace all docker library from dock

null 15 Dec 2, 2021
CLI based tools to find the secrets in docker Images

docker-secrets CLI based tools to find the secrets in docker Images This tool use detect-secrets to find the secrets in the docker Image file system P

joshua_jebaraj 2 Mar 22, 2022
run regular Docker images in KVM/Qemu

runq runq is a hypervisor-based Docker runtime based on runc to run regular Docker images as a lightweight KVM/Qemu virtual machine. The focus is on s

null 721 Jul 27, 2022
🥑 Language focused docker images, minus the operating system.

"Distroless" Docker Images "Distroless" images contain only your application and its runtime dependencies. They do not contain package managers, shell

null 13.4k Aug 4, 2022
Woodpecker CI plugin to build multiarch Docker images with buildx

plugin-docker-buildx Woodpecker CI plugin to build multiarch Docker images with buildx Woodpecker CI plugin to build multiarch Docker images with buil

Woodpecker CI 2 Apr 25, 2022
A tool to check whether docker images exist in the remote registry.

Check Docker Image A tool to check whether docker images exist in the remote registry. Build project: go build -o check-image . Example usage: REGISTR

Hao-Ming, Hsu 3 Jul 26, 2022
Show dependency graph of docker images/containers

docker-graph Show dependency graph of docker images/containers like this: Orange is images and green is containers. Features Collect docker images, co

Tomohisa Hirami 0 Feb 7, 2022
Go-http-server-docker - Simple sample server using docker and go

go-http-server-docker Simple sample webserver using docker and go.

null 0 Jan 8, 2022
Dotnet-appsettings-env - Convert .NET appsettings.json file to Kubernetes, Docker and Docker-Compose environment variables

dotnet-appsettings-env Convert .NET appsettings.json file to Kubernetes, Docker

Daniel Dias de Assumpção 5 Jun 13, 2022
Dotnet-appsettings-env - Convert .NET appsettings.json file to Kubernetes, Docker and Docker-Compose environment variables

dotnet-appsettings-env Convert .NET appsettings.json file to Kubernetes, Docker

Daniel Dias de Assumpção 1 Feb 16, 2022
A Simple and Comprehensive Vulnerability Scanner for Container Images, Git Repositories and Filesystems. Suitable for CI

A Simple and Comprehensive Vulnerability Scanner for Containers and other Artifacts, Suitable for CI. Table of Contents Abstract Features Installation

Aqua Security 13.2k Aug 7, 2022
Stream, Mutate and Sign Images with AWS Lambda and ECR

ocistow About How it works Try it yourself Prerequisites CLI (cmd/ocistow) Lambda (cmd/ocistow-lambda) Deploy Invoke Verify signatures with =cosign= I

Martin Baillie 17 May 12, 2022
Docker-based remote code runner / 基于 Docker 的远程代码运行器

Docker-based remote code runner / 基于 Docker 的远程代码运行器

E99p1ant 35 Feb 22, 2022
ecsk is a CLI tool to interactively use frequently used functions of docker command in Amazon ECS. (docker run, exec, cp, logs, stop)

English / 日本語 ecsk ECS + Task = ecsk ?? ecsk is a CLI tool to interactively use frequently used functions of docker command in Amazon ECS. (docker run

null 103 Jul 12, 2022
Hassle-free minimal CI/CD for git repositories with docker or docker-compose projects.

GIT-PIPE Hassle-free minimal CI/CD for git repos for docker-based projects. Features: zero configuration for repos by default automatic encrypted back

Aleksandr Baryshnikov 51 Oct 8, 2021
Tool to convert docker-compose files to set of simple docker commands

docker-decompose Tool to convert docker-compose files to set of simple docker commands. Install Use go get to install the latest version of the librar

Liri S 2 Apr 12, 2022
Docker-hub-rate-limit - Show pulling rate status of Docker-hub

Docker-Hub Pull Rate Status This tool shows current status of docker hub pull ra

Tak 1 Jan 28, 2022