A magic shim for Docker credential helpers

Overview

docker-credential-magic

GitHub Actions status GoDoc

docker-credential-magic

Overview

This repo contains the source for two separate tools:

  • docker-credential-magic - credential helper which proxies auth to other helpers based on domain name
  • docker-credential-magician - tool to augment images with various credential helpers (including magic)

The following third-party Docker credential helpers are currently supported:

Installation

Download latest release tarball for your system and install both tools manually:

cat docker-credential-magic*.tar.gz | tar x -C /usr/local/bin 'docker-credential-magic*'

Usage

How to use docker-credential-magic

$ echo  | docker-credential-magic get

The following example shows how docker-credential-magic can be used to proxy auth to docker-credential-gcr, based on the detection of a *.gcr.io domain:

Note: Example requires docker-credential-gcr to be pre-installed

$ export GOOGLE_APPLICATION_CREDENTIALS="${PWD}/service-account-key.json"
$ echo "us.gcr.io" | docker-credential-magic get
{"ServerURL":"us.gcr.io","Username":"_dcgcr_token","Secret":"*****"}

Note: docker-credential-magic is a "read-only" credential helper, and does not modify credentials in any way (some helpers implement other subcommands like store or erase).

Local setup

The primary purpose of magic is to be added to images via magician. However, you may wish to also use this tool on your local machine.

It is required that mappings files for each supported helper are present on disk. The environment variable DOCKER_CREDENTIAL_MAGIC_CONFIG is used by magic to find these files, nested under an etc/ subdirectory.

If DOCKER_CREDENTIAL_MAGIC_CONFIG is not set, magic respects the XDG Base Directory Specification, looking for mappings files under $XDG_CONFIG_HOME/magic/etc/. This is equivalent to the following on each operating system:

  • Linux: $HOME/.config/magic/etc/
  • macOS: $HOME/Library/Application Support/magic/etc
  • Windows: %APPDATA%\magic\etc

magic has a useful subcommand, init, which will auto-create this directory and populate it with the default mappings files, as well as a catch-all Docker config.json file in the parent directory:

$ docker-credential-magic init
Creating directory '/Users/me/Library/Application Support/magic/etc' ...
Creating mapping file '/Users/me/Library/Application Support/magic/etc/aws.yml' ...
Creating mapping file '/Users/me/Library/Application Support/magic/etc/azure.yml' ...
Creating mapping file '/Users/me/Library/Application Support/magic/etc/gcp.yml' ...
Creating magic config file '/Users/me/Library/Application Support/magic/config.json' ...

magic has another subcommand, home, which you can use to modify the DOCKER_CONFIG env var to point to the magic directory:

$ export DOCKER_CONFIG="$(docker-credential-magic home)"

You may wish to add the previous command to your ~/.bashrc / ~/.bash_profile.

If no matching domains are found, magic will fall back to use your existing $HOME/.docker/config.json.

Note: At this time, magic will not automatically install the supported helpers on your machine. You should install each of these manually. For example, to install ecr-login on macOS via Homebrew:

$ brew install docker-credential-helper-ecr

How to use docker-credential-magician

$ docker-credential-magician mutate 

The following example shows how docker-credential-magician can be used to (1) augment the cosign image with various credential helpers, (2) set the default credential store to magic, and (3) push the new image to a registry running at localhost:5000:

$ docker-credential-magician mutate \
    gcr.io/projectsigstore/cosign/ci/cosign:v0.5.0 \
    -t localhost:5000/cosign:v0.5.0-magic
2021/07/29 17:06:59 Pulling gcr.io/projectsigstore/cosign/ci/cosign:v0.5.0 ...
2021/07/29 17:07:01 Adding /opt/magic/etc/aws.yml ...
2021/07/29 17:07:01 Adding /opt/magic/etc/azure.yml ...
2021/07/29 17:07:01 Adding /opt/magic/etc/gcp.yml ...
2021/07/29 17:07:01 Adding /opt/magic/bin/docker-credential-ecr-login ...
2021/07/29 17:07:01 Adding /opt/magic/bin/docker-credential-acr-env ...
2021/07/29 17:07:01 Adding /opt/magic/bin/docker-credential-gcr ...
2021/07/29 17:07:01 Adding /opt/magic/bin/docker-credential-magic ...
2021/07/29 17:07:01 Adding /opt/magic/config.json ...
2021/07/29 17:07:02 Prepending PATH with /opt/magic/bin ...
2021/07/29 17:07:02 Setting DOCKER_CONFIG to /opt/magic ...
2021/07/29 17:07:02 Setting DOCKER_CREDENTIAL_MAGIC_CONFIG to /opt/magic ...
2021/07/29 17:07:02 Pushing image to localhost:5000/cosign:v0.5.0-magic ...
2021/07/29 17:07:04 Done.
$ docker run --rm --entrypoint sh \
    localhost:5000/cosign:v0.5.0-magic \
    -c 'ls -lah /opt/magic/etc && \
        ls -lah /opt/magic/bin &&
        env | grep DOCKER_ &&
        cat $DOCKER_CONFIG/config.json'
total 20K
drwxr-xr-x    2 root     root        4.0K Jul 29 21:00 .
drwxr-xr-x    4 root     root        4.0K Jul 29 21:00 ..
-r-xr-xr-x    1 root     root          45 Jan  1  1970 aws.yml
-r-xr-xr-x    1 root     root          40 Jan  1  1970 azure.yml
-r-xr-xr-x    1 root     root          44 Jan  1  1970 gcp.yml
total 25M
drwxr-xr-x    2 root     root        4.0K Jul 29 21:00 .
drwxr-xr-x    4 root     root        4.0K Jul 29 21:00 ..
-r-xr-xr-x    1 root     root        8.7M Jan  1  1970 docker-credential-acr-env
-r-xr-xr-x    1 root     root        7.8M Jan  1  1970 docker-credential-ecr-login
-r-xr-xr-x    1 root     root        5.6M Jan  1  1970 docker-credential-gcr
-r-xr-xr-x    1 root     root        3.0M Jan  1  1970 docker-credential-magic
DOCKER_CREDENTIAL_MAGIC_CONFIG=/opt/magic
DOCKER_CONFIG=/opt/magic
{"credsStore":"magic"}

If the -t / --tag flag is not provided, magician will default to publishing the image back to its original location (overwriting the existing tag).

Note: At this time, docker-credential-magician is only designed for x86–64/AMD64 Linux containers. More platforms may be supported in the future.

Including a subset of helpers

You may specify the -i / --include flag (one or more times) to limit the helpers that are added to the image.

For example, to only include the azure and gcp helpers:

$ docker-credential-magician mutate example.com/myimage:1.2.3 \
    -i azure -i gcp

Note: These each must match one of the supported helpers found in the mappings/ directory.

Using custom mappings and/or helpers

In some scenarios, you may wish to supply a custom directory of mappings, for example to add extra domains. Or, you may wish to supply a custom directory of helper binaries, if you need to use a different version of a helper, or add your own.

For these cases, you can use the following flags:

  • --mappings-dir
  • --helpers-dir

Please note that all mappings and helpers must be provided (as in, magician will not automatically resolve any missing binaries in ).

In addition, all helpers in must be built for a Linux amd64 architecture.

Lastly, the magic helper will always be sourced from the one baked into magician.

Go library

You may wish to make use of magician functionality in your Go application.

Here is a Go example which mimics the command-line example found above:

package main

import (
	"github.com/docker-credential-magic/docker-credential-magic/pkg/magician"
)

func main() {
	src := "gcr.io/projectsigstore/cosign/ci/cosign:v0.5.0"
	dst := "localhost:5000/cosign:v0.5.0-magic"

	err := magician.Mutate(src, magician.MutateOptWithTag(dst))
	if err != nil {
		panic(err)
	}
}

Note: since magician makes use of embedded files (helper binaries and mappings), if you try running this example, you will see errors such as the following:

pattern embedded/*: no matching files found

To fix this, you must manually populate these directories in your GOPATH after running go mod tidy:

git clone https://github.com/docker-credential-magic/docker-credential-magic.git tmp/
pushd tmp/
make clean vendor fetch-helpers copy-mappings build-magic-embedded
popd
for d in $(find "$(go env | grep GOPATH | awk -F "=" '{print $2}' | tr -d '"')/pkg/mod/github.com/docker-credential-magic" -mindepth 1 -maxdepth 1 -type d); do
  sudo cp -r tmp/internal/embedded/helpers/embedded "${d}/internal/embedded/helpers" || true
  sudo cp -r tmp/internal/embedded/mappings/embedded "${d}/internal/embedded/mappings" || true
done
rm -rf tmp/

(If there is an easier way to approach this, please let us know)

Project history

The original concept for this project and its design can be found in the following GitHub conversations:

If you are interested in understanding how Docker credential helpers work, you may enjoy this image.

Contributing

Contributions are welcome!

Prior to submitting a pull request, please check the list of open issues. If there is not an existing issue related to your changes, please open a new issue to first discuss your thoughts with the project maintainers.

Adding support for a new helper

If you are contributing support for another helper, here are the necessary steps:

For more info on project development, please see this page.

Comments
  • Domain public.aws.ecr fails if not authed to AWS

    Domain public.aws.ecr fails if not authed to AWS

    Discovered in https://github.com/helm/helm/issues/10557

    $ cat ~/.docker/config.json 
    {
    	"credsStore": "magic"
    }
    $ bin/helm pull oci://public.ecr.aws/aws-controllers-k8s/apigatewayv2-chart --version v0.0.8
    Error: error getting credentials - err: exit status 1, out: `credentials not found in native keychain
    [magic] exec "docker-credential-ecr-login": exit status 1`
    

    If I remove ecr.aws from the mapping file with

    cat "$(docker-credential-magic home)/etc/aws.yml" | grep -v "ecr.aws" > ./aws.yml && mv ./aws.yml "$(docker-credential-magic home)/etc/aws.yml"
    

    then this works:

    $ bin/helm pull oci://public.ecr.aws/aws-controllers-k8s/apigatewayv2-chart --version v0.0.8
    Pulled: public.ecr.aws/aws-controllers-k8s/apigatewayv2-chart:v0.0.8
    Digest: sha256:299d8b520291ade6d136a7529b7fd44338d58b5b8239813aed97e8fd81ca1f00
    

    Running the ECR helper directly:

    $ echo "public.ecr.aws" | docker-credential-ecr-login get
    credentials not found in native keychain
    

    Important to note, I am not authed to AWS on this machine.

    Looks like we introduced ecr.aws in https://github.com/docker-credential-magic/docker-credential-magic/pull/37 cc @rothgar

    Is this a bug in docker-credential-ecr-login then? Should that binary give us an empty token response?

    bug 
    opened by jdolitsky 4
  • Prevent endless loop on fallback, update README, Go, depends

    Prevent endless loop on fallback, update README, Go, depends

    If no matching domain is found, "getFallback()" may end up loading a Docker config file that uses "magic" as the global cred store.

    ~~To prevent this, we add DOCKER_CREDENTIAL_MAGIC_CHILD=1 to any subprocess spawned by magic itself, then short circuit the process if it is detected.~~

    To prevent this, we check if credStore is magic (or if a credHelper is mapped to magic) then short circuit the process if detected.

    Resolves #43


    Clarify initial usage of magic

    Resolves #39


    Update go mods, use Go 1.17

    opened by jdolitsky 2
  • Add cred helper wrappers for ECR and ACR

    Add cred helper wrappers for ECR and ACR

    This currently depends on https://github.com/google/go-containerregistry/pull/1227, so there's a replace in go.mod which should be removed after that's merged, and this should depend on [email protected]

    This will enable folks who want to auth with GCR, ECR and ACR to do so without using invoking cred helper executables, by doing:

    import (
       "github.com/docker-credential-magic/docker-credential-magic/pkg/amazon"
       "github.com/docker-credential-magic/docker-credential-magic/pkg/azure"
       "github.com/google/go-containerregistry/pkg/authn"
       "github.com/google/go-containerregistry/pkg/v1/google"
       "github.com/google/go-containerregistry/pkg/v1/remote"
    )
    
    func main() {
      kc := authn.NewMultiKeychain(
        authn.DefaultKeychain,
        google.Keychain,
        amazon.Keychain,
        azure.Keychain,
      )
      img, err := remote.Get(ref, remote.WithAuthFromKeychain(kc))
    }
    

    This also enables users to change the preferred ordering (if it matters), and to remove ones they don't care to take a dependency on.

    We could also add a kubernetes.NewKeychain(client) that looks up imagePullSecrets in a K8s cluster and uses those, which would effectively give us an a la carte version of k8schain, and let us deprecate it.

    cc @jonjohnsonjr @jdolitsky @mattmoor

    opened by imjasonh 2
  • Add ECR public to default AWS mapping

    Add ECR public to default AWS mapping

    By default will authenticate to public ECR repos which will help with increased limits.

    https://docs.aws.amazon.com/AmazonECR/latest/public/public-service-quotas.html

    opened by rothgar 2
  • Always push image

    Always push image

    In magician, always push the image, skip Docker daemon entirely.

    If users want this locally, they can use a -t localhost:5000/....

    If no -t flag is provided, attempt to mutate in place (push over existing tag)

    opened by jdolitsky 2
  • Automagically build out ~/.docker/config

    Automagically build out ~/.docker/config

    Currently it is still required to append domains manually to ~/.docker/config.

    Somehow enumerate all domains per helper and append them to this file.

    Also, determine the correct user (not always /root/.docker)

    opened by jdolitsky 2
  • Fresh installation throws `init` error while following

    Fresh installation throws `init` error while following "Usage" section in the README

    I was following the Usage section in the README just after installed the tool. After running the following command, it throws an error.

    $ echo "index.docker.io" | docker-credential-magic get
    
    [magic] getting helper executable for domain: Directory '/Users/furkan.turkal/Library/Application Support/magic/etc' does not exist.
    Hint: Try running "docker-credential-magic init"
    

    It had better to point this out in the usage instructions, before running the get:

    $ docker-credential-magic init
    

    Thanks!

    opened by Dentrax 1
  • YAML file containing mappings for domain -> helper

    YAML file containing mappings for domain -> helper

    YAML file in this repo which contains the mappings. Magician built using this file, and users can override this when augmenting an image using a flag such as --mappings=mymap.yml

    opened by jdolitsky 1
  • Handle existing credential file

    Handle existing credential file

    If an image already has a a ~/.docker/config.json or DOCKER_CONFIG env var set, then this should be set in some env var such as ORIG_DOCKER_CONFIG, and the magic helper should fallback to use this if no domains match

    opened by jdolitsky 1
  • sign binaries with keyless approach by making use of cosign in GoReleaser and generate SBOMs for them

    sign binaries with keyless approach by making use of cosign in GoReleaser and generate SBOMs for them

    hello folks, I'd like to help with this issue to make that happen. GoReleaser has all the support that I mentioned in the title of the issue, so, we can use them to make our binaries more secure.

    Please feel free to assign this issue to me 🙋🏻‍♂️

    @jonjohnsonjr

    opened by developer-guy 4
  • Support for other platforms in magician

    Support for other platforms in magician

    Currently only Linux x86-64/AMD64 is supported for magician, but should support more. Possibly release separate binaries if we are still embedding the credential helpers.

    help wanted 
    opened by jdolitsky 0
  • Auto-build new version when new credential helpers are available

    Auto-build new version when new credential helpers are available

    Ideally, when a new version of one of the supported helpers is released, a new version of this tool is built and released, so that we do not need to manually bump versions and perform releases.

    help wanted 
    opened by jdolitsky 0
Releases(v0.8.1)
Owner
A magic shim for Docker credential helpers 🪄
null
Runwasi - A containerd shim which runs wasm workloads in wasmtime

containerd-shim-wasmtime-v1 This is a containerd shim which runs wasm workloads

Brian Goff 155 Nov 22, 2022
Small helper to bridge between Vault and AWS Credential Process.

vault-aws-credential-helper The Vault AWS Credential Helper is a component that can be injected into a task environment and be used as a credential he

Michael Aldridge 1 Nov 21, 2021
Ydb-go-yc-metadata - Helpers to connect to YDB inside yandex-cloud using metadata service

ydb-go-yc-metadata helpers to connect to YDB inside yandex-cloud using metadata

YDB Platform 2 Apr 19, 2022
Explore Docker registries and manipulate Docker images!

L/S tags Utility and API to manipulate (analyze, synchronize and aggregate) images across different Docker registries. Example invocation $ lstags alp

Ivan Ilves 302 Nov 25, 2022
Docker-based remote code runner / 基于 Docker 的远程代码运行器

Docker-based remote code runner / 基于 Docker 的远程代码运行器

E99p1ant 35 Nov 9, 2022
ecsk is a CLI tool to interactively use frequently used functions of docker command in Amazon ECS. (docker run, exec, cp, logs, stop)

English / 日本語 ecsk ECS + Task = ecsk ?? ecsk is a CLI tool to interactively use frequently used functions of docker command in Amazon ECS. (docker run

null 109 Nov 15, 2022
Hassle-free minimal CI/CD for git repositories with docker or docker-compose projects.

GIT-PIPE Hassle-free minimal CI/CD for git repos for docker-based projects. Features: zero configuration for repos by default automatic encrypted back

Aleksandr Baryshnikov 50 Sep 23, 2022
Tool to convert docker-compose files to set of simple docker commands

docker-decompose Tool to convert docker-compose files to set of simple docker commands. Install Use go get to install the latest version of the librar

Liri S 2 Apr 12, 2022
Go-http-server-docker - Simple sample server using docker and go

go-http-server-docker Simple sample webserver using docker and go.

null 0 Jan 8, 2022
Docker-hub-rate-limit - Show pulling rate status of Docker-hub

Docker-Hub Pull Rate Status This tool shows current status of docker hub pull ra

Tak 1 Jan 28, 2022
Dotnet-appsettings-env - Convert .NET appsettings.json file to Kubernetes, Docker and Docker-Compose environment variables

dotnet-appsettings-env Convert .NET appsettings.json file to Kubernetes, Docker

Daniel Dias de Assumpção 15 Oct 28, 2022
Dotnet-appsettings-env - Convert .NET appsettings.json file to Kubernetes, Docker and Docker-Compose environment variables

dotnet-appsettings-env Convert .NET appsettings.json file to Kubernetes, Docker

Daniel Dias de Assumpção 1 Feb 16, 2022
Copy files and artifacts via SSH using a binary, docker or Drone CI.

drone-scp Copy files and artifacts via SSH using a binary, docker or Drone CI. Feature Support routines. Support wildcard pattern on source list. Supp

Bo-Yi Wu 114 Nov 16, 2022
Vilicus is an open source tool that orchestrates security scans of container images(docker/oci) and centralizes all results into a database for further analysis and metrics.

Vilicus Table of Contents Overview How does it work? Architecture Development Run deployment manually Usage Example of analysis Overview Vilicus is an

Ederson Brilhante 79 Nov 17, 2022
Reward is a Swiss Army knife CLI utility for orchestrating Docker based development environments.

Reward Reward is a Swiss Army knife CLI utility for orchestrating Docker based development environments. It makes possible to run multiple local envir

null 49 Nov 10, 2022
Fast docker image distribution plugin for containerd, based on CRFS/stargz

[ ⬇️ Download] [ ?? Browse images] [ ☸ Quick Start (Kubernetes)] [ ?? Quick Start (nerdctl)] Stargz Snapshotter Read also introductory blog: Startup C

containerd 746 Nov 28, 2022
Docker LNMP (Nginx, PHP7/PHP5, MySQL, Redis)

DNMP(Docker + Nginx + MySQL + PHP7/5 + Redis)是一款全功能的LNMP一键安装程序。 使用前最好提前阅读一遍目录,以便快速上手,遇到问题也能及时排除。

Gary Meng 3.3k Nov 30, 2022
Testcontainers is a Golang library that providing a friendly API to run Docker container. It is designed to create runtime environment to use during your automatic tests.

When I was working on a Zipkin PR I discovered a nice Java library called Testcontainers. It provides an easy and clean API over the go docker sdk to

null 1.7k Nov 23, 2022
The lazier way to manage everything docker

A simple terminal UI for both docker and docker-compose, written in Go with the gocui library. This Just In: Github Sponsors is matching every donatio

Jesse Duffield 24.4k Nov 23, 2022