Go library and CLIs for working with container registries

Related tags

Command Line docker registry container container-registry
Overview

go-containerregistry

GitHub Actions Build Status GoDoc Go Report Card Code Coverage

Introduction

This is a golang library for working with container registries. It's largely based on the Python library of the same name.

The following diagram shows the main types that this library handles. OCI image representation

Philosophy

The overarching design philosophy of this library is to define interfaces that present an immutable view of resources (e.g. Image, Layer, ImageIndex), which can be backed by a variety of medium (e.g. registry, tarball, daemon, ...).

To complement these immutable views, we support functional mutations that produce new immutable views of the resulting resource (e.g. mutate). The end goal is to provide a set of versatile primives that can compose to do extraordinarily powerful things efficiently and easily.

Both the resource views and mutations may be lazy, eager, memoizing, etc, and most are optimized for common paths based on the tooling we have seen in the wild (e.g. writing new images from disk to the registry as a compressed tarball).

Experiments

Over time, we will add new functionality under experimental environment variables listed here.

Env Var Value(s) What is does
GGCR_EXPERIMENT_ESTARGZ "1" When enabled this experiment will direct tarball.LayerFromOpener to emit estargz compatible layers, which enable them to be lazily loaded by an appropriately configured containerd.

v1.Image

Sources

Sinks

v1.ImageIndex

Sources

Sinks

v1.Layer

Sources

Sinks

Overview

mutate

The simplest use for these libraries is to read from one source and write to another.

For example,

  • crane pull is remote.Image -> tarball.Write,
  • crane push is tarball.Image -> remote.Write,
  • crane cp is remote.Image -> remote.Write.

However, often you actually want to change something about an image. This is the purpose of the mutate package, which exposes some commonly useful things to change about an image.

partial

If you're trying to use this library with a different source or sink than it already supports, it can be somewhat cumbersome. The Image and Layer interfaces are pretty wide, with a lot of redundant information. This is somewhat by design, because we want to expose this information as efficiently as possible where we can, but again it is a pain to implement yourself.

The purpose of the partial package is to make implementing a v1.Image much easier, by filling in all the derived accessors for you if you implement a minimal subset of v1.Image.

transport

You might think our abstractions are bad and you just want to authenticate and send requests to a registry.

This is the purpose of the transport and authn packages.

Tools

This repo hosts some tools built on top of the library.

crane

crane is a tool for interacting with remote images and registries.

gcrane

gcrane is a GCR-specific variant of crane that has richer output for the ls subcommand and some basic garbage collection support.

k8schain

k8schain implements the authentication semantics use by kubelets in a way that is easily consumable by this library.

k8schain is not a standalone tool, but it's linked here for visibility.

Emeritus: ko

This tool was originally developed in this repo but has since been moved to its own repo.

Comments
  • add FindManifests

    add FindManifests

    Add FindManifests utility, that allows one to pass a v1.ImageIndex and a match.Matcher, and get back the v1.Descriptor entries in the index that match.

    This is a spin-out from #823 and will include the salient original comments below.

    We expect to have several more such utilities that build on Matcher in this PR and/or others before it is done.

    opened by deitch 37
  • match package with matcher and utils to filter v1.Descriptors; v1.Platform.Equals utility

    match package with matcher and utils to filter v1.Descriptors; v1.Platform.Equals utility

    Starts to address #821

    • creates a ManifestMatcher and utilities that can be used to find manifests that match the provided matcher
    • provides convenience matcher utilities to match by annotation, media-types, name (specific annotation), platform
    • provides a FindManifests that takes a v1.ImageIndex and returns the manifests in it that match. This is useful in its own right, but will be even more useful later for tree walking.

    Some questions and follow-ons:

    • is this the right location for it? partial was suggested but seems, well, off. This is a utilities package, but mostly seems to have internal utils, although it still works better than partial. I think this could handle being renamed to util since v1/v1util is a mouthful.
    • while this is really simple, it doesn't provide a good option for depth. E.g. we might want in the future to be able to walk down and filter based on depth. I don't think that matters much now because (a) this is just the manifests, while the walker can handle depth; (b) this is simple and clean; (b) we haven't proven we have a real use case for infinite depth. Theoretically, and index can include an index can include (ad infinitum), but in practice we only ever see index->manifest or, in the case of v1.layout, index->index->manifest. That can be handled separately.

    Once this is good, we can start to add the next parts.

    cc @jonjohnsonjr

    opened by deitch 29
  • add download progress option for v1.tarball

    add download progress option for v1.tarball

    Initial implementation of download progress, per the discussion here with @jonjohnsonjr .

    This only does it for v1.tarball; it still will need to be added to legacy.tarball and v1.layout. I also have given it zero testing. The goal is to have a basis for discussion.

    I chose to change the calls to Write/MultiWrite/MultiWriteRef rather than entirely new funcs because:

    • it would create an over-proliferation of functions and be very messy
    • you always can pass in nil to a channel arg if you don't want it (which is what I did for almost all consumers
    • with go modules, versioning of consumers is easy, so breaking changes on a new commit is not a big deal.
    opened by deitch 28
  • Use hamilton instead of autorest for Azure Container Registry authentication

    Use hamilton instead of autorest for Azure Container Registry authentication

    Hi!

    Thank you for all the hard work you are doing. :1st_place_medal:

    Right now, autorest from Microsoft is used to provide authentication to Azure Container Registry. There's a much smaller project used by the folks over at Hashicorp (used by terraform, azurerm provider, azuread provider) that's much more ... Go. And smaller!

    I've been testing it out and it seems to be working. Will need a bit of polish but I thought I'd bring up the question here and see what you think about it.

    The project is called hamilton and can be found here: https://github.com/manicminer/hamilton

    My idea is that we would provide the Azure authentication and ACR token exchange in hamilton and add the helper to go-containerregistry (or a separate repository if that's preferred). I think it would be counter productive to remove autorest from the project to add a new one (github.com/docker/docker-credential-helpers/credentials) directly after.

    We would have to do the following to complete this task:

    • [ ] Move usage of autorest from hamilton to separate module: https://github.com/manicminer/hamilton/issues/138
    • [ ] Add Azure Container Registry token exchange support: https://github.com/manicminer/hamilton/issues/140
    • [ ] Decide where the helper should live (go-containerregistry or separate repository)
    • [x] Create the helper
    • [ ] Switch go-containerregistry from using https://github.com/chrismellard/docker-credential-acr-env to the new helper
    • [ ] Profit???

    What are your thoughts on this?

    question 
    opened by simongottschlag 22
  • v1/layout.WriteIndex ignores platform

    v1/layout.WriteIndex ignores platform

    We don't check the platform option when calling WriteIndex() see here:

    func (l Path) WriteIndex(ii v1.ImageIndex) error {
	// Always just write oci-layout file, since it's small.
	if err := l.WriteFile("oci-layout", []byte(layoutFile), os.ModePerm); err != nil {
		return err
	}

	h, err := ii.Digest()
	if err != nil {
		return err
	}

	indexFile := filepath.Join("blobs", h.Algorithm, h.Hex)
	return l.writeIndexToFile(indexFile, ii)

}

    This predates #815, as it was the same for writeIndex(). We do pass it to AppendIndex:

    func (l Path) AppendIndex(ii v1.ImageIndex, options ...Option) error {
	if err := l.WriteIndex(ii); err != nil {
		return err
	}

    where one Option is WithPlatform, but we don't actually pass it through to WriteIndex.

    Any objection to a PR to fix it?

    lifecycle/stale 
    opened by deitch 20
  • question: does crane copy support multiarch image built with docker buildx?

    question: does crane copy support multiarch image built with docker buildx?

    I would like to use crane to make the deployment to different registries easier. I have one gitlab repo, with one pipeline building a multiarch build to the Gitlab registry.

    docker buildx build --platform "linux/amd64,linux/arm64" --build-arg REGISTRY=$TARGET_REGISTRY -t $TARGET_REGISTRY_WITH_NAMESPACE/$IMAGE_NAME:$TARGET_IMAGE_TAG --push .

    and then, I would like to push this multarch image to AWS ECR registry (actually a job used twice with 2 different target registries).

    - crane auth login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
- aws ecr get-login-password --region $AWS_REGION | crane auth login --username AWS --password-stdin $TARGET_REGISTRY
- crane cp $CI_REGISTRY_IMAGE/$IMAGE_NAME:$IMAGE_TAG $TARGET_REGISTRY_WITH_NAMESPACE/$IMAGE_NAME:$TARGET_IMAGE_TAG

    the auth to the 2 registries (gitlab and aws) seems to be ok, but the crane cp always results with a 400:

    2022/03/15 15:28:40 failed to copy index: HEAD <aws-registry>/v2/<aws-namespace>/manifests/sha256:fwed95...: unsupported status code 400

    Any idea why this 400? Could be more a question for AWS support.

    But at the first place, it is supposed to work ? Does crane cp support copying a multi arch image?

    question lifecycle/stale 
    opened by ericbl 19
  • Add API for

    Add API for "podman" (service), similar to (docker) "daemon"

    Currently we have daemon.Image and daemon.Write

    We would also need podman.Image and podman.Write.

    https://github.com/kubernetes/minikube/pull/11063 pkg/minikube/image/podman.go

    It does not have any daemon, when not running remotely...

    Current root authentication is handled by running with sudo.

    docker | podman --- | --- docker save | sudo -n podman save docker tag | sudo -n podman tag docker load | sudo -n podman load

    lifecycle/stale 
    opened by afbjorklund 19
  • crane auth login fails when docker login works

    crane auth login fails when docker login works

    Steps to reproduce on macOS catalina bash. For what its worth I'm behind an enterprise proxy:

    $ echo "acctid.dkr.ecr.us-east-2.amazonaws.com" | crane auth get | jq .password | sed 's/"//g'

    long string password is output

    $ crane auth login acctid.dkr.ecr.us-east-2.amazonaws.com -u AWS -p stringfromabove $ crane pull acctid.dkr.ecr.us-east-2.amazonaws.com/reponame:tag tarball.tar

    Error: 2020/12/07 13:27:15 GET https://acctid.dkr.ecr.us-east-2.amazonaws.com/v2/repo-name/manifests/tag-name: unexpected status code 401 Unauthorized: Not Authorized

    $ docker login acctid.dkr.ecr.us-east-2.amazonaws.com -u AWS -p longstringfromabove warning about --password via cli followed by Login Succeeded docker pull succeeds

    Config.json:

    $ vim ~/.docker/config.json 
{
        "auths": {
                "acctid-that-I-want-to-pull-from.dkr.ecr.us-east-2.amazonaws.com": {},
                "different-actid.dkr.ecr.us-east-2.amazonaws.com": {},
                "jfrog.company.com": {},
                "jfrogdev.company.com": {}
        },
        "HttpHeaders": {
                "User-Agent": "Docker-Client/19.03.13 (darwin)"
        },
        "credsStore": "desktop",
        "experimental": "disabled",
        "stackOrchestrator": "swarm"
}

    I also tried adding a

    "credHelpers": {
                "acct-id.dkr.ecr.us-east-1.amazonaws.com": "ecr-login"
        },

    and setting the credsStore to ecr-login. But that didn't help my crane pull either. Running a docker pull with the ecr credHelpers set did work.

    My AWS Environment Variables are set, as is the certificate path location.

    opened by kumprj 19
  • how do I check if a layout contains all of the blobs?

    how do I check if a layout contains all of the blobs?

    Is it possible to somehow try to "resolve", or walk, my local v1 layout from an index that is there and see if all parts are there?

    Use case: let's say I have an image as follows. I am using part of docker.io/library/alpine:3.11

    9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb9a54 - index
|
|- 39eda93d15866957feaee28f8fc5adb545276a64147445c64992ef69804dbf01 - manifest linux/amd64
    |- f70734b6a266dcb5f44c383274821207885b549b75c8e119404917a61335981a - config
    |-cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 - layer0
|- ad295e950e71627e9d0d14cdc533f4031d42edae31ab57a841c5b9588eacc280 - manifest linux/arm64
    |- c20d2a9ab6869161e3ea6d8cb52d00be9adac2cc733d3fbc3955b9268bfd7fc5 - config
    |- 29e5d40040c18c692ed73df24511071725b74956ca1a61fe6056a651d86a13bd - layer0

    For the above, I have all of the parts from root index through manifests, configs, and layers for linux/amd64 and linux/arm64.

    If my local layout directory has some of those, but not linux/arm64 parts, then it might look like this:

    9a839e63dad54c3a6d1834e29692c8492d93f90c59c978c1ed79109ea4fb9a54 - index
|
|- 39eda93d15866957feaee28f8fc5adb545276a64147445c64992ef69804dbf01 - manifest linux/amd64
    |- f70734b6a266dcb5f44c383274821207885b549b75c8e119404917a61335981a - config
    |-cbdbe7a5bc2a134ca8ec91be58565ec07d037386d1f1d8385412d224deafca08 - layer0

    How do I check if all of the parts are there without going to docker.io? I can do the following:

    		desc, err := remote.Get("docker.io/library/alpine:3.11")
		ii, err := desc.ImageIndex()
		err = p.AppendIndex(ii)

    and it will go to docker.io, get the index, and then download any missing parts.

    However, it should be possible to somehow try to "resolve" my local index and see if all parts are there.

    As usual, happy to open a PR on it.

    opened by deitch 19
  • Calculate digest from tarball

    Calculate digest from tarball

    Our build system produces tarballs and K8s deployment configurations, later to be deployed to multiple environments. We would like to get the image's digest at build time so we could reference it in the configuration, rather than relying on a tag, which is not stable.

    Could Crane calculate the digest from a tarball? From what I gather, crane digest only works with a remote image.

    lifecycle/frozen 
    opened by aelij 18
  • Allow Reference also for daemon write

    Allow Reference also for daemon write

    I thought this was going to be a simple one-liner, like before:

    -func Write(tag name.Tag, img v1.Image) (string, error) {
+func Write(ref name.Reference, img v1.Image) (string, error) {

    But I was wrong, since I actually wanted to keep the tag too...

    Closes #702

    lifecycle/frozen 
    opened by afbjorklund 18
  • test: use `T.TempDir` to create temporary test directory

    test: use `T.TempDir` to create temporary test directory

    A testing cleanup.

    This pull request replaces os.MkdirTemp with t.TempDir. We can use the T.TempDir function from the testing package to create temporary directory. The directory created by T.TempDir is automatically removed when the test and all its subtests complete.

    This saves us at least 2 lines (error check, and cleanup) on every instance, or in some cases adds cleanup that we forgot.

    Reference: https://pkg.go.dev/testing#T.TempDir

    func TestFoo(t *testing.T) {
	// before
	tmpDir, err := os.MkdirTemp("", "")
	if err != nil {
		t.Fatal(err)
	}
	defer os.RemoveAll(tmpDir)

	// now
	tmpDir := t.TempDir()
}
    opened by Juneezee 0
  • question: [crane] How to use the crane for multi-platform images?

    question: [crane] How to use the crane for multi-platform images?

    Hi maintainers, How to use the crane for multi-platform images?

    I want to mutate some labels to multi-platform images(like amd64 and arm64). For the same architecture as the environment where the crane command is executed, I have confirmed that it is possible to set the labels, for example, as follows:

    $ crane mutate --label org.opencontainers.image.source=<source> <target image>

    How can I do the same mutating for images with different architectures?

    question 
    opened by bells17 0
  • ggcr: XDG_RUNTIME_DIR should be XDG_CONFIG_HOME

    ggcr: XDG_RUNTIME_DIR should be XDG_CONFIG_HOME

    Describe the bug

    When searching for 'containers/auth.json', in go-containerregistry/pkg/authn/keychain.go, the environment variable XDG_RUNTIME_DIR should be replaced with XDG_CONFIG_HOME and if missing with $HOME/.config.

    See: https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html

    So f, err := os.Open(filepath.Join(os.Getenv("XDG_RUNTIME_DIR"), "containers/auth.json"))

    Should be f, err := os.Open(filepath.Join(os.Getenv("XDG_CONFIG_HOME"), "containers/auth.json")) and f, err := os.Open(filepath.Join(os.Getenv("HOME"), ".config/containers/auth.json"))

    bug 
    opened by masarlabs 0
  • crane: feature request mutate --workdir

    crane: feature request mutate --workdir

    Describe the bug

    Currently, the mutate command has no option to set the working dir of the config.

    To Reproduce

    Expected behavior

    mutate should have a --workdir option as it is commonly used.

    Additional context

    Add any other context about the problem here.

    • Output of crane version
    • Registry used (e.g., GCR, ECR, Quay)
    bug 
    opened by thesayyn 0
  • crane: push should output ref to stdout

    crane: push should output ref to stdout

    Describe the bug

    Currently, in order to get digest ref to the pushed image, one has to use --image-refs option introduced in #1217

    There's already a TODO in the relevant code for this; https://github.com/google/go-containerregistry/blob/ffc14a072284ee850ebff7ec5e6ce963e82e07c0/cmd/crane/cmd/push.go#L76-L77

    To Reproduce

    Expected behavior

    crane push should output to stdout like

    Additional context

    Add any other context about the problem here.

    • Output of crane version
    • Registry used (e.g., GCR, ECR, Quay)
    bug 
    opened by thesayyn 0
  • crane: append -f should be variadic

    crane: append -f should be variadic

    Describe the bug

    Currently, crane append requires -f argument to be provided at least once which makes crane append unusable if you want to create empty image with no layers.

    To Reproduce

    crane append --oci-empty-base -t registry/empty_image:latest  -f <(tar -cf tarfilename.tar -T /dev/null))

    Expected behavior

    crane append should not fail when the -f argument is not present.

    Additional context

    Add any other context about the problem here.

    • Output of crane version
    • Registry used (e.g., GCR, ECR, Quay)
    bug 
    opened by thesayyn 1
Releases(v0.12.1)
Owner
Google
Google ❤️ Open Source
Google
GitHub
Libraries and CLIs for my personal all-in-one productivity system including components like bookmarks, notes, todos, projects, etc.

bntp.go Libraries and CLIs for my personal all-in-one productivity system including components like bookmarks, notes, todos, projects, etc. Neovim int

Jonas Mühlmann 13 Sep 13, 2022
top in container - Running the original top command in a container

Running the original top command in a container will not get information of the container, many metrics like uptime, users, load average, tasks, cpu, memory, are about the host in fact. topic(top in container) will retrieve those metrics from container instead, and shows the status of the container, not the host.

silenceshell 131 Dec 2, 2022
CLI tool and library for generating a Software Bill of Materials from container images and filesystems

A CLI tool and Go library for generating a Software Bill of Materials (SBOM) from container images and filesystems. Exceptional for vulnerability dete

Anchore, Inc. 3.5k Jan 6, 2023
espnwrapper to track real time scores for your fav match... in a terminal while working.

An espncricinfo wrapper written in go to track scores in real time and in the cmd/cli you can find the Command Line Interface wrapped over this wrapper.

Vishal Sharma 5 Mar 13, 2022
Command line tool makeing working with remote files esier.

Command line tool makeing working with remote files esier.

Techiecaro 2 Jul 27, 2022
Golang-based simple file server to serve files of the current working directory

fileserve Golang-based simple file server to serve static files of the current working directory File sharing in LAN or home network Web application t

Frank Hübner 2 Jan 9, 2022
A CLI tool for working with CloudWatch logs. It performs functions that I need at work.

CloudWatch Logs Utility A simple utility for working with CloudWatch Logs. AWS should probably build this themselves, but since they won't, I am here

J. Alexander Curtis 0 Dec 31, 2021
An os/exec like interface for running a command in a container, and being able to easily interact with stdin, stdout, and other adjustments

dockerexec An "os/exec" like interface for running a command in a container, and being able to easily interact with stdin, stdout, and other adjustmen

Segev Finer 2 Jul 14, 2022
Top-like interface for container metrics

Top-like interface for container metrics ctop provides a concise and condensed overview of real-time metrics for multiple containers: as well as a sin

bradley 13.8k Jan 9, 2023
wy : a set of command-line tools to test your container-based platform

wy wy (Abbreviation of Would You) is a set of command-line tools to test your container-based platform. ToC: Commands Deployment Monitoring Contributi

Yusuke Kuoka 2 Apr 30, 2022
Option container for golang

Option Option provides an Option container which can be used to force some addit

Nik Polakov 4 Dec 21, 2021
A tiny Go library + client for downloading Youtube videos. The library is capable of fetching Youtube video metadata, in addition to downloading videos.

A tiny Go library + client (command line Youtube video downloader) for downloading Youtube videos. The library is capable of fetching Youtube video metadata, in addition to downloading videos. If ffmpeg is available, client can extract MP3 audio from downloaded video files.

Kunal Diwan 3 Oct 14, 2022
A Go library and common interface for running local and remote commands

go-runcmd go-runcmd is a Go library and common interface for running local and remote commands providing the Runner interface which helps to abstract

AUCloud 1 Nov 25, 2021
A Simple and Clear CLI library. Dependency free.

A Simple and Clear CLI library. Dependency free. Features Nested Subcommands Uses the standard library flag package Auto-generated help Custom banners

Lea Anthony 148 Jan 1, 2023
Golang library with POSIX-compliant command-line UI (CLI) and Hierarchical-configuration. Better substitute for stdlib flag.

cmdr cmdr is a POSIX-compliant, command-line UI (CLI) library in Golang. It is a getopt-like parser of command-line options, be compatible with the ge

hz 116 Oct 28, 2022
A command-line tool and library for generating regular expressions from user-provided test cases

Table of Contents What does this tool do? Do I still need to learn to write regexes then? Current features How to install? 4.1 The command-line tool 4

Peter M. Stahl 5.8k Jan 9, 2023
A golang library for building interactive prompts with full support for windows and posix terminals.

Survey A library for building interactive prompts on terminals supporting ANSI escape sequences. package main import ( "fmt" "github.com/Alec

Alec Aivazis 3.5k Jan 6, 2023
Small, fast library to create ANSI colored strings and codes. [go, golang]

ansi Package ansi is a small, fast library to create ANSI colored strings and codes. Install Get it go get -u github.com/mgutz/ansi Example import "gi

Mario Gutierrez 380 Dec 23, 2022
Integrated console application library, using Go structs as commands, with menus, completions, hints, history, Vim mode, $EDITOR usage, and more ...

Gonsole - Integrated Console Application library This package rests on a readline console library, (giving advanced completion, hint, input and histor

null 18 Nov 20, 2022