Fast docker image distribution plugin for containerd, based on CRFS/stargz

Overview

[ ⬇️ Download] [ 📔 Browse images] [ Quick Start (Kubernetes)] [ 🤓 Quick Start (nerdctl)]

Stargz Snapshotter

Tests Status Benchmarking Nightly

Read also introductory blog: Startup Containers in Lightning Speed with Lazy Image Distribution on Containerd

Pulling image is one of the time-consuming steps in the container lifecycle. Research shows that time to take for pull operation accounts for 76% of container startup time[FAST '16]. Stargz Snapshotter is an implementation of snapshotter which aims to solve this problem by lazy pulling. Lazy pulling here means a container can run without waiting for the pull completion of the image and necessary chunks of the image are fetched on-demand.

eStargz is a lazily-pullable image format proposed by this project. This is compatible to OCI/Docker images so this can be pushed to standard container registries (e.g. ghcr.io) as well as this is still runnable even on eStargz-agnostic runtimes including Docker. eStargz format is based on stargz image format by CRFS but comes with additional features like runtime optimization and content verification.

The following histogram is the benchmarking result for startup time of several containers measured on Github Actions, using GitHub Container Registry.

The benchmarking result on ecdb227

legacy shows the startup performance when we use containerd's default snapshotter (overlayfs) with images copied from docker.io/library without optimization. For this configuration, containerd pulls entire image contents and pull operation takes accordingly. When we use stargz snapshotter with eStargz-converted images but without any optimization (estargz-noopt) we are seeing performance improvement on the pull operation because containerd can start the container without waiting for the pull completion and fetch necessary chunks of the image on-demand. But at the same time, we see the performance drawback for run operation because each access to files takes extra time for fetching them from the registry. When we use eStargz with optimization (estargz), we can mitigate the performance drawback observed in estargz-noopt images. This is because stargz snapshotter prefetches and caches likely accessed files during running the container. On the first container creation, stargz snapshotter waits for the prefetch completion so create sometimes takes longer than other types of image. But it's still shorter than waiting for downloading all files of all layers.

The above histogram is the benchmarking result on the commit ecdb227. We are constantly measuring the performance of this snapshotter so you can get the latest one through the badge shown top of this doc. Please note that we sometimes see dispersion among the results because of the NW condition on the internet and the location of the instance in the Github Actions, etc. Our benchmarking method is based on HelloBench.

Stargz Snapshotter is a non-core sub-project of containerd.

Quick Start with Kubernetes

For using stargz snapshotter on kubernetes nodes, you need the following configuration to containerd as well as run stargz snapshotter daemon on the node. We assume that you are using containerd (> v1.4.2) as a CRI runtime.

version = 2

# Plug stargz snapshotter into containerd
# Containerd recognizes stargz snapshotter through specified socket address.
# The specified address below is the default which stargz snapshotter listen to.
[proxy_plugins]
  [proxy_plugins.stargz]
    type = "snapshot"
    address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"

# Use stargz snapshotter through CRI
[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "stargz"
  disable_snapshot_annotations = false

Note that disable_snapshot_annotations = false is required since containerd > v1.4.2

This repo contains a Dockerfile as a KinD node image which includes the above configuration. You can use it with KinD like the following,

$ docker build -t stargz-kind-node https://github.com/containerd/stargz-snapshotter.git
$ kind create cluster --name stargz-demo --image stargz-kind-node

Then you can create eStargz pods on the cluster. In this example, we create a stargz-converted Node.js pod (ghcr.io/stargz-containers/node:13.13.0-esgz) as a demo.

apiVersion: v1
kind: Pod
metadata:
  name: nodejs
spec:
  containers:
  - name: nodejs-stargz
    image: ghcr.io/stargz-containers/node:13.13.0-esgz
    command: ["node"]
    args:
    - -e
    - var http = require('http');
      http.createServer(function(req, res) {
        res.writeHead(200);
        res.end('Hello World!\n');
      }).listen(80);
    ports:
    - containerPort: 80

The following command lazily pulls ghcr.io/stargz-containers/node:13.13.0-esgz from Github Container Registry and creates the pod so the time to take for it is shorter than the original image library/node:13.13.

$ kubectl --context kind-stargz-demo apply -f stargz-pod.yaml && kubectl get po nodejs -w
$ kubectl --context kind-stargz-demo port-forward nodejs 8080:80 &
$ curl 127.0.0.1:8080
Hello World!

Stargz snapshotter also supports further configuration including private registry authentication, mirror registries, etc.

Creating eStargz images with optimization

For lazy pulling images, you need to prepare eStargz images first. You can use ctr-remote command for do this. You can also try our pre-converted images listed in Trying pre-converted images.

In this section, we introduce ctr-remote command for converting images into eStargz with optimization for reading files. As shown in the above benchmarking result, on-demand lazy pulling improves the performance of pull but causes runtime performance penalty because reading files induce remotely downloading contents. For solving this, ctr-remote has workload-based optimization for images.

For trying the examples described in this section, you can also use the docker-compose-based demo environment. You can setup this environment as the following commands (put this repo on ${GOPATH}/src/github.com/containerd/stargz-snapshotter). Note that this runs privileged containers on your host.

$ cd ${GOPATH}/src/github.com/containerd/stargz-snapshotter/script/demo
$ docker-compose build containerd_demo
$ docker-compose up -d
$ docker exec -it containerd_demo /bin/bash
(inside container) # ./script/demo/run.sh

Generally, container images are built with purpose and the workloads are defined in the Dockerfile with some parameters (e.g. entrypoint, envvars and user). By default, ctr-remote optimizes the performance of reading files that are most likely accessed in the workload defined in the Dockerfile. You can also specify the custom workload using options if needed.

The following example converts the legacy library/ubuntu:18.04 image into eStargz. The command also optimizes the image for the workload of executing ls on /bin/bash. The thing actually done is it runs the specified workload in a temporary container and profiles all file accesses with marking them as likely accessed also during runtime. The converted image is still docker-compatible so you can run it with eStargz-agnostic runtimes (e.g. Docker).

# ctr-remote image pull docker.io/library/ubuntu:18.04
# ctr-remote image optimize --entrypoint='[ "/bin/bash", "-c" ]' --args='[ "ls" ]' docker.io/library/ubuntu:18.04 registry2:5000/ubuntu:18.04
# ctr-remote image push --plain-http registry2:5000/ubuntu:18.04

Finally, the following commands clear the local cache then pull the eStargz image lazily. Stargz snapshotter prefetches files that are most likely accessed in the optimized workload, which hopefully increases the cache hit rate for that workload and mitigates runtime overheads as shown in the benchmarking result shown top of this doc.

# ctr-remote image rm --sync registry2:5000/ubuntu:18.04
# ctr-remote images rpull --plain-http registry2:5000/ubuntu:18.04
fetching sha256:728332a6... application/vnd.docker.distribution.manifest.v2+json
fetching sha256:80026893... application/vnd.docker.container.image.v1+json
# ctr-remote run --rm -t --snapshotter=stargz registry2:5000/ubuntu:18.04 test /bin/bash
[email protected]:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Importing Stargz Snapshotter as go module

Currently, Stargz Snapshotter repository contains two Go modules as the following and both of them need to be imported.

  • github.com/containerd/stargz-snapshotter
  • github.com/containerd/stargz-snapshotter/estargz

Please make sure you import the both of them and they point to the same commit version.

Project details

Stargz Snapshotter is a containerd non-core sub-project, licensed under the Apache 2.0 license. As a containerd non-core sub-project, you will find the:

information in our containerd/project repository.

Issues
  • snapshotter doesn't like paths in ko images

    snapshotter doesn't like paths in ko images

    In github.com/google/ko built images, the paths we use within the tarball seem to be causing problems for the estargz snapshotter. These images run fine against a standard containerd, so it would be good to harden this snapshotter to support them:

    Dec 20 21:36:45 ubuntu containerd-stargz-grpc[1179]: {"error":"failed to cache prefetched layer: invalid child path \"/ko-app\"; must be child of \"\"","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/28/fs","msg":"failed to prefetched layer","time":"2020-12-20T21:36:45.919228015Z"}
    ...
    Dec 20 21:36:52 ubuntu containerd-stargz-grpc[1179]: {"error":"invalid child path \"/var\"; must be child of \"\"","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/27/fs","msg":"failed to fetch whole layer","time":"2020-12-20T21:36:52.422678402Z"}
    

    You can produce an image with ko + estargz by:

    1. clone github.com/google/ko
    2. go install ./cmd/ko
    3. GGCR_EXPERIMENT_ESTARGZ=1 KO_DOCKER_REPO=docker.io/{username} ko publish -B ./cmd.foo
    4. kubectl run foo --image={digest from above}
    opened by mattmoor 14
  • no state directory

    no state directory

    I am concerned about how much data has been pull to local. According to https://github.com/containerd/stargz-snapshotter/blob/main/docs/overview.md , it should have a state directory under container's root path. But I can't find any: command: ctr-remote images rpull --snapshotter=stargz iregistry.harbo.com/research/wordpress-estargz:latest ctr-remote run --rm -t --snapshotter=stargz iregistry.harbo.com/research/wordpress-estargz:latest test /bin/bash cat /.stargz-snapshotter/* output: cat: '/.stargz-snapshotter/*': No such file or directory

    My environment has install containerd & stargz-snappshotter & fuse successfully, and lazy pull has been observed, the only problem is that there is no state directory.

    Any help will be appreciated !

    needs-more-info 
    opened by fannaihao 13
  • Dockerfile: bump up components (Kubernetes 1.20, ...)

    Dockerfile: bump up components (Kubernetes 1.20, ...)

    ~- Go: 1.13 -> 1.15 (incurs s/go install/go get/g)~

    • containerd: 1.4.2 -> 1.4.3
    • CNI plugins: 0.8.6 -> 0.9.0
    • kind node: 1.19.0 -> 1.20.0
    opened by AkihiroSuda 11
  • Store filesystem metadata on disk

    Store filesystem metadata on disk

    Currently, stargz snapshotter holds filesystem metadata in memory but this ends up consuming a large amount of memory and they aren't free until the filesystem layer is unmounted. This commit tries to solve this problem by fixing snapshotter to store the metadata on disk using bbolt.

    • TODO
      • [x] resolve conflict against main version (add metrics support)

    Performance comparison

    Summary:

    • PR improves memory consumption (about 5x smaller)
    • PR increases *node.Getattr latency (about 2x ~ 3x slower) but actual workloads (HelloBench, stat commands) don't report significant performance degradation
      • Welcome for any suggestions about benchmarks where this change will be problematic

    Memory consumption

    • host memory: ~16GB
    • command: nerdctl --snapshotter=stargz --insecure-registry run -it --rm registry2:5000/kdeneon/plasma:unstable-esgz echo hello
      • registry2 runs as a container, on the same host of contianerd.
      • size of registry2:5000/kdeneon/plasma:unstable-esgz is 1.1 GiB according to containerd.
    • config
      metrics_address = "127.0.0.1:8234"
      disable_verification = false
      [[resolver.host."registry2:5000".mirrors]]
      host = "registry2:5000"
      insecure = true
      [directory_cache]
      direct = true
      

    This change makes memory consumption 5x smaller.

    |main|PR| |---|---| |main-mem02|entriesdb-mem02|

    latency of *node.Getattr operation

    • image: ghcr.io/stargz-containers/postgres:13.1-esgz
    • command: time nerdctl exec -it $NAME find /usr/ -type d -exec /bin/bash -c "stat {}/*" \;
    • config: same as the above

    This change makes *node.Getattr operation about 2x ~ 3x slower. Though this doesn't seems to cause large performance degradation in actual workloads (HelloBench and stat commands) as shown in the following sections, we should continuously seek the way to optimize the performance of reading metadata.

    |main|PR| |---|---| |main-getattr01|entriesdb-getattr01|

    Total time for find /usr/ -type d -exec /bin/bash -c "stat {}/*" \;: (Reading metadata doesn't seem to be a bottleneck of this command?)

    |main|PR| |---|---| |10.209s|10.112s|

    HelloBench

    https://github.com/ktock/stargz-snapshotter/runs/3351635146

    result

    opened by ktock 10
  • Enable to run containers on IPFS

    Enable to run containers on IPFS

    See https://github.com/containerd/stargz-snapshotter/blob/main/docs/ipfs.md for the latest spec.

    This commit enables to run of containers on IPFS.

    OCI image is extended in an OCI-compatible way. Stargz Snapshotter mounts the image from IPFS to the container's rootfs with lazy pulling support.

    The image must have the following OCI-compatible extension. This constructs a CID-based DAG of blobs in an OCI image.

    • Each descriptor in an image must have the following annotation
      • key: containerd.io/snapshot/remote/ipfs/cid
      • value: CID of the blob that the descriptor points to

    This commit adds ipfs library which includes the basic functionality to make containerd aware of containers on IPFS. These components are eStargz-agnostic and can be used for running non-eStargz images without lazy pulling (e.g. on overlayfs snapshotter).

    • ipfs.IndexConvertFunc provides containerd's converter.ConvertFunc implementation which converts an image to the IPFS-enabled image format as described above. This also adds the image contents to IPFS.
    • ipfs.NewResolver provides containerd's remote.Resovler implementation for IPFS. If a descriptor contains the CID annotation, it fetches the pointed content from IPFS.

    The following component enables lazy pulling of containers from IPFS.

    • (fs/remote/ipfs).Reader provides the way to read a range of a file on IPFS. This enables stargz snapshotter to mount the container's rootfs from IPFS with lazy pulling.

    Examples

    Storing image to IPFS and run with stargz snapshotter:

    # ipfs daemon
    # ctr-remote i pull ghcr.io/stargz-containers/python:3.9-org
    # ctr-remote i ipfs-add ghcr.io/stargz-containers/python:3.9-org test
    INFO[0098] Pushed                                        CID=QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV
    # ipfs cat QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV | jq
    {
      "mediaType": "application/vnd.oci.image.index.v1+json",
      "digest": "sha256:d8e862a13071692edbf4cffc3954591ca98954571615a0698a7ed59a09dc04df",
      "size": 342,
      "annotations": {
        "containerd.io/snapshot/remote/ipfs/cid": "QmSUi34RNpoxY4zqZFGZnCQcH6odXBRHT6VTGAduANwTj6"
      }
    }
    # # clear containerd cache here
    # time ( ctr-remote i rpull --ipfs QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV && \
      ctr-remote run --snapshotter=stargz --rm -t QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV foo python -c 'print("Hello, World!")' )
    fetching sha256:6fd287cf... application/vnd.oci.image.index.v1+json
    fetching sha256:5c13568a... application/vnd.oci.image.manifest.v1+json
    fetching sha256:236b4bd7... application/vnd.oci.image.config.v1+json
    Hello, World!
    
    real	0m1.609s
    user	0m0.047s
    sys	0m0.031s
    

    The container can also run with overlayfs snapshotter without lazy pulling but it's slower than stargz snapshotter:

    # time ( ctr-remote i rpull --snapshotter=overlayfs --ipfs QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV && \
      ctr-remote run --snapshotter=overlayfs --rm -t QmfTVLXMG9TH7X523NytcXj35XtEDx4wgNWYepVumqpJZV foo python -c 'print("Hello, World!")' )
    fetching sha256:d8e862a1... application/vnd.oci.image.index.v1+json
    fetching sha256:b8df0fee... application/vnd.oci.image.manifest.v1+json
    fetching sha256:236b4bd7... application/vnd.oci.image.config.v1+json
    fetching sha256:94584b60... application/vnd.oci.image.layer.v1.tar+gzip
    fetching sha256:cecf6f1a... application/vnd.oci.image.layer.v1.tar+gzip
    fetching sha256:cfd1c7a0... application/vnd.oci.image.layer.v1.tar+gzip
    fetching sha256:9c3d8238... application/vnd.oci.image.layer.v1.tar+gzip
    fetching sha256:cce4a0fe... application/vnd.oci.image.layer.v1.tar+gzip
    fetching sha256:3467b44a... application/vnd.oci.image.layer.v1.tar+gzip
    fetching sha256:8ff4c537... application/vnd.oci.image.layer.v1.tar+gzip
    fetching sha256:b60d5d60... application/vnd.oci.image.layer.v1.tar+gzip
    fetching sha256:68277180... application/vnd.oci.image.layer.v1.tar+gzip
    Hello, World!
    
    real	0m11.655s
    user	0m0.571s
    sys	0m0.309s
    
    opened by ktock 8
  • How to inspect state directory to monitor on-demand filesystem errors

    How to inspect state directory to monitor on-demand filesystem errors

    Hi All,

    The documentation here mentions the presence of hidden state directory at the root of the filesystem. Unfortunately I do not see any such directory at the root file system path.

    [[email protected] stargz-snapshotter]$ ls /.stargz-snapshotter/*
    ls: cannot access /.stargz-snapshotter/*: No such file or directory
    

    Can someone please help me with the steps to locate the state directory. It will be super useful to monitor any on-demand filesystem errors by inspecting state directory contents.

    opened by iamsumee 8
  • Use single range request for registries which don't support multi range request

    Use single range request for registries which don't support multi range request

    Some cloud-provided registries including GCR doesn't support multi range request (e.g. range: bytes=0-3,5-9,16-20) For example, when we pull an image from GCR,

    # ctr-remote image rpull gcr.io/stargz-275102/ubuntu:18.04
    fetching sha256:36d56a39... application/vnd.docker.distribution.manifest.v2+json
    fetching sha256:b993a6c1... application/vnd.docker.container.image.v1+json
    fetching sha256:ce77d9a4... application/vnd.docker.image.rootfs.diff.tar.gzip
    fetching sha256:cc65679c... application/vnd.docker.image.rootfs.diff.tar.gzip
    fetching sha256:dcc819e8... application/vnd.docker.image.rootfs.diff.tar.gzip
    fetching sha256:1b677c82... application/vnd.docker.image.rootfs.diff.tar.gzip
    

    We get 400 response because of multi range request.

    DEBU[2020-04-23T02:27:15.607869311Z] failed to read layer                          digest="sha256:cc65679ceb37dd5d861fe76ced27bbca8897d3c73f9e5a4422491c5932837804" error="failed to parse stargz: error reading footer: unexpected status code on \"https://storage.googleapis.com/artifacts.stargz-275102.appspot.com/containers/images/sha256:cc65679ceb37dd5d861fe76ced27bbca8897d3c73f9e5a4422491c5932837804\": 400 Bad Request" mountpoint=/var/lib/containerd-stargz-grpc/snapshotter/snapshots/1/fs ref="gcr.io/stargz-275102/ubuntu:18.04"
    

    This commit solves this issue by using a single request instead. We might need tests with cloud-provided registries. I'll add them with another PR.

    opened by ktock 8
  • [WIP] Buildkit integration

    [WIP] Buildkit integration

    The re-opened version of #52. Thread on buildkit is moby/buildkit#1396.

    This is an experimental integration with buildkit for speeding up fetching base image.

    Our patched version of buildkit is here.

    This commit includes benchmark scripts to measure the time for buliding sample images. See this doc for manual testing.

    Though it seems good for the lazy distribution of the base images, we currently have two problems for exporting output image.

    1. Archiving layers takes a long time. It is because of the low READ performance including fetching contents from the registry.
    2. We only support exporting type tar. It is because when we use remote snapshotters, containerd doesn't store the image contents in the content store but buildkit needs to use these contents for exporting images.

    cc @AkihiroSuda @tonistiigi

    opened by ktock 7
  • [WIP] buildkit integration

    [WIP] buildkit integration

    This is an experimental integration with buildkit for speeding up fetching base image.

    Our patched version of buildkit is here.

    This commit includes benchmark scripts to measure the time for buliding sample images. See this doc for manual testing.

    Though it seems good for the lazy distribution of the base images, we currently have two problems for exporting output image.

    1. Archiving layers takes a long time. It is because of the low READ performance including fetching contents from the registry.
    2. We only support exporting type tar. It is because when we use remote snapshotters, containerd doesn't store the image contents in the content store but buildkit needs to use these contents for exporting images.
    opened by ktock 7
  • Use of `--reuse` option during `ctr-remote i optimize`

    Use of `--reuse` option during `ctr-remote i optimize`

    Hello,

    I'm bit confused about the reuse flag available when optimizing images. It allows the tool to skip over already existing eStargz layers in the image. I'm curious what the use case is? Authoring new images using an eStargz image as base image (docker and podman strip away annotations when pulling eStargz images, so eStargz images are still converted)?

    Thanks.

    opened by rdpsin 6
  • Error message: unable to start container process: exec:

    Error message: unable to start container process: exec: "bash": executable file not found in $PATH: unknown

    Problem Description

    I tried to launch containerd in a default network namespace, and start stargz-snapshotter daemon in a different network namespace "stargz". And pull the image and run the container using stargz-snapshotter inside the "stargz" network namespace, and an error was returned.

    ctr: failed to create shim: OCI runtime create failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown
    
    1. The network namespace is created as sudo ip netns add stargz, and configured the network insider the "stargz" network namespace.
    2. Then started the containerd-stargz-grpc inside the "stargz" network namespace as following
    sudo ip netns exec stargz /usr/local/bin/containerd-stargz-grpc --log-level=debug 
    
    1. Pull image and run container as shown in the below section: What I have tried but does not work

    I have looked into many resolved issues and discussion questions in containerd community, but none of them really helps me to solve my problem. And I think it is more related to the snapshotter not the containerd, so I decide to post the question here. I apologize if it is not a good place to post it.

    What I have tried but does not work ?

    1. Followed stargz-snapshotter-INSTALL.md to start the containerd, with exactly the same containerd config.
    2. Followed stargz-snapshotter-INSTALL.md to download stargz-snapshotter binaries, with one step different, instead of using systemctl, I just manually start stargz-snapshotter

    Notice: All pulling images with rpull were succeeded, but couldn't run the container All images come from https://github.com/containerd/stargz-snapshotter/blob/main/docs/pre-converted-images.md

    Pull image & run container inside the "stargz" network namespace
    
    $sudo ip netns exec stargz /usr/local/bin/ctr-remote image rpull ghcr.io/stargz-containers/golang:1.12.9-esgz
    fetching sha256:56a508b1... application/vnd.oci.image.index.v1+json
    fetching sha256:4cd4fe3f... application/vnd.oci.image.manifest.v1+json
    fetching sha256:b04dc2ba... application/vnd.oci.image.config.v1+json
    $sudo ip netns exec stargz /usr/local/bin/ctr-remote run --rm -t --snapshotter=stargz ghcr.io/stargz-containers/golang:1.12.9-esgz test
    ctr: failed to create shim: OCI runtime create failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown
    
    ====================================================================================
    Pull image & run container inside the default network namespace
    
    $sudo ctr-remote image rpull ghcr.io/stargz-containers/golang:1.12.9-esgz
    $sudo ctr-remote run --snapshotter=stargz --rm -t ghcr.io/stargz-containers/golang:1.12.9-esgz test
    ctr: failed to create shim: OCI runtime create failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown
    

    What I have tried work ?

    I can successfully use rpull to pull the image and run container without the network namespace as followed

    1. Run the stargz daemon
    sudo /usr/local/bin/containerd-stargz-grpc --log-level=debug
    
    1. pull the estargz image and run a container
    $ sudo ctr-remote image rpull ghcr.io/stargz-containers/golang:1.12.9-esgz
    fetching sha256:56a508b1... application/vnd.oci.image.index.v1+json
    fetching sha256:4cd4fe3f... application/vnd.oci.image.manifest.v1+json
    fetching sha256:b04dc2ba... application/vnd.oci.image.config.v1+json
    $ sudo ctr-remote run --snapshotter=stargz --rm -t ghcr.io/stargz-containers/golang:1.12.9-esgz test
    [email protected]:/go# exit
    

    I can also run the container with regular image(non-stargz image) inside the 'stargz' network namespace. So only stargz image can not be run inside the 'stargz' network namespace and return the above error.

    [[email protected] stargz-snapshotter]$ sudo ctr-remote image rpull ghcr.io/stargz-containers/golang:1.12.9-org
    fetching sha256:4d75d32f... application/vnd.docker.distribution.manifest.list.v2+json
    fetching sha256:328736f6... application/vnd.docker.distribution.manifest.v2+json
    fetching sha256:66922093... application/vnd.docker.container.image.v1+json
    fetching sha256:fa06ef98... application/vnd.docker.image.rootfs.diff.tar.gzip
    fetching sha256:14408c8d... application/vnd.docker.image.rootfs.diff.tar.gzip
    fetching sha256:a2a2197e... application/vnd.docker.image.rootfs.diff.tar.gzip
    fetching sha256:ea67eaa7... application/vnd.docker.image.rootfs.diff.tar.gzip
    fetching sha256:4a56a430... application/vnd.docker.image.rootfs.diff.tar.gzip
    fetching sha256:4b5cacb6... application/vnd.docker.image.rootfs.diff.tar.gzip
    fetching sha256:5915f3af... application/vnd.docker.image.rootfs.diff.tar.gzip
    
    [[email protected] stargz-snapshotter]$ sudo ctr-remote run --snapshotter=stargz --rm -t ghcr.io/stargz-containers/golang:1.12.9-org test
    [email protected]:/go# 
    

    Config

    $ echo $PATH
    /usr/local/go/bin:/home/ec2-user/go/bin:/home/ec2-user/go/src/github.com/aws/two/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
    
    $ /usr/local/bin/containerd --version
    containerd github.com/containerd/containerd v1.5.4 69107e47a62e1d690afa2b9b1d43f8ece3ff4483
    
    $ cat /etc/containerd/config.toml 
    version = 2
    
    # Plug stargz snapshotter into containerd
    [proxy_plugins]
      [proxy_plugins.stargz]
        type = "snapshot"
        address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"
    

    Error Message

    ctr: failed to create shim: OCI runtime create failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown
    

    In addition, I tried solutions from the issues in the containerd community includes https://github.com/containerd/containerd/issues/3597 . Unfortunately, all of them didn't help solve the problem.

    Much appreciated if anyone has any thought on it. Thank you for your help!

    opened by RoyceDavison 6
  • Bump github.com/urfave/cli from 1.22.5 to 1.22.9 in /cmd

    Bump github.com/urfave/cli from 1.22.5 to 1.22.9 in /cmd

    Bumps github.com/urfave/cli from 1.22.5 to 1.22.9.

    Release notes

    Sourced from github.com/urfave/cli's releases.

    v1.22.9

    What's Changed

    Full Changelog: https://github.com/urfave/cli/compare/v1.22.8...v1.22.9

    v1.22.8

    What's Changed

    Full Changelog: https://github.com/urfave/cli/compare/v1.22.7...v1.22.8

    Release 1.22.7

    What's Changed

    Full Changelog: https://github.com/urfave/cli/compare/v1.22.6...v1.22.7

    Release 1.22.6

    What's Changed

    Full Changelog: https://github.com/urfave/cli/compare/v1.22.5...v1.22.6

    Commits
    • 575b8b4 Merge pull request #1383 from kolyshkin/v1-no-docs
    • 2930925 ci: test newly added tag
    • fc47b1a Add urfave_cli_no_docs build tag
    • ba801d7 Move some test helpers from docs_test to fish_test
    • 9810d12 Merge pull request #1384 from kolyshkin/v1-do-fix-ci
    • 22281d3 Fix CI
    • b370d04 Really fix TestApp_RunAsSubCommandIncorrectUsage
    • 936bd2e Merge pull request #1379 from kolyshkin/v1-fix-ci
    • 6564c0f ci: add go 1.17.x and 1.18.x, fix failing test
    • 1b4a05e Merge pull request #1362 from kolyshkin/v1-bad-open
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 0
  • Bump github.com/ipfs/go-cid from 0.1.0 to 0.2.0 in /ipfs

    Bump github.com/ipfs/go-cid from 0.1.0 to 0.2.0 in /ipfs

    Bumps github.com/ipfs/go-cid from 0.1.0 to 0.2.0.

    Release notes

    Sourced from github.com/ipfs/go-cid's releases.

    v0.2.0

    What's Changed

    New Contributors

    Full Changelog: https://github.com/ipfs/go-cid/compare/v0.1.0...v0.2.0

    Commits
    • b2064d7 fix: remove invalid multicodec2string mappings (#137)
    • ddd9ef7 update .github/workflows/go-check.yml
    • 6be8b63 update .github/workflows/go-test.yml
    • 395d392 update .github/workflows/automerge.yml
    • 1d11990 bump go.mod to Go 1.17 and run go fix
    • dc3bb41 sync: update CI config files (#136)
    • 28f4a5e Benchmark existing ways to check for IDENTITY CIDs
    • 5640b01 avoid double alloc in NewCidV1
    • cf76220 Merge pull request #131 from ipfs/web3-bot/sync
    • 9e2855d update .github/workflows/go-check.yml
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 0
  • 3 error(s) occurred  gzip: invalid header* legacy: failed to get footer gzip reader: gzip invalid header

    3 error(s) occurred gzip: invalid header* legacy: failed to get footer gzip reader: gzip invalid header

    when use stargz snapshotter to pull private images on Kubernetes , it don't work, and just pull all layers. i use kubectl create secret regcred, then use CRI-based authentication according to https://github.com/containerd/stargz-snapshotter/blob/main/docs/overview.md test_pol.yaml: the registry.baidubce.com/stargz-work/wordpress-stargz-opt:latest is stagz image image

    /etc/containerd/conf.toml image

    /etc/containerd-starhz-grpc/config.toml image

    systemctl status stargz-snapshotter.service

    image

    opened by yueyuep 1
  • Add CI to automate building and pushing `estargz-kind-node` and pre-converted estargz images

    Add CI to automate building and pushing `estargz-kind-node` and pre-converted estargz images

    opened by ktock 1
  • Support non-registry mount sources

    Support non-registry mount sources

    Stargz snapshotter should support more mount sources other than registry. They include the following (discussed at https://github.com/moby/buildkit/pull/2685#issuecomment-1058888215).

    • local OCI directory
    • BuildKit Github Actions cache
    enhancement 
    opened by ktock 0
Releases(v0.11.4)
A Rancher and Kubernetes optimized immutable Linux distribution based on openSUSE

RancherOS v2 WORK IN PROGRESS RancherOS v2 is an immutable Linux distribution built to run Rancher and it's corresponding Kubernetes distributions RKE

Rancher 68 Apr 20, 2022
Repositório para a aula sobre integração do containerd com Golang

Integrando containers nativamente usando Golang Este é o código finalizado da aplicação Já pensou em uma alternativa ao Docker? Que tal manipular cont

Rocketseat Experts Club 3 May 4, 2021
Installs containerd on Windows, optionally with default CNI plugins

containerd-installer Installs containerd on Windows, optionally with default CNI plugins Usage NAME: containerd-installer.exe - Install containerd

Markus Lippert 7 Apr 15, 2022
Nydus-snapshotter - A containerd snapshotter with capability of on-demand read

Nydus Snapshotter Nydus-snapshotter is a non-core sub-project of containerd. Pul

containerd 27 May 19, 2022
Runwasi - A containerd shim which runs wasm workloads in wasmtime

containerd-shim-wasmtime-v1 This is a containerd shim which runs wasm workloads

Brian Goff 100 May 20, 2022
Fast, Docker-ready image processing server written in Go and libvips, with Thumbor URL syntax

Imagor Imagor is a fast, Docker-ready image processing server written in Go. Imagor uses one of the most efficient image processing library libvips (w

Adrian Shum 2.3k May 15, 2022
resource manifest distribution among multiple clusters.

Providing content to managed clusters Support a primitive that enables resources to be applied to a managed cluster. Community, discussion, contributi

Open Cluster Management 19 May 18, 2022
Truly Minimal Linux Distribution for Containers

Statesman Statesman is a minimal Linux distribution, running from memory, that has just enough functionality to run OCI-compatible containers. Rationa

James Cunningham 3 Nov 12, 2021
Walker's alias method is an efficient algorithm to sample from a discrete probability distribution.

walker-alias Walker's alias method is an efficient algorithm to sample from a discrete probability distribution. This means given an arbitrary probabi

null 2 Mar 7, 2022
A Kubernetes operator that allows for automatic provisioning and distribution of cert-manager certs across namespaces

cached-certificate-operator CachedCertificate Workflow When a CachedCertificate is created or updated the operator does the following: Check for a val

Weave Development Lab 2 Apr 27, 2022
Apachedist-resource - A concourse resource to track updates of an apache distribution, e.g. tomcat

Apache Distribution Resource A concourse resource that can track information abo

Gareth Evans 0 Feb 2, 2022
RancherOS v2 is an immutable Linux distribution built to run Rancher and it's corresponding Kubernetes distributions RKE2 and k3s

RancherOS v2 is an immutable Linux distribution built to run Rancher and it's corresponding Kubernetes distributions RKE2 and k3s. It is built using the cOS-toolkit and based on openSUSE

Rancher Sandbox 19 May 13, 2022
This action prints "true" if image is required to update based on the base image update.

container-image-updater This action prints "true" if image is required to update based on the base image update. Inputs Name Type Description base-ima

Manjunath Kumatagi 1 Apr 15, 2022
Docker-based remote code runner / 基于 Docker 的远程代码运行器

Docker-based remote code runner / 基于 Docker 的远程代码运行器

E99p1ant 35 Feb 22, 2022
k8s-image-swapper Mirror images into your own registry and swap image references automatically.

k8s-image-swapper Mirror images into your own registry and swap image references automatically. k8s-image-swapper is a mutating webhook for Kubernetes

Enrico Stahn 300 May 1, 2022
A tool to restart a Docker container with a newer version of the image

repull A tool to restart a Docker container with a newer version of an image used by the container Often you may need to pull a newer version of an im

Eugene 23 May 14, 2022
Triggers an update to a Koyeb app service to re-deploy the latest docker image

Triggers an update to a Koyeb app service to re-deploy the latest docker image

Reece May 3 May 5, 2021
Docker image for setting up one or multiple TCP ports forwarding, using socat

Docker socat Port Forward Docker image for setting up one or multiple TCP ports forwarding, using socat. Getting started The ports mappings are set wi

David Lorenzo 4 May 15, 2022
A docker image and a launcher to run sasm on Windows and MacOS

Sasm-docker Sasm-docker simplifies the setup and use of SASM by running it inside a docker container and using x11 (X Window System) in order to displ

Keinen Clue 8 Dec 20, 2021