Build automation for the container era

Overview

Earthly

GitHub Actions CI Join the chat on Slack Docs Website Install Earthly Docker Hub License BSL

🐳 Build anything via containers - build images or standalone artifacts (binaries, packages, arbitrary files)

πŸ›  Programming language agnostic - allows use of language-specific build tooling

πŸ” Repeatable builds - does not depend on user's local installation: runs the same locally, as in CI

β›“ Parallelism that just works - build in parallel without special considerations

🏘 Mono and Poly-repo friendly - ability to split the build definitions across vast project hierarchies

πŸ’Ύ Shared caching - share build cache between CI runners

πŸ”€ Multi-platform - build for multiple platforms in parallel


🌍 Earthly is a build automation tool for the container era. It allows you to execute all your builds in containers. This makes them self-contained, repeatable, portable and parallel. You can use Earthly to create Docker images and artifacts (eg binaries, packages, arbitrary files).


Get Earthly




Table of Contents



Why Use Earthly?

πŸ” Reproduce CI failures

Earthly builds are self-contained, isolated and repeatable. Regardless of whether Earthly runs in your CI or on your laptop, there is a degree of guarantee that the build will run the same way. This allows for faster iteration on the build scripts and easier debugging when something goes wrong. No more git commit -m "try again".

🀲 Builds that run the same for everyone

Repeatable builds also mean that your build will run the same on your colleagues' laptop without any additional project-specific or language-specific setup. This fosters better developer collaboration and mitigates works-for-me type of issues.

πŸš€ From zero to working build in minutes

Jump from project to project with ease, regardless of the language they are written in. Running the project's test suites is simply a matter of running an Earthly target (without fiddling with project configuration to make it compile and run on your system). Contribute across teams with confidence.

πŸ“¦ Reusability

A simple, yet powerful import system allows for reusability of builds across directories or even across repositories. Importing other builds does not have hidden environment-specific implications - it just works.

❀️ It's like Makefile and Dockerfile had a baby

Taking some of the best ideas from Makefiles and Dockerfiles, Earthly combines two build specifications into one.



Where Does Earthly Fit?

Earthly fits between language-specific tooling and the CI

Earthly is meant to be used both on your development machine and in CI. It can run on top of popular CI systems (like Jenkins, Circle, GitHub Actions). It is typically the layer between language-specific tooling (like maven, gradle, npm, pip, go build) and the CI build spec.



How Does It Work?

In short: containers, layer caching and complex build graphs!

Earthly executes builds in containers, where execution is isolated. The dependencies of the build are explicitly specified in the build definition, thus making the build self-sufficient.

We use a target-based system to help users break-up complex builds into reusable parts. Nothing is shared between targets, other than clearly declared dependencies. Nothing shared means no unexpected race conditions. In fact, the build is executed in parallel whenever possible, without any need for the user to take care of any locking or unexpected environment interactions.

ℹ️ Note

Earthfiles might seem very similar to Dockerfile multi-stage builds. In fact, the same technology is used underneath. However, a key difference is that Earthly is designed to be a general purpose build system, not just a Docker image specification. Read more about how Earthly is different from Dockerfiles.


Installation

See installation instructions.

To build from source, check the contributing page.



Quick Start

Here are some resources to get you started with Earthly

See also the full documentation.

Reference pages

A simple example (for Go)

# Earthfile
FROM golang:1.15-alpine3.13
RUN apk --update --no-cache add git
WORKDIR /go-example

all:
  BUILD +lint
  BUILD +docker

build:
  COPY main.go .
  RUN go build -o build/go-example main.go
  SAVE ARTIFACT build/go-example AS LOCAL build/go-example

lint:
  RUN go get golang.org/x/lint/golint
  COPY main.go .
  RUN golint -set_exit_status ./...

docker:
  COPY +build/go-example .
  ENTRYPOINT ["/go-example/go-example"]
  SAVE IMAGE go-example:latest
// main.go
package main

import "fmt"

func main() {
  fmt.Println("hello world")
}

Invoke the build using earthly +all.

Demonstration of a simple Earthly build

Examples for other languages are available on the examples page.



Features

πŸ“¦ Modern import system

Earthly can be used to reference and build targets from other directories or even other repositories. For example, if we wanted to build an example target from the github.com/earthly/earthly repository, we could issue

# Try it yourself! No need to clone.
earthly github.com/earthly/earthly/examples/go:main+docker
# Run the resulting image.
docker run --rm go-example:latest

πŸ”¨ Reference other targets using +

Use + to reference other targets and create complex build inter-dependencies.

Target and artifact reference syntax

Examples

  • Same directory (same Earthfile)

    BUILD +some-target
    FROM +some-target
    COPY +some-target/my-artifact ./
  • Other directories

    BUILD ./some/local/path+some-target
    FROM ./some/local/path+some-target
    COPY ./some/local/path+some-target/my-artifact ./
  • Other repositories

    BUILD github.com/someone/someproject:v1.2.3+some-target
    FROM github.com/someone/someproject:v1.2.3+some-target
    COPY github.com/someone/someproject:v1.2.3+some-target/my-artifact ./

πŸ’Ύ Caching that works the same as Docker builds

Demonstration of Earthly's caching

Cut down build times in CI through Shared Caching.

πŸ›  Multi-platform support

Build for multiple platforms in parallel.

all:
    BUILD \
        --platform=linux/amd64 \
        --platform=linux/arm64 \
        --platform=linux/arm/v7 \
        --platform=linux/arm/v6 \
        +build

build:
    FROM alpine:3.13
    CMD ["uname", "-m"]
    SAVE IMAGE multiplatform-image

β›“ Parallelization that just works

Whenever possible, Earthly automatically executes targets in parallel.

Demonstration of Earthly's parallelization

🀲 Make use of build tools that work everywhere

No need to ask your team to install protoc, a specific version of Python, Java 1.6 or the .NET Core ecosystem. You only install once, in your Earthfile, and it works for everyone. Or even better, you can just make use of the rich Docker Hub ecosystem.

FROM golang:1.15-alpine3.13
WORKDIR /proto-example

proto:
  FROM namely/protoc-all:1.29_4
  COPY api.proto /defs
  RUN --entrypoint -- -f api.proto -l go
  SAVE ARTIFACT ./gen/pb-go /pb AS LOCAL pb

build:
  COPY go.mod go.sum .
  RUN go mod download
  COPY +proto/pb pb
  COPY main.go ./
  RUN go build -o build/proto-example main.go
  SAVE ARTIFACT build/proto-example

See full example code.

πŸ”‘ Cloud secrets support built-in

Secrets are never stored within an image's layers and they are only available to the commands that need them.

earthly set /user/github/token 'shhh...'
release:
  RUN --push --secret GITHUB_TOKEN=+secrets/user/github/token github-release upload file.bin


FAQ

How is Earthly different from Dockerfiles?

Dockerfiles were designed for specifying the make-up of Docker images and that's where Dockerfiles stop. Earthly takes some key principles of Dockerfiles (like layer caching), but expands on the use-cases. For example, Earthly can output regular artifacts, run unit and integration tests and also create several Docker images at a time - all of which are outside the scope of Dockerfiles.

It is possible to use Dockerfiles in combination with other technologies (eg Makefiles or bash files) in order to solve for such use-cases. However, these combinations are difficult to parallelize, difficult to scale across repositories as they lack a robust import system and also they often vary in style from one team to another. Earthly does not have these limitations as it was designed as a general purpose build system.

As an example, Earthly introduces a richer target, artifact and image referencing system, which allows for better reuse in complex builds spanning a single large repository or multiple repositories. Because Dockerfiles are only meant to describe one image at a time, such features are outside the scope of applicability of Dockerfiles.

How do I tell apart classical Dockerfile commands from Earthly commands?

Check out the Earthfile reference doc page. It has all the commands there and it specifies which commands are the same as Dockerfile commands and which are new.

Can Earthly build Dockerfiles?

Yes! You can use the command FROM DOCKERFILE to inherit the commands in an existing Dockerfile.

build:
  FROM DOCKERFILE .
  SAVE IMAGE some-image:latest

You may also optionally port your Dockerfiles to Earthly entirely. Translating Dockerfiles to Earthfiles is usually a matter of copy-pasting and making small adjustments. See the getting started page for some Earthfile examples.

How is Earthly different from Bazel?

Bazel is a build tool developed by Google for the purpose of optimizing speed, correctness and reproducibility of their internal monorepo codebase. Earthly draws inspiration from some of the principles of Bazel (mainly the idea of repeatable builds), but it is different in a few key ways:

  • Earthly does not replace language-specific tools, like Maven, Gradle, Webpack etc. Instead, it leverages and integrates with them. Adopting Bazel usually means that all build files need to be completely rewritten. This is not the case with Earthly as it mainly acts as the glue between builds.
  • The learning curve of Earthly is more accessible, especially if the user already has experience with Dockerfiles. Bazel, on the other hand, introduces some completely new concepts.
  • Bazel has a purely descriptive specification language. Earthly is a mix of descriptive and imperative language.
  • Bazel uses tight control of compiler toolchain to achieve true hermetic builds, whereas Earthly uses containers and well-defined inputs.

Overall, compared to Bazel, Earthly sacrifices some correctness and reproducibility in favor of significantly better usability and composability with existing open-source technologies.



Contributing

  • Please report bugs as GitHub issues.
  • Join us on Slack!
  • Questions via GitHub issues are welcome!
  • PRs welcome! But please give a heads-up in GitHub issue before starting work. If there is no GitHub issue for what you want to do, please create one.
  • To build from source, check the contributing page.


Licensing

Earthly is licensed under the Business Source License 1.1. See licenses/BSL for more information.

Issues
  • Some DIND scenarios fail when using cgroups v2

    Some DIND scenarios fail when using cgroups v2

    Notable, podman and kind have been documented to fail on systems using cgroups v2. Here is all the relevant information:

    • Docker updated to use cgroups v2 in Docker Desktop at version 4.3.0. I do not know what docker CLI version this correlates to
    • Our unit tests that use podman appear to be trying to use cgroups v1. We may need to update podman for these unit tests since docker is on v2 (and podman is inside docker here, for our unit tests), or add compatibility flags
    • To turn off/on cgroups v2, I use the following kernel param: systemd.unified_cgroup_hierarchy=0 (Pop!_OS 21.10)
    • v2 is a unified, single-root hierarchy; vs the v1 multi-root approach
    • You can examine currently running cgroups via systemd-cgtop
    • kind also does not run when using v2, inside a WITH DOCKER. I haven't dug too much into regular docker, but I assume its only partially functional in this case
    • You can check what cgroups versions you have by grep cgroup /proc/filesystems
    type:bug 
    opened by dchw 24
  • SAVE IMAGE is slow, even when there's no work to be done

    SAVE IMAGE is slow, even when there's no work to be done

    I was observing that for highly optimized builds the slowest part can be saving images. For example, here’s a repo where earth +all takes 12s if everything is cached (like I run earth +all twice). Yet if I comment the SAVE IMAGE lines out the total time drops to 2s. This implies that SAVE IMAGE is doing a lot of work, even when nothing has changed.

    Is there anything I can do speed up SAVE IMAGE in instances like this? I’m surprised that SAVE IMAGE does anything if the image hasn’t changed, is it possible for it do some more sophisticated content negotiation with the layers?

    After talking with @agbell in Slack I hypothesized that it might not be possible for Earthly to what images/layers the host has. This is all conjecture on my part, but:

    If Earth is running in a container then it doesn’t know the state of the registry on the host machine, and what layers it has. It’s only option is to export the entire image to the host, which on a Mac could be slow because containers on a mac are actually running in a VM.

    Maybe if Earth could mount/be aware of the host docker registry it could just do docker push? This reminds me of similar problems that are being solved in the Kubernetes local cluster space https://github.com/kubernetes/enhancements/tree/master/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry

    type:performance 
    opened by jazzdan 20
  • Earthly image --push to Artifactory fails with 401

    Earthly image --push to Artifactory fails with 401

    I'm using Earthly with the --push flag to push images to Artifactory and upon doing so Earthly returns the error:

                  output | [          ] pushing layers ... 0%
                 ongoing | output (15 seconds ago)
                  output | [β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ] pushing layers ... 100%
                  output | WARN: (exporting outputs) failed commit on ref "layer-sha256:b8c8466005032e78723c242ae920cf0eb59f27af010c8646dea7a16f4dda57db": unexpected status: 401 Unauthorized
    ================================ SUCCESS [main] ================================
    Error: failed commit on ref "layer-sha256:b8c8466005032e78723c242ae920cf0eb59f27af010c8646dea7a16f4dda57db": unexpected status: 401 Unauthorized
    

    and the earthly-buildkitd logs show:

    time="2021-05-28T10:12:32Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed commit on ref \"layer-sha256:b8c8466005032e78723c242ae920cf0eb59f27af010c8646dea7a16f4dda57db\": unexpected status: 401 Unauthorized\n"
    err EOF
    

    I issue a docker login before attempting the Earthly build.

    Using docker to push the image works as does docker buildx build --push so Artifactory is accepting image uploads from other sources, I just can't get it to work directly from Earthly.

    I have tried this with some other registries and they all succeed:

    • Nexus
    • Docker v2 registry (running locally)
    • JFrog Container Registry

    Would you have any suggestions for what could be wrong?

    This is a corporate Artifactory server so I don't currently have logs from it but I am trying to get them to see if they reveal anything.

    Thanks!

    type:bug 
    opened by jamesalucas 17
  • bug: docker authentication expires after a few hours causing 401 errors while building

    bug: docker authentication expires after a few hours causing 401 errors while building

    I have an Earthfile that extends a private image. After a few hours of using earthly, I start getting 401 errors. I'm still able to pull the image with docker pull, and running docker login doesn't have any effect.

    I am able to get it working again by restarting the earthly container manually, or by changing the arguments I run with.

    $ earthly -i +test
         buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
         quay.io/<org>/<image>:<tag> | --> Load metadata linux/amd64
         quay.io/<org>/<image>:<tag> | WARN: (Load metadata linux/amd64) unexpected status code [manifests ci]: 401 UNAUTHORIZED
    Error: unexpected status code [manifests ci]: 401 UNAUTHORIZED
    
    $ docker login quay.io                                              
    Authenticating with existing credentials...
    Login Succeeded
    
    $ docker pull quay.io/<org>/<image>:<tag>
    <tag>: Pulling from <org>/<image>
    Digest: sha256:fcacc41495f6a7655032299d0ea83de0a9e577b64eaaaab8e4bf53a8f81bddb3
    Status: Image is up to date for quay.io/<org>/<image>:<tag>
    quay.io/<org>/<image>:<tag>
    
    $ earthly -i +test
         buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
         quay.io/<org>/<image>:<tag> | --> Load metadata linux/amd64
         quay.io/<org>/<image>:<tag> | WARN: (Load metadata linux/amd64) unexpected status code [manifests ci]: 401 UNAUTHORIZED
    Error: unexpected status code [manifests ci]: 401 UNAUTHORIZED
    
    $ earthly +test
                       buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
                       buildkitd | Settings do not match. Restarting buildkit daemon with updated settings...
                       buildkitd | ...Done
     quay.io/<org>/<image>:<tag> | --> Load metadata linux/amd64
                           +base | --> FROM quay.io/<org>/<image>:<tag>
                         context | --> local context .
                           +base | [β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ] resolve quay.io/<org>/<image>:<tag>@sha256:fcacc41495f6a7655032299d0ea83de0a9e577b64eaaaab8e4bf53a8f81bddb3 ... 100%
                         context | [β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ] transferring .: ... 100%
    

    This documentation suggests that Earthly should be able to inherited from the host. This page doesn't contain anything special for quay.

    Is it possible I'm missing something? Or is this unintended behavior?

    opened by tchupp 17
  • SSL/TLS trust issues: unable to add custom certificates or disable verification

    SSL/TLS trust issues: unable to add custom certificates or disable verification

    Hi, I'm working in a corporate environment where I need to specify the certificates which should be used for verification of remote connections. Unfortunately, right now I can't pull from internal registries since I get the following error:

    ...snip...
         r/r/ubi8:latest | --> Load metadata linux/amd64
         r/r/ubi8:latest | WARN: (Load metadata linux/amd64) failed to do request: Head https://REGISTRY/PATH: x509: certificate signed by unknown authority
    Error: failed to do request: Head https://REGISTRY/PATH: x509: certificate signed by unknown authority
    

    Since the registry is contacted every time regardless of local image presence (outlined further here in #345), it appears that I have a blocker to adopting Earthly.

    How can I provide more information to troubleshooting this? I would be happy to try some things out, as well. I really appreciate all of your time and efforts.

    Thanks, +Jonathan

    type:enhancement 
    opened by yonkeltron 17
  • panic: failed to get edge

    panic: failed to get edge

    I've recently upgraded to Earthly 0.5.13 and our CI jobs that run multiple Earthly targets in parallel have started failing with errors such as this:

    # ...other yarn install logs...
    ./p/sometarget+dependencies | [5/5] Building fresh packages...
    ./p/sometarget+dependencies | success Saved lockfile.
    ./p/sometarget+dependencies | Done in 102.14s.
    ./p/sometarget+dependencies | yarn cache v1.22.0
    ./p/sometarget+dependencies | success Cleared cache.
    ./p/sometarget+dependencies | Done in 6.45s.
    =================================Buildkit Logs==================================
    Error: transport is closing
    It seems that buildkitd is shutting down or it has crashed. You can report crashes at https://github.com/earthly/earthly/issues/new.
    starting earthly-buildkit with EARTHLY_GIT_HASH=a7f082f8fade1badbee0a221c9a9da2317449e4c BUILDKIT_BASE_IMAGE=github.com/earthly/buildkit:c1749dff2545b0202fc15f33eaa3278b1aa8803e+build
    BUILDKIT_ROOT_DIR=/tmp/earthly/buildkit
    CACHE_SIZE_MB=150000
    EARTHLY_ADDITIONAL_BUILDKIT_CONFIG=
    CNI_MTU=1500
    ======== CNI config ==========
    {
    	"cniVersion": "0.3.0",
    	"name": "buildkitbuild",
    	"type": "bridge",
    	"bridge": "cni0",
    	"isGateway": true,
    	"ipMasq": true,
    	"mtu": 1500,
    	"ipam": {
    		"type": "host-local",
    		"subnet": "172.30.0.0/16",
    		"routes": [
    			{ "dst": "0.0.0.0/0" }
    		]
    	}
    }
    ======== End CNI config ==========
    ======== Buildkitd config ==========
    debug = false
    root = "/tmp/earthly/buildkit"
    insecure-entitlements = [ "security.insecure" ]
    [worker.oci]
      enabled = true
      snapshotter = "auto"
      gc = true
      networkMode = "cni"
      cniBinaryPath = "/usr/libexec/cni"
      cniConfigPath = "/etc/cni/cni-conf.json"
        # Please note the required indentation to fit in buildkit.toml.template accordingly.
      # 1/100 of total cache size.
      gckeepstorage = 1500000000
      [[worker.oci.gcpolicy]]
        # 1/10 of total cache size.
        keepBytes = 15000000000
        filters = [ "type==source.local", "type==source.git.checkout"]
      [[worker.oci.gcpolicy]]
        all = true
        # Cache size MB with 6 zeros, to turn it into bytes.
        keepBytes = 150000000000
    ======== End buildkitd config ==========
    Detected container architecture is x86_64
    starting shellrepeater
    time="2021-05-27T14:06:14Z" level=info msg="auto snapshotter: using overlayfs"
    time="2021-05-27T14:06:29Z" level=info msg="found worker \"7dl0od0h2nmn3awn6fgkr04er\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:9a81ae26219d org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/386]"
    time="2021-05-27T14:06:29Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
    time="2021-05-27T14:06:29Z" level=info msg="found 1 workers, default=\"7dl0od0h2nmn3awn6fgkr04er\""
    time="2021-05-27T14:06:29Z" level=warning msg="currently, only the default worker can be used."
    time="2021-05-27T14:06:29Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
    panic: failed to get edge
    goroutine 16 [running]:
    github.com/moby/buildkit/solver.(*pipeFactory).NewInputRequest(0xc00c90def8, 0x0, 0x1549b20, 0xc00c6d8700, 0xc01bed6400, 0xc001961928, 0xc00453d200)
    	/src/solver/scheduler.go:354 +0x1fc
    github.com/moby/buildkit/solver.(*edge).createInputRequests(0xc001960dc0, 0x2, 0xc00c90def8, 0x0, 0x1dff070)
    	/src/solver/edge.go:809 +0x31d
    github.com/moby/buildkit/solver.(*edge).unpark(0xc001960dc0, 0xc01bed8420, 0x1, 0x1, 0xc00c90de48, 0x0, 0x0, 0x1dff070, 0x0, 0x0, ...)
    	/src/solver/edge.go:360 +0x179
    github.com/moby/buildkit/solver.(*scheduler).dispatch(0xc00023d180, 0xc001960dc0)
    	/src/solver/scheduler.go:136 +0x430
    github.com/moby/buildkit/solver.(*scheduler).loop(0xc00023d180)
    	/src/solver/scheduler.go:104 +0x179
    created by github.com/moby/buildkit/solver.newScheduler
    	/src/solver/scheduler.go:35 +0x1ab
    

    Similar logs in the job that is in progress at the same time...

    ./p/someothertarget+dependencies | [5/5] Building fresh packages...
    ./p/someothertarget+dependencies | success Saved lockfile.
    ./p/someothertarget+dependencies | Done in 85.08s.
    ./p/someothertarget+dependencies | yarn cache v1.22.0
    ./p/someothertarget+dependencies | success Cleared cache.
    ./p/someothertarget+dependencies | Done in 7.32s.
    ./p/someothertarget+dependencies | package=someothertarget
    ./p/someothertarget+dependencies | *cached* --> SAVE ARTIFACT ./ ./packages/someothertarget+dependencies/app
    =================================Buildkit Logs==================================
    Error: transport is closing
    It seems that buildkitd is shutting down or it has crashed. You can report crashes at https://github.com/earthly/earthly/issues/new.
    starting earthly-buildkit with EARTHLY_GIT_HASH=a7f082f8fade1badbee0a221c9a9da2317449e4c BUILDKIT_BASE_IMAGE=github.com/earthly/buildkit:c1749dff2545b0202fc15f33eaa3278b1aa8803e+build
    BUILDKIT_ROOT_DIR=/tmp/earthly/buildkit
    CACHE_SIZE_MB=150000
    EARTHLY_ADDITIONAL_BUILDKIT_CONFIG=
    CNI_MTU=1500
    ======== CNI config ==========
    {
    	"cniVersion": "0.3.0",
    	"name": "buildkitbuild",
    	"type": "bridge",
    	"bridge": "cni0",
    	"isGateway": true,
    	"ipMasq": true,
    	"mtu": 1500,
    	"ipam": {
    		"type": "host-local",
    		"subnet": "172.30.0.0/16",
    		"routes": [
    			{ "dst": "0.0.0.0/0" }
    		]
    	}
    }
    ======== End CNI config ==========
    ======== Buildkitd config ==========
    debug = false
    root = "/tmp/earthly/buildkit"
    insecure-entitlements = [ "security.insecure" ]
    [worker.oci]
      enabled = true
      snapshotter = "auto"
      gc = true
      networkMode = "cni"
      cniBinaryPath = "/usr/libexec/cni"
      cniConfigPath = "/etc/cni/cni-conf.json"
        # Please note the required indentation to fit in buildkit.toml.template accordingly.
      # 1/100 of total cache size.
      gckeepstorage = 1500000000
      [[worker.oci.gcpolicy]]
        # 1/10 of total cache size.
        keepBytes = 15000000000
        filters = [ "type==source.local", "type==source.git.checkout"]
      [[worker.oci.gcpolicy]]
        all = true
        # Cache size MB with 6 zeros, to turn it into bytes.
        keepBytes = 150000000000
    ======== End buildkitd config ==========
    Detected container architecture is x86_64
    starting shellrepeater
    time="2021-05-27T14:06:14Z" level=info msg="auto snapshotter: using overlayfs"
    time="2021-05-27T14:06:29Z" level=info msg="found worker \"7dl0od0h2nmn3awn6fgkr04er\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:9a81ae26219d org.mobyproject.buildkit.worker.snapshotter:overlayfs], platforms=[linux/amd64 linux/386]"
    time="2021-05-27T14:06:29Z" level=warning msg="skipping containerd worker, as \"/run/containerd/containerd.sock\" does not exist"
    time="2021-05-27T14:06:29Z" level=info msg="found 1 workers, default=\"7dl0od0h2nmn3awn6fgkr04er\""
    time="2021-05-27T14:06:29Z" level=warning msg="currently, only the default worker can be used."
    time="2021-05-27T14:06:29Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
    panic: failed to get edge
    goroutine 16 [running]:
    github.com/moby/buildkit/solver.(*pipeFactory).NewInputRequest(0xc00c90def8, 0x0, 0x1549b20, 0xc00c6d8700, 0xc01bed6400, 0xc001961928, 0xc00453d200)
    	/src/solver/scheduler.go:354 +0x1fc
    github.com/moby/buildkit/solver.(*edge).createInputRequests(0xc001960dc0, 0x2, 0xc00c90def8, 0x0, 0x1dff070)
    	/src/solver/edge.go:809 +0x31d
    github.com/moby/buildkit/solver.(*edge).unpark(0xc001960dc0, 0xc01bed8420, 0x1, 0x1, 0xc00c90de48, 0x0, 0x0, 0x1dff070, 0x0, 0x0, ...)
    	/src/solver/edge.go:360 +0x179
    github.com/moby/buildkit/solver.(*scheduler).dispatch(0xc00023d180, 0xc001960dc0)
    	/src/solver/scheduler.go:136 +0x430
    github.com/moby/buildkit/solver.(*scheduler).loop(0xc00023d180)
    	/src/solver/scheduler.go:104 +0x179
    created by github.com/moby/buildkit/solver.newScheduler
    	/src/solver/scheduler.go:35 +0x1ab
    

    Should running jobs in parallel on the same machine be OK? They are both run in Gitlab via Docker executors but on the same Docker host so the buildkitd instance is shared.

    type:bug 
    opened by jamesalucas 15
  • Ruby on rails example

    Ruby on rails example

    We need an example rails application added to the examples folder: https://github.com/earthly/earthly/tree/master/examples

    An example Earthfile should also be included which can build, test, and produce a docker file of the example.

    The Go example can serve as a reference point for how to structure an example.

    good first issue hacktoberfest type:example 
    opened by adamgordonbell 15
  • React example

    React example

    We need an example React JS app added to the examples folder: https://github.com/earthly/earthly/tree/master/examples

    An example Earthfile should also be included which can build and run tests.

    The Go example can serve as a reference point for how to structure an example.

    good first issue hacktoberfest type:example 
    opened by adamgordonbell 15
  • expand autocomplete to work with user home directories

    expand autocomplete to work with user home directories

    Currently if you type earth ~<tab><tab> this auto completes to earth ~/, instead it should auto complete to list all available users (e.g. ~/, ~adam, ~alex, ~corey, ~vlad).

    Here's the relevant section of the code that needs to be expanded to list other users:

    https://github.com/earthly/earthly/blob/master/autocomplete/complete.go#L106-L109

    This will also have to be able to do autocompletion so when you type earth ~a<tab><tab>, it'll only display users starting with an a (e.g. ~adam, ~alex)

    good first issue hacktoberfest 
    opened by alexcb 15
  • Support multi-platform builds

    Support multi-platform builds

    Is it possible to support multi-platform (e.g. amd64, arm64, arm v7) image build with docker buildx feature? With docker command, by using compatible builder, we can now provide --platform flag to specify target platform.

    docker [buildx] build --platform linux/amd64,linux/arm64,linux/arm/v7 .

    https://docs.docker.com/buildx/working-with-buildx/

    type:proposal 
    opened by LuaxY 14
  • Earthly on WSL2 fails with

    Earthly on WSL2 fails with "is not absolute" Git path error when checkout on Windows 10

    I'm trying to run Earthly on WSL2:

    earthly version v0.6.7 30a39d9b10c429e5a34cdd753652baf9966a3b63 linux/amd64; Ubuntu 20.04
    

    but the command fails on detectgit.go: https://github.com/earthly/earthly/blob/7e883a82c30f6d8d530f3f027db489b9d5c9926b/util/gitutil/detectgit.go#L261-L264

    Error: build target: build main: bkClient.Build: failed to solve: resolve build context for target +build: get rel dir for . when base git path is C:/Users/gianluca.massera/Developer/repos/osirium-main: git base path C:/Users/gianluca.massera/Developer/repos/osirium-main is not absolute
    

    because the Git checkout has been performed on Windows 10 using Git for Windows and when earthly run inside WSL2 is not able to detect that and correctly detect the full path of the repository inside WSL2

    type:bug 
    opened by GMaxera 13
  • [Question] How to share an env var exported inside a target to other targets

    [Question] How to share an env var exported inside a target to other targets

    Hi,

    I was wondering, if there is any recommended way to share env vars between targets.

    Let's say that I have the following and the word "HELLO" is not actually static but something dynamic like GitVersion variables calculated inside the target_a.

    target_a:
       FROM ubuntu:latest
       RUN export A="HELLO" && echo $A
    target_b:
       FROM ubuntu:latest
       RUN export A="HELLO" && echo $A
    

    At this moment I have to do something like that in every single place that requires the env var and I was wondering if Earthly provides something smarter to store and share env vars created within a target. I have searched the docs but I haven't found anything.

    I was thinking that something like the following could be useful:

    target_a:
       FROM ubuntu:latest
    
       ENV --from-command "export A="HELLO" --save A
       
       # Or saving all the env vars after sourcing an env file
       ENV --from-command ". envfile" --save-all
       
       RUN echo $A
       SAVE VARIABLE A
    target_b:
       FROM ubuntu:latest
    
       ENV --load A=+target_a
       RUN  echo $A
    
    opened by jrodrigv 0
  • Support (or document usage of) Kaniko

    Support (or document usage of) Kaniko

    We're exploring Earthly. We're running on a GitLab environment which we don't fully control. One of the limitations of that environment is that they don't allow dind. We therefore use a custom kaniko image to build our docker images in GitLab.

    The CI Integration guide's GitLab example section shows how to get started on GitLab with dind; is it technically possible to use kaniko? (note, I'm unsure if you're mostly using dind for building and pushing images, or if you're using it in a much broader sense that Kaniko couldn't cover).

    Or is there another way to use earthly without requiring privileged containers?

    opened by rubin55 0
  • Strange behavior with `SAVE ARTIFACT`

    Strange behavior with `SAVE ARTIFACT`

    I’m seeing some strange behavior with SAVE ARTIFACT . I’m trying to save an artifact both locally and to be used by other Earthfiles.

    VERSION 0.6
    FROM amazon/aws-cli:latest
    
    COPY --if-exists ./cache/ /root/.aws/
    COPY aws_config /root/.aws/config
    
    dev.login:
        IF --no-cache ! aws sts get-caller-identity
          RUN --no-cache aws sso login
          SAVE ARTIFACT /root/.aws /aws AS LOCAL ./cache
        END
    

    When I run earthly+dev.login in my aws project this works fine. The cache directory gets created. https://ci.earthly.dev/logs?logId=cf8cd11f-c3cd-4e06-a77d-435859e4ad0c

    When I run another target in another directory, I get an error:

    build:
        ARG only
        FROM +init
        COPY ../aws+dev.login/aws /root/.aws/
        <truncated>
    

    https://ci.earthly.dev/logs?logId=91ecbe9b-89e4-4acd-8693-1a7ae9194a2f

    The build target works if I delete the cache directory and then run the build target again.

    Is this a bug?

    type:bug 
    opened by shepherdjerred 3
  • Validation for `ARG` values

    Validation for `ARG` values

    I'd love to see some way to constrain the value of ARG values.

    It'd be nice to show developers what valid ARGs are, and provide nice help messages when their input doesn't match an expected value.

    For example, consider an Earthfile that builds binaries for two operating systems:

    build:
      ARG os
      IF $os not in ["ubuntu", "debian"] # fake shell syntax
        exit 1
      END
      FROM $os
      RUN <my compilation command> 
    

    Clearly, I can accomplish this through IF statements, but this quickly becomes cumbersome as I add more targets. For example, I might have release, test, or package targets which all accept an OS value. This boilerplate code would quickly become repetitive.

    The syntax could look something like this:

    # static list of valid values
    build:
      ARG --valid-values="ubuntu, debian" os
      FROM $os
      RUN <my compilation command> 
    
    # dynamic list of valid values
    build:
      ARG --valid-values=$(ls ./os) os
      FROM $os
      RUN <my compilation command> 
    

    Earthly could then show some helpful message when an ARG is invalid or even list the valid possibilities.

    type:enhancement category:developer-experience 
    opened by shepherdjerred 0
  • LOCALLY commands don't run in parallel

    LOCALLY commands don't run in parallel

    When commands are defined LOCALLY, they don't seem to be running in parallel. Is this expected?

    Use case:

    • I want to run a single command to start my application locally.
    • The application consists of a backend API and a front end UI.
    • I want the commands to run locally to benefit from hot reloading.

    I have defined earthfiles in the root and the two project folders that look like something like this:

    // ./Earthfile
    start:
        BUILD ./api+start
        BUILD ./ui+start
    
    // ./api/Earthfile
    start:
        LOCALLY
        RUN yarn start
    
    // ./ui/Earthfile
    start:
        LOCALLY
        RUN yarn start
    

    Expected behaviour

    I would expect the two independent build commands to run concurrently, starting both applications and remaining running until the command is cancelled in the terminal.

    Actual behaviour

    The commands run sequentially. Because the commands are blocking, the second command never runs.

    Simple test case

    VERSION 0.6
    
    FROM busybox:latest
    
    start:
        BUILD +test
        BUILD +lint
    
    test:
        RUN sleep 5 && echo "Done Test"
    
    lint:
        RUN sleep 5 && echo "Done Lint"
    
    
    start-locally:
        BUILD +test-locally
        BUILD +lint-locally
    
    test-locally:
        LOCALLY
        RUN sleep 5 && echo "Done Test"
    
    lint-locally:
        LOCALLY
        RUN sleep 5 && echo "Done Lint"
    

    +start Runs commands in parallel taking about 8 seconds

    $ time earthly --no-cache +start
    
    ...
    
          busybox:latest | --> Load metadata linux/amd64
                   +base | --> FROM busybox:latest
                   +base | [β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ] 100% resolve docker.io/library/busybox:[email protected]
                   +test | --> RUN sleep 5 && echo "Done Test"
                   +lint | --> RUN sleep 5 && echo "Done Lint"
                 ongoing | +test (4 seconds ago), +lint (4 seconds ago)
                   +lint | Done Lint
                   +test | Done Test
                  output | --> exporting outputs
    
    ========================== 🌍 Earthly Build  βœ… SUCCESS ==========================
    
    real	0m8.834s
    user	0m1.673s
    sys	0m0.908s
    

    +start-locally Runs commands sequential taking about 15 seconds

    $ time earthly --no-cache +start-locally
    
          busybox:latest | --> Load metadata linux/amd64
           +lint-locally *local* | --> RUN sleep 5 && echo "Done Lint"
           +test-locally *local* | --> RUN sleep 5 && echo "Done Test"
                 ongoing | +test-locally (4 seconds ago), +lint-locally (4 seconds ago)
           +lint-locally *local* | Done Lint
                 ongoing | +test-locally (10 seconds ago)
           +test-locally *local* | Done Test
                   +base | --> FROM busybox:latest
                   +base | [β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ] 100% resolve docker.io/library/busybox:[email protected]
                  output | --> exporting outputs
    
    
    ========================== 🌍 Earthly Build  βœ… SUCCESS ==========================
    
    Share your logs with an Earthly account (experimental)! Register for one at https://ci.earthly.dev.
    
    real	0m15.207s
    user	0m1.817s
    sys	0m1.258s
    
    opened by mrjgreen 1
Releases(v0.6.18)
Owner
Earthly
Build automation for the post-container era
Earthly
🌍 Earthly is a build automation tool for the container era

?? Earthly is a build automation tool for the container era. It allows you to execute all your builds in containers. This makes them self-contained, repeatable, portable and parallel. You can use Earthly to create Docker images and artifacts (eg binaries, packages, arbitrary files).

Earthly 7.3k Jun 30, 2022
Boxygen is a container as code framework that allows you to build container images from code

Boxygen is a container as code framework that allows you to build container images from code, allowing integration of container image builds into other tooling such as servers or CLI tooling.

nitric 5 Dec 13, 2021
Composable, observable and performant config handling for Go for the distributed processing era

Konfig Composable, observable and performant config handling for Go. Written for larger distributed systems where you may have plenty of configuration

Lalamove 628 Jun 12, 2022
k6 is a modern load testing tool for developers and testers in the DevOps era.

k6 is a modern load testing tool, building on our years of experience in the load and performance testing industry. It provides a clean, approachable scripting API, local and cloud execution, and flexible configuration.

k6 16.9k Jun 22, 2022
Baker is a high performance, composable and extendable data-processing pipeline for the big data era

Baker is a high performance, composable and extendable data-processing pipeline for the big data era. It shines at converting, processing, extracting or storing records (structured data), applying whatever transformation between input and output through easy-to-write filters.

AdRoll 148 Jun 17, 2022
gobetween - modern & minimalistic load balancer and reverse-proxy for the ☁️ Cloud era.

gobetween - modern & minimalistic load balancer and reverse-proxy for the ☁️ Cloud era. Current status: Maintenance mode, accepting PRs. Currently in

Yaroslav Pogrebnyak 1.7k Jun 27, 2022
Scrape the web in the eink era. Convert websites into books.

Scrape the web in the eink era. Convert websites into books.

null 32 Jun 21, 2022
Moby Project - a collaborative project for the container ecosystem to assemble container-based systems

The Moby Project Moby is an open-source project created by Docker to enable and accelerate software containerization. It provides a "Lego set" of tool

Moby 63.3k Jun 23, 2022
Yeqllo 22 Nov 26, 2021
Moby Project - a collaborative project for the container ecosystem to assemble container-based systems

The Moby Project Moby is an open-source project created by Docker to enable and accelerate software containerization. It provides a "Lego set" of tool

Moby 63.4k Jun 24, 2022
top in container - Running the original top command in a container

Running the original top command in a container will not get information of the container, many metrics like uptime, users, load average, tasks, cpu, memory, are about the host in fact. topic(top in container) will retrieve those metrics from container instead, and shows the status of the container, not the host.

silenceshell 69 Jun 20, 2022
Amazon ECS Container Agent: a component of Amazon Elastic Container Service

Amazon ECS Container Agent The Amazon ECS Container Agent is a component of Amazon Elastic Container Service (Amazon ECS) and is responsible for manag

null 0 Dec 28, 2021
The Container Storage Interface (CSI) Driver for Fortress Block Storage This driver allows you to use Fortress Block Storage with your container orchestrator

fortress-csi The Container Storage Interface (CSI) Driver for Fortress Block Storage This driver allows you to use Fortress Block Storage with your co

Fortress 0 Jan 23, 2022
Ixia-c-one - A re-packaged (as a single-container) flavor of multi-container application ixia-c

ixia-c-one ixia-c-one is a re-packaged (as a single-container) flavor of multi-c

Open Traffic Generator 3 Apr 1, 2022
Build & Scan - Container Image

BSImage (build&scan image) Requirements Trivy Docker BSImage (shell script version) Usage of build (shell script version) ./bsimage.sh start <image:ta

Carlos Eduardo Vieira 1 Apr 12, 2022
Concurrent task runner, developer's routine tasks automation toolkit. Simple modern alternative to GNU Make 🧰

taskctl - concurrent task runner, developer's routine tasks automation toolkit Simple modern alternative to GNU Make. taskctl is concurrent task runne

null 192 Jun 18, 2022
Enterprise-Grade Continuous Delivery & DevOps Automation Open Source Platform

CDS: Continuous Delivery Service CDS is an Enterprise-Grade Continuous Delivery & DevOps Automation Platform written in Go(lang). This project is unde

OVHcloud 3.9k Jun 19, 2022
Gentee - script programming language for automation. It uses VM and compiler written in Go (Golang).

Gentee script programming language Gentee is a free open source script programming language. The Gentee programming language is designed to create scr

Alexey Krivonogov 95 May 20, 2022
RobotGo, Go Native cross-platform GUI automation @vcaesar

Robotgo Golang Desktop Automation. Control the mouse, keyboard, bitmap, read the screen, Window Handle and global event listener. RobotGo supports Mac

vgo 7.6k Jun 23, 2022
End to end functional test and automation framework

Declarative end to end functional testing (endly) This library is compatible with Go 1.12+ Please refer to CHANGELOG.md if you encounter breaking chan

Viant, Inc 209 Jun 26, 2022
Playwright for Go a browser automation library to control Chromium, Firefox and WebKit with a single API.

?? Playwright for API reference | Example recipes Playwright is a Go library to automate Chromium, Firefox and WebKit with a single API. Playwright is

Max Schmitt 833 Jun 22, 2022
A Devtools driver for web automation and scraping

Overview Documentation | API reference Rod is a high-level driver directly based on DevTools Protocol. It's designed for web automation and scraping.

Rod 2.5k Jun 29, 2022
Β΅Task is an automation engine that models and executes business processes declared in yaml. βœοΈπŸ“‹

Β΅Task, the Lightweight Automation Engine Β΅Task is an automation engine built for the cloud. It is: simple to operate: only a postgres DB is required s

OVHcloud 665 Jun 18, 2022
Gentee - script programming language for automation. It uses VM and compiler written in Go (Golang).

Gentee script programming language Gentee is a free open source script programming language. The Gentee programming language is designed to create scr

Alexey Krivonogov 95 May 20, 2022
🏠 An HTTP-based command runner for home automation.

Badges Reporting Issues If you are facing a problem with this package or found any bug, please open an issue on GitHub. License The MIT License (MIT).

Homettp 27 Apr 11, 2022
πŸ”₯ Kubernetes multi-cluster deployment automation service.

Beetle Kubernetes multi-cluster deployment automation service ?? Check out the demo! Application deployment and management should be automated, audita

Ahmed 162 Jun 17, 2022
Fluux XMPP is a Go XMPP library, focusing on simplicity, simple automation, and IoT.

Fluux XMPP is a Go XMPP library, focusing on simplicity, simple automation, and IoT. The goal is to make simple to write simple XMPP client

Fluux 270 Jun 19, 2022
Serverless SOAR (Security Orchestration, Automation and Response) framework for automatic inspection and evaluation of security alert

DeepAlert DeepAlert is a serverless framework for automatic response of security alert. Overview DeepAlert receives a security alert that is event of

null 34 Jun 15, 2022
Terraform provider to help with various AWS automation tasks (mostly all that stuff we cannot accomplish with the official AWS terraform provider)

terraform-provider-awsutils Terraform provider for performing various tasks that cannot be performed with the official AWS Terraform Provider from Has

Cloud Posse 18 Jun 17, 2022