Please is a cross-language high-performance extensible build system for reproducible multi-language builds.

Overview

Please Build Status Build Status Go Report Card Gitter chat

Please is a cross-language build system with an emphasis on high performance, extensibility and reproducibility. It supports a number of popular languages and can automate nearly any aspect of your build process.

See http://please.build for more information.

Currently Linux (tested on Ubuntu), macOS and FreeBSD are actively supported.

If you're a fan of Please, don't forget to add yourself to the adopters file.

Getting Started

The easiest way to install it on your own machine is to run:

curl -s https://get.please.build | bash

Or, if you prefer, grab one of the tarballs off our releases page and extract it yourself; it typically lives in ~/.please.

You can also install using Homebrew:

brew tap thought-machine/please
brew install please

Then you simply run plz init at the root of your project to set up a default config and you're ready to go. The best way to get to grips with Please is through the codelabs! There's also the getting started guide that explains the core concepts.

How is it so fast?

Please has a robust and correct caching model that enables us to aggressively cache artifacts. Caching is based on the hashes of inputs (both files, and environment variables) to each rule rather than last modified timestamps. Builds are hermetic so don't have access to anything they haven't explicitly defined as inputs. This means that if anything changes, we know exactly what might've been affected, so the minimal set of targets get built and tested.

Because each task is hermetic, they can be run in parallel without any chance of interfering with each-other. Combine these two concepts with shared remote caches, and it makes for a blazing fast build system for any language or technology.

Please is also written in Go and every effort has been made to make it as fast as possible. There's no startup time waiting to bring up VMs, interpreting code or communicating with remote JVM processes. The code itself takes full advantage of Go's concurrency and asynchronicity. The end result is a snappy command line tool that gets to work immediately and feels great to use.

Why Please, and not Maven, pip, or go build?

A build system is more than just a mechanism for invoking the compiler. If you're working on just one language, don't have any code generation, and don't need to publish any artifacts anywhere, you might not need Please. Chances are this is not the case.

Building software often involves more than just compiling code. There's deployment config to template, code to generate, and quite often, there's more than one language involved. Please provides a powerful, comprehensive, and understandable framework that you can use to craft a truly holistic build process.

Please does this through a consistent and seamless command line interface; there's no need to learn new build systems and technologies for different languages. Build any target with plz build, test any target with plz test, no matter what's going on under the hood.

The Docker & Kubernetes codelab covers building a Kubernetes based application with Please, including reliably deploying code to a local cluster, and pushing it to a remote registry.

The genrule() codelab covers extending Please with custom build definitions to truly automate any part of your deployment process.

Why Please, and not make?

Make is a great tool for running tasks. It's easy enough to understand because it leaves you very close to the shell. The problem is, it has limited capability to build out complexity. There have been attempts to generate make files from higher level tools like cmake and ninja, but they fall short of what Please sets out to achieve.

The Please build language is a full programming language. There are a high level set of build rules that make up a declarative DSL to define build targets, however you can drop into an imperative language that resembles python when necessary:

subinclude("//my_custom_defs:markdown_page")

pages = []
for page in glob(include = ["*.md"]):
    pages += markdown_page(
        name = page.removesuffix(".md"),
        srcs = [page],
        visibility = ["//website/..."],
    )   

go_binary (
    name = "webserver",
    srcs = ["main.go"],
    deps = ["//third_party/go:protobuf"],
    data = pages,
    visibility = ["//services/foo/..."],
)

This is distinctively operating at a higher level when compared to make:

protobuf:
    go install google.golang.org/protobuf

webserver: protobuf
    go tool compile --pack foo.go -o foo.a

pages: ???

Additionally, make builds are not hermetic. The above make example installs protobuf into the host machines Go path. Please builds only have access to files and environment variables they have explicitly been given access to. You can play around in the environment targets are built in with plz build //some/target --shell. Additionally, on linux systems, Please can take this a step further with Linux namespaces to improve sandboxing especially of the network. Please also has built in task parallelism so can take full advantage of multi-core machines which were not a consideration 40 years ago when make was designed.

Finally, Please has a robust caching mechanism base on hashing the inputs of each rule. Makes cache invalidation is based on the last modified timestamp which can change unexpectedly forwards and backwards in time. Combine this with hermetic builds, and Please caching is never incorrect.

Why Please, not Bazel, Buck or Pants?

These build systems are all very similar so choosing between them can be hard. Please orginally replaced buck implementing the subset of features we used. We found that buck (and the competition) worked great when using the supported languages but fell flat when breaking new ground.

The biggest difference between Please and the competition is that Please is designed from the ground up to be extensible. The built-in languages are all defined in the same build language as you use to define your targets, demonstrating that there's nothing special about them. This puts the build definitions where they should be: in your domain. You have all the same tools Please has to expand your build definitions to satisfy your needs.

Please does all this with a focus on simplicity and transparency. There are a limited amount of concepts that are needed to get started and once they are grocked, the possibilities are endless. Please relies on these concepts rather than requiring lots of magic and incantation. Configuration is simple and largely optional so getting going is easy, and there's no single WORKSPACE file nobody really owns, with lines of configuration that nobody really understands.

The command line interface is designed with similar considerations in mind. Subcommands can be added to Please though aliases and tie into the Please tab-completions. Not only can flags and arguments be completed, but they can also leverage the build graph to complete labels enabling you to truly craft your developer experience the way you want it.

Building Please

If you're looking to get involved, check out the contributor guidance to help you get started. If you're a fan of Please, don't forget to add yourself to the adopters file.

To build Please yourself, run ./bootstrap.sh in the repo root. This will bootstrap a minimal version of Please using Go and then rebuild it using itself.

You'll need to have Go 1.16+ installed to build Please although once built it can target any version from 1.8+ onwards.

Optional dependencies for various tests include Python, Java, clang, gold and docker - none of those are required to build components so their tests will be excluded if they aren't available.

If you'd rather not worry about installing the dependencies, we provide a prebuilt Docker image based on Ubuntu which is capable of building the whole thing for you: docker run -it thoughtmachine/please_ubuntu

You can install a locally built copy of Please using plz install, or if it's the first time, by running ./install.sh after it's built. This will overwrite whatever you currently have in ~/.please with your local version, although you can get back to a released version again by running plz update --force.

To automatically fix linting and code generation issues, run plz autofix.

Documentation

Status

Please is released & we consider it stable; we follow semver for releases, so major versions indicate potentially breaking changes to the BUILD language, command line or other behaviour. We try to minimise this where possible.

We're very happy to accept pull requests, feature requests, and bugs if it's not working for you. We don't always have time for everything but Please is under active development.

Comments
  • Supporting Windows 10 via MinGW

    Supporting Windows 10 via MinGW

    I am curious whether there would be any interest to support this project on Windows.

    Just cause of my curiosity - I decided to build this:

    $ ./bootstrap.sh Building Please...

    github.com/pkg/xattr

    .....golang\pkg\mod\github.com\pkg\[email protected]\xattr.go:40:10: undefined: getxattr .....golang\pkg\mod\github.com\pkg\[email protected]\xattr.go:47:10: undefined: lgetxattr .....golang\pkg\mod\github.com\pkg\[email protected]\xattr.go:54:10: undefined: fgetxattr .....golang\pkg\mod\github.com\pkg\[email protected]\xattr.go:106:12: undefined: setxattr .....golang\pkg\mod\github.com\pkg\[email protected]\xattr.go:115:12: undefined: lsetxattr .....golang\pkg\mod\github.com\pkg\[email protected]\xattr.go:123:12: undefined: fsetxattr .....golang\pkg\mod\github.com\pkg\[email protected]\xattr.go:132:12: undefined: setxattr .....golang\pkg\mod\github.com\pkg\[email protected]\xattr.go:141:12: undefined: lsetxattr .....golang\pkg\mod\github.com\pkg\[email protected]\xattr.go:149:12: undefined: fsetxattr .....golang\pkg\mod\github.com\pkg\[email protected]\xattr.go:157:12: undefined: removexattr .....golang\pkg\mod\github.com\pkg\[email protected]\xattr.go:157:12: too many errors

    github.com/thought-machine/please/src/cli

    src\cli\logging.go:182:23: undefined: syscall.SIGWINCH

    $ go env && go version set GO111MODULE= set GOARCH=386 set GOBIN= set GOCACHE=C:\Users\clang\AppData\Local\go-build set GOENV=C:\Users\clang\AppData\Roaming\go\env set GOEXE=.exe set GOFLAGS= set GOHOSTARCH=386 set GOHOSTOS=windows set GOINSECURE= set GOMODCACHE=C:\Users\clang.golang\pkg\mod set GONOPROXY= set GONOSUMDB= set GOOS=windows set GOPATH=C:\Users\clang.golang set GOPRIVATE= set GOPROXY=https://proxy.golang.org,direct set GOROOT=C:\Users\clang\golang set GOSUMDB=sum.golang.org set GOTMPDIR= set GOTOOLDIR=C:\Users\clang\golang\pkg\tool\windows_386 set GCCGO=gccgo set GO386=sse2 set AR=ar set CC=gcc set CXX=g++ set CGO_ENABLED=1 set GOMOD=C:\Users\clang\Desktop\please\go.mod set CGO_CFLAGS=-g -O2 set CGO_CPPFLAGS= set CGO_CXXFLAGS=-g -O2 set CGO_FFLAGS=-g -O2 set CGO_LDFLAGS=-g -O2 set PKG_CONFIG=pkg-config set GOGCCFLAGS=-m32 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\clang\AppData\Local\Temp\go-build341091154=/tmp/go-build -gno-record-gcc-switches go version go1.15 windows/386

    enhancement help wanted hard not-stale 
    opened by advancedwebdeveloper 28
  • Please diff

    Please diff

    First thanks for the amazing tool,

    Now I have this #1592 TL;DR I'm building multiple docker images with please in our monorepo where all docker files are put in the root of repo because of the dependencies stuff. now, my question is I want to build docker image based on changes in repo so you know when CI runs not all docker images will push/build and only changed one push/build.

    something like this:

    subinclude("//pleasing//docker")
    
    docker_image(
        name = "abc",
        dockerfile = "abc.Dockerfile",
        srcs = ["abc","efg","abc.go"],
        visibility = ["PUBLIC"],
        version = f'$(echo "$(git rev-parse --abbrev-ref HEAD)-$(git rev-parse --short HEAD)")'
    )
    docker_image(
        name = "xyz",
        dockerfile = "xyz.Dockerfile",
        srcs = ["xyz","efg","xyz.go"],
        visibility = ["PUBLIC"],
        version = f'$(echo "$(git rev-parse --abbrev-ref HEAD)-$(git rev-parse --short HEAD)")'
    )
    

    and in CI I'm running

    ./pleasew run //:abc_push
    ./pleasew run //:xyz_push
    

    so now if there is a change in xyz directory, target abc is also running and building docker image. which is not appropriate because abc image/srcs has no change since it's getting list of srcs!!. so am I missing something or please is not doing git diffstuff

    wontfix 
    opened by pratikbin 14
  • Is there a way to bootstrap without network access?

    Is there a way to bootstrap without network access?

    Is there a way to vendor all 3rd party dependencies to avoid requiring network access during bootstrapping? We are working on FreeBSD port and the build currently fails with

    1 error occurred:
            * Get https://github.com/kevinburke/go-bindata/archive/46eb4c183bfc1ebb527d9d19bcded39476302eb8.zip: dial tcp: lookup github.com on 1.0.0.1:53: write udp 127.0.0.1:61403->1.0.0.1:53: write: permission denied
    
    Build stopped after 40ms. 1 target failed:
        //third_party/go:_go-bindata-go-bindata#download
    1 error occurred:
            * Get https://github.com/kevinburke/go-bindata/archive/46eb4c183bfc1ebb527d9d19bcded39476302eb8.zip: dial tcp: lookup github.com on 1.0.0.1:53: write udp 127.0.0.1:61403->1.0.0.1:53: write: permission denied
    

    because network access is not allowed during build phase.

    question 
    opened by dmgk 14
  • Go module support

    Go module support

    Module support is now included in Go 1.11 and will be on by default with Go 1.12 (now in beta). How will this effect please, do we need to support it?

    On a side note, Bazels Go rules support an importpath variable. If we had something similar, we could move away from building everything in a single "virtual" gopath and decouple the filesystem layout from the import paths.

    enhancement help wanted not-stale 
    opened by nekinie 14
  • Platform specfic targets

    Platform specfic targets

    Certain targets are only required (and can only built on) for certain platforms. For example: certain dependencies might only be required by linux.

    My current approach is an if, combined with a custom select_config rule:

    if select_config([{
        "os": "linux",
        "cpu": "amd64",
    }]):
        go_library(name = "logrus", srcs = [":_logrus#go_source"], cover = False, import_path = "github.com/sirupsen/logrus", visibility = ["PUBLIC"], deps = select({"//example/third_party/go/__config:darwin_amd64": [], "//example/third_party/go/__config:linux_amd64": ["//example/third_party/go/golang.org/x/sys/unix"], "default": []}))
    

    The problem is: it's more generated code, and the buildifier package formats the target itself on a single line.

    Is there a better way to represent platform specific targets? (It's not the end of the world, but it's hard to read and more changes in a VCS)

    wontfix 
    opened by sagikazarmark 12
  • Add gocloud to allow Blob Storage as a cache

    Add gocloud to allow Blob Storage as a cache

    A follow-up from #1140: It might be beneficial to add gocloud into please, in order to allow blob storage to be used as a cache backend (so backends like AWS S3, Google Cloud Storage, or Azure Blob Storage could be used).

    This would allow users who run please builds in different CI providers to connect to a remote blob storage for their cache (or run their own one! there's always the S3-compatible flavour of the week version).

    Gocloud provides a common interface, so configuration is universal; however, it seems to call each services' specific SDKs below, so it might be a heavy dependency.

    enhancement not-stale 
    opened by robxu9 12
  • Follow symlink outputs (maybe opt-in)

    Follow symlink outputs (maybe opt-in)

    I know that I'm doing something that's against the rules of Please. But I've been doing the same thing in Bazel and it handles this use case well, so maybe Please could also do this.

    The idea is to allow build_rule to produce outputs that are symlinks to files generated in the source tree, and make Please follow the symlink to determine if the rule is dirty.

    Right now Please won't rerun the rule if the file that symlink points to changes or disappears. And Bazel does this.

    Bazel still warns you if you're using genrule for that, but if you're doing it via a custom rule - it works. Would be great if Please could do the same.

    not-stale 
    opened by burdiyan 11
  • Multiple versions can't coexist when installing please via Homebrew

    Multiple versions can't coexist when installing please via Homebrew

    When please is installed with Homebrew and there different Please versions in use, Please always downloads a new copy differing from the current version.

    I guess it's related to Homebrew.

    opened by sagikazarmark 11
  • FYI: there's a Skylark interpreter in Go

    FYI: there's a Skylark interpreter in Go

    I just learned of 'please' today, and I notice you're calling CPython 2.x in process. You might be interested in a pure-Go implementation of Skylark here: github.com/google/skylark. It has certain benefits relative to Python:

    • no GIL, so you can execute multiple threads in parallel, which can make the loading phase (BUILD file execution) and dependency analysis (rule.implementation function execution) much faster.
    • restricted import, no OS access, etc
    • control over mutability, so no possibility of a data race, despite familiar imperative paradigm.
    • compatibility with Bazel.

    I gave a talk on it: https://www.youtube.com/watch?v=9P_YKVhncWI

    cheers alan

    opened by alandonovan 11
  • Fix go_binary linking of third party dependencies outside of root #1061

    Fix go_binary linking of third party dependencies outside of root #1061

    For reference I am using MacOS.

    This issue only exists for the go_binary rule and third party dependencies which are added as library paths through command substitution. i.e. third party dependencies defined outside of the root third_party/go/BUILD.plz. The go_library rule works only because it's not actually quoting the file paths due to a subtle bug.

    $(for i in $(find "$TMP_DIR" -name pkg -type d); do echo -n " -I \"$i\"/{CONFIG.GOOS}_{CONFIG.GOARCH} "; done)
    

    The snippet above in the python go_library rule becomes the following in the actual shell.

    $(for i in $(find "$TMP_DIR" -name pkg -type d); do echo -n " -I "$i"/darwin_amd64 "; done)
    

    The escaped quote is lost and instead we get string concatenation which is probably not what was originally intended. In regards to the go_binary rule it is actually quoting the file paths. However, because it is being injected via command substitution it does not work properly. There are a couple issues here.

    Issue 1) The output of command substitution is split on whitespace regardless of quotes within the output. So, for example, a file path which includes spaces printargs $(echo '"foo bar\baz.txt"') would become the following arguments.

    arg 0: "foo
    arg 1: bar\baz.txt"
    

    Issue 2) The quotes within the command substitution output are preserved. So, if we have -L "foo/bar/pkg/darwin_amd64" it's going to attempt to look in a directory named "foo, which begins with a quote, and not the correct directory foo.

    This fix removes the need for quoting by using null separated arguments instead and providing them to the link/compile command via xargs instead of command substitution. Once I began manually testing with file paths which included spaces I noticed that the _LINK_PKGS_CMD also didn't account for spaces so I fixed it in a similar manner.

    opened by brian-bice 10
  • [Go] - use full url on package imports

    [Go] - use full url on package imports

    I see here https://github.com/thought-machine/please/blob/master/src/please.go#L27 the import of package hashes instead of github.com/thought-machine/src/hashes as usual on Go

    is there any way of importing the package using the full URL?

    opened by mishudark 10
  • Allow subrepos to output into a package root to avoid collisions

    Allow subrepos to output into a package root to avoid collisions

    WIP while I try and iron out a few things. This allows us to have subrepos to have their own output directories within the temp dir so they don't end up clobbering eachother.

    opened by Tatskaari 0
  • Override plugin config values are not inherited for subrepos

    Override plugin config values are not inherited for subrepos

    If you override plz build -o pluigins.go.gotool:go1.19, this value isn't inheritted in the same was as if it was actually written to the .plzconfig file.

    opened by Tatskaari 0
  • Fix add warning on out dir

    Fix add warning on out dir

    Draft PR fixing #2599

    Unfortunately, we can't be sure that the package will be in the build graph when we build the target. Packages are added at the end of parsing the package. We now activate targets early for subincludes, and might actually build the target before we have finished parsing the package.

    To fix this, we'd need to add packages to the graph in a pending state. We could potentially replace the pending packages map, moving the channel into the Packages struct.

    opened by Tatskaari 0
  • The github_repo moves the build file into a repository subdirectory named build

    The github_repo moves the build file into a repository subdirectory named build

    I am playing around with WASM and I have noticed that when I build this target, the build_file ends up in the subrepo's build/ subfolder.

    github_repo(
        name = "wasmtime",
        repo = "bytecodealliance/wasmtime-go",
        revision = "v1.0.0",
        build_file="wasmtime.build",
    )
    
    opened by PeterBocan 2
  • Embed the sandbox tool

    Embed the sandbox tool

    We used to re-exec Please when using the built in sandboxing which has performance issues as there's a lot of allocation and general fluff that happens in package initialisation. This PR does two things:

    1. Moves the sandbox tool into its own go_binary() rule that doesn't import anything from Please eliminating the performance issues.
    2. Embeds this in the Please binary, unpacking it on demand when we sandbox the first rule.

    We might want to consider enabling sandboxing out of the box using this method in v17.

    It's quite hard to measure the difference as we're talking milliseconds. We can get a sense for this based on how much stuff we see in strace:

    ➜  please git:(master) ✗ strace plz-out/bin/src/sandbox/cmd/sandbox 2>&1 | wc -l
    167
    ➜  please git:(master) ✗ strace plz-out/bin/src/please sandbox 2>&1 | wc -l
    589
    

    So the new approach does roughly 30% of the stuff (mostly system calls and memory allocations) the old approach did.

    opened by Tatskaari 0
Releases(v16.25.1)
Owner
Thought Machine
Thought Machine
Builds and restarts a Go project when it crashes or some watched file changes

gaper Used to build and restart a Go project when it crashes or some watched file changes Aimed to be used in development only. Changelog See Releases

Max Claus Nunes 61 Nov 26, 2022
GoReleaser builds Go binaries as fast and easily as possible

GoReleaser builds Go binaries for several platforms, creates a GitHub release and then pushes a Homebrew formula to a tap repository. All that wrapped in your favorite CI.

GoReleaser 11k Dec 4, 2022
🚀 gowatch is a command line tool that builds and (re)starts your go project everytime you save a Go or template file.

gowatch 中文文档 gowatch is a command line tool that builds and (re)starts your go project everytime you save a Go or template file. Installation To insta

silenceper 732 Dec 4, 2022
a build tool for Go, with a focus on cross-compiling, packaging and deployment

goxc NOTE: goxc has long been in maintenance mode. Ever since Go1.5 supported simple cross-compilation, this tool lost much of its value. There are st

Am Laher 1.7k Nov 30, 2022
Build system and task runner for Go projects

Gilbert is task runner that aims to provide declarative way to define and run tasks like in other projects like Gradle, Maven and etc.

Gilbert 104 Nov 14, 2022
Blueprint Build System For Golang

Blueprint Build System Blueprint is being archived on 2021 May 3. On 2021 May 3, we will be archiving the Blueprint project. This means it will not be

GengKapak WIP 0 Nov 20, 2021
go language generics system

Gotgo This document describes the third iteration of my attempt at a reasonable implementation of generics for go based on the idea of template packag

David Roundy 119 Dec 4, 2021
Create build pipelines in Go

taskflow Create build pipelines in Go This package aims to simplify the creation of build pipelines in Go instead of using scripts or Make. taskflow A

Robert Pająk 343 Nov 26, 2022
Colorize (highlight) `go build` command output

colorgo colorgo is a wrapper to go command that colorizes output from go build and go test. Installation go get -u github.com/songgao/colorgo Usage c

Song Gao 111 Sep 27, 2022
EGo lets you build, debug und run Go apps on Intel SGX - as simple as conventional Go programming!

EGo is a framework for building confidential apps in Go. Confidential apps run in always-encrypted and verifiable enclaves on Intel SGX-enabled ha

Edgeless Systems GmbH 364 Dec 8, 2022
Build and (re)start go web apps after saving/creating/deleting source files.

unmaintained Fresh Fresh is a command line tool that builds and (re)starts your web application everytime you save a Go or template file. If the web f

Andrea Franz 3.5k Nov 30, 2022
KintoHub is an open source build and deployment platform designed with a developer-friendly interface for Kubernetes.

What is Kintohub? KintoHub is an open source build and deployment platform designed with a developer-friendly interface for Kubernetes. Build your cod

KintoHub 30 Jun 7, 2022
Build systems with Go examples

What is this? This is a repository containing all the examples from the book BUILD SYSTEMS with GO (and save the world). This book is written to help

Juan M. Tirado 99 Nov 20, 2022
🌍 Earthly is a build automation tool for the container era

?? Earthly is a build automation tool for the container era. It allows you to execute all your builds in containers. This makes them self-contained, repeatable, portable and parallel. You can use Earthly to create Docker images and artifacts (eg binaries, packages, arbitrary files).

Earthly 8.4k Dec 8, 2022
An experimental way to apply patches to the Go runtime at build time.

go-patch-overlay An experimental way to apply patches to the Go runtime at build time. Assuming you have a directory of patches to apply to the Go sou

Felix Geisendörfer 17 Oct 31, 2022
A Distributed Continuous Integration System from MongoDB

Evergreen Evergreen is a distributed continuous integration system built by MongoDB. It dynamically allocates hosts to run tasks in parallel across ma

Evergreen 370 Nov 28, 2022
Golang binaries compiled on-demand for your system

Go Binaries Go Binaries is an on-demand binary server, allowing non-Go users to quickly install tools written in Go without installing go itself, and

TJ Holowaychuk 761 Nov 25, 2022
A reproducible example for an issue I'm encoutering when deploying my gunicorn app to nginx

Ngnix Issue This is a minimum, reproducible example for an issue I'm encoutering when deploying my gunicorn app to nginx. Installation (Linux) Webserv

Space Simulation Services of Canada Corporation 1 May 29, 2022
A command line tool that builds and (re)starts your web application everytime you save a Go or template fileA command line tool that builds and (re)starts your web application everytime you save a Go or template file

# Fresh Fresh is a command line tool that builds and (re)starts your web application everytime you save a Go or template file. If the web framework yo

null 0 Nov 22, 2021
high performance distributed task scheduling system, Support multi protocol scheduling tasks

high performance distributed task scheduling system, Support multi protocol scheduling tasks

null 67 Dec 2, 2022