Create build pipelines in Go

Related tags

Build Tools go golang
Overview

taskflow

Create build pipelines in Go

GitHub Release Go Reference go.mod Build Status Go Report Card codecov Mentioned in Awesome Go

This package aims to simplify the creation of build pipelines in Go instead of using scripts or Make.

taskflow API is mainly inspired by the testing, http and flag packages.

Check Go Build Pipeline Demo to compare taskflow with Make and Mage.

Star this repository if you find it valuable and worth maintaining.

Table of Contents:

Example

Create a file in your project build/build.go.

Copy and paste the content from below.

package main

import "github.com/pellared/taskflow"

func main() {
	flow := taskflow.New()

	fmt := flow.MustRegister(taskFmt())
	test := flow.MustRegister(taskTest())

	flow.MustRegister(taskflow.Task{
		Name:        "all",
		Description: "build pipeline",
		Dependencies: taskflow.Deps{
			fmt,
			test,
		},
	})

	flow.Main()
}

func taskFmt() taskflow.Task {
	return taskflow.Task{
		Name:        "fmt",
		Description: "go fmt",
		Command:     taskflow.Exec("go", "fmt", "./..."),
	}
}

func taskTest() taskflow.Task {
	return taskflow.Task{
		Name:        "test",
		Description: "go test with race detector and code covarage",
		Command: func(tf *taskflow.TF) {
			if err := tf.Cmd("go", "test", "-race", "-covermode=atomic", "-coverprofile=coverage.out", "./...").Run(); err != nil {
				tf.Errorf("go test: %v", err)
			}
			if err := tf.Cmd("go", "tool", "cover", "-html=coverage.out", "-o", "coverage.html").Run(); err != nil {
				tf.Errorf("go tool cover: %v", err)
			}	
		},
	}
}

Sample usage:

$ go run ./build -h
Usage: [flag(s)] [key=val] task(s)
Flags:
  -v    Verbose output: log all tasks as they are run. Also print all text from Log and Logf calls even if the task succeeds.
Tasks:
  all     build pipeline
  fmt     go fmt
  test    go test with race detector and code covarage
$ go run ./build all
ok     0.453s
$ go run ./build -v all
===== TASK  fmt
Exec: go fmt ./...
----- PASS: fmt (0.06s)
===== TASK  test
Exec: go test -race -covermode=atomic -coverprofile=coverage.out ./...
?       github.com/pellared/taskflow/example    [no test files]
Exec: go tool cover -html=coverage.out -o coverage.html
----- PASS: test (0.11s)
ok      0.176s

Tired of writing go run ./build each time? Just add an alias to your shell. For example, add the line below to ~/.bash_aliases:

alias gake='go run ./build'

Additionally, take a look at this project's own build pipeline script - build.go.

Features

Task registration

The registered tasks are required to have a non-empty name. For future compatibility, it is strongly suggested to use only the following characters:

  • letters (a-z and A-Z)
  • digits (0-9)
  • underscore (_)
  • hyphens (-)

Do not use the equals sign (=) as it is used for assigning parameters.

A task with a given name can be only registered once.

A task without description is not listed in taskflow's CLI usage.

Task dependencies

During task registration it is possible to add a dependency to an already registered task. When taskflow is processed, it makes sure that the dependency is executed before the current task is run. Take note that each task will be executed at most once.

Task command

Task command is a function which is executed when a task is executed. It is not required to to set a command. Not having a command is very handy when registering "pipelines".

Task runner

You can use type Runner to execute a single command.

It may be handy during development of a new task, when debugging some issue or if you want to have a test suite for reusable commands.

Verbose mode

Enable verbose output using the -v CLI flag. It works similar to go test -v. When enabled, the whole output will be streamed. If disabled, only logs from failed task are send to the output.

Use func (*TF) Verbose to check if verbose mode was set within the task's command.

Parameters

The parameters can be set via CLI using the key=val syntax after CLI flags. For example, go run ./build -v ci=true all would run the all task with ci parameter set to "true" in verbose mode.

Default values can be assigned via Params field in type Taskflow.

The task's command can get the parameters using func (*TF) Params.

Helpers for running programs

Use func Exec(name string, args ...string) func(*TF) to create a task's command which only runs a single program.

Use func (tf *TF) Cmd(name string, args ...string) *exec.Cmd if within a task's command function when you want to execute more programs or you need more granular control.

Supported Go versions

Minimal required Go version is 1.10.

FAQ

Why not use Make?

While Make is currently the de facto standard, it has some pitfalls:

  • Requires to learn Make, which is not so easy.
  • It is hard to develop a Makefile which is truly cross-platform.
  • Debugging and testing Make targets is not fun.

However, if you (and your team) know Make and are happy with it, do not change it.

Make is very powerful and a lot of stuff can be made a lot faster, if you know how to use it.

taskflow is intended to be simpler and easier to learn, while still being able to handle most use cases.

Why not use Mage?

taskflow is intended to be an alternative to Mage.

Mage is a framework/tool which magically discovers the targets from magefiles.

taskflow takes a different approach as it is a regular Go library (package).

This results in following benefits:

  • It is easy to debug. Like a regular Go application.
  • Tasks and helpers are testable. See exec_test.go.
  • Reusing tasks is easy and readable. Just create a function which registers common tasks. Mage does it in a hacky way.
  • API similar to testing so it is possible to use e.g. testify for asserting.

To sum up, taskflow is not magical. Write regular Go code. No build tags or special names for functions.

Why not use Task?

While Task is simpler and easier to use than Make it still has some problems:

  • Requires to learn Task's YAML sturcture and the minimalistic, cross-platform interpreter which it uses.
  • Debugging and testing tasks is not fun.
  • Harder to make some reusable tasks.
  • Requires to "install" the tool. taskflow leverages go run and Go Modules so that you can be sure that everyone uses the same version of taskflow.

Why not use Bazel?

Bazel is a very sophisticated tool which is created to efficiently handle complex and long-running build pipelines. It requires the build target inputs and outputs to be fully specified.

taskflow is just a simple library that is mainly supposed to create a build pipeline consisting of commands like go vet, go test, go build. However, take notice that taskflow is a library. Nothing prevents you from, for example, using Mage's target package to make your build pipeline more efficient.

Contributing

I am open to any feedback and contribution.

Use Discussions or write to me: Robert Pajak @ Gophers Slack.

You can also create an issue or a pull request.

Comments
  • Rework output handling

    Rework output handling

    Why

    This PR is a proposal to better separate stdout/stderr usage. Primary motivation for this was the recommendation from clig.dev to send messages to stderr and have stdout only for net information from the tasks/tools. Since the change is an incompatible one, further, related incompatibilities are included, see below.

    What

    1. Expand Taskflow.Output as a separate type having two writer, one for Primary output, one for Message output. (incompatible API change)
    2. Goyek code was modified to only use Message output. (incompatible user-interface change) As a side-effect, any fmt.Fprintf error "handling" has been centralized and explicitly skipped.
    3. Removed the DefaultOutput global variable. It seems as if this was only used for the tests. Tests should override this explicitly anyway, to avoid side-effects. Furthermore, such global defaults are tricky; I refer to the potential problems as seen with http.Client and the transport defaults. (incompatible API change)

    Documentation/Changelog has not yet been updated, would do so if proposed changes are discussed and settled.

    Checklist

    • [ ] CHANGELOG.md is updated.
    • [ ] README.md is updated.
    • [x] The code changes follow Effective Go.
    opened by dertseha 12
  • Feature: registered parameters

    Feature: registered parameters

    This PR is the result of #37 .

    Thinking further, I figured that the registered parameters could be tracked by Go's built-in flags system. This handles usage output, as well as invalid parameter handling out of the box; As well as having less code to handle within this library.

    Example output of go run build\build.go -h (output shortened to highlight diff):

    Usage: [flag(s)] task(s)
    Flags:
      -ci string
            Whether CI is calling the build script (default "false")
      -v    Verbose output: log all tasks as they are run. Also print all text from Log and Logf calls even if the task succeeds.
    Tasks:
      ...
      diff        git diff; -ci
      ...
    

    This is a breaking change as Taskflow.Params is dropped and tasks need to specify which parameters they read (a change as intended).

    More thoughts on these parameters:

    • I wish the type information could be saved. As it is now, implementation of TFParams mirrors all the parsing what flag package has internally for the typed parameters. As it stands, the current setup does not support keeping the type information. Unless the type information of the parameter (string, int, ...) is carried over to a RegisteredParamString, RegisteredParamInt, and so forth - though this generates even more code.
    • Is there big of a need of a built-in support for text/json parsing of parameters? This sounds a bit overengineered to me.

    Most of the tests could carry over or needed only slight modification. Configure and MustConfigure aren't covered yet.

    How does this look?

    opened by dertseha 12
  • Add possibility to change the OOTB parameters

    Add possibility to change the OOTB parameters

    Why

    I wanted a method to set the default values for OOTB params which wasn't possible, because if you tried to register a verbose parameter again, goyek would panic because it was already registered.

    What

    I added a new method to create a flow (NewFlow) which accepts an Options-struct in which the defaults can be set.

    Checklist

    • [x] CHANGELOG.md is updated.
    • [x] README.md is updated.
    • [x] The code changes follow Effective Go.
    opened by arnovanliere 11
  • Use HTML comment syntax in the PR template

    Use HTML comment syntax in the PR template

    Why

    Why this pull request was created? Explain the problem that this pull request is trying to solve.

    The current pull request template:

    • requires to add new lines manually (if not the explanation is be added to the section comments):
      • write: write
      • preview: preview
    • doesn't inform what to do with the checklist items when the mentioned points are not applicable to the introduced changes

    What

    What does this pull request contain? Explain what is this pull request changing.

    Add comments to the pull request template so filling it is easier. The comment in the checklist informs that when no changes are required, the items should be checked (better visibility). The comments are not visible in the preview mode, so all new PRs will look the same as the older one.

    Checklist

    • [x] CHANGELOG.md is updated.
    • [x] README.md is updated.
    • [x] The code changes follow Effective Go.
    opened by agabrys 9
  • Feature: Registered parameters

    Feature: Registered parameters

    Continuation of #39 .

    Main points:

    • Parameters need to be registered in order to be used
    • Registered parameters are described in usage output
    • Parameters are provided uniformly via flags (either as --longform or, optionally, -l) -- see https://clig.dev for inspiration

    Note: This is an incompatible change, for both the CLI and the Go interface, in line with current v0 of Taskflow.

    opened by dertseha 9
  • Draft: Option to mark params as required

    Draft: Option to mark params as required

    Why

    Params could not be marked as required, so for every parameter you had to check whether it had a value or not inside the task handler. It seems like basic functionality for a task runner to be able to mark params as required.

    What

    It is a draft pull request to get feedback on the way I've set the validation up. I've made it so only the params which are used in the task which is being run, are checked whether they are required. All params which are required, but not set, are printed and the task fails.

    If a maintainer or other contributor thinks this is the right direction for such a feature, I will write tests, update the CHANGELOG etc, but I wanted to have feedback first.

    Checklist

    • [ ] CHANGELOG.md is updated.
    • [ ] README.md is updated.
    • [x] The code changes follow Effective Go.
    • [ ] The code changes are covered by tests.
    opened by arnovanliere 8
  • Change API

    Change API

    Why

    Partially addresses https://github.com/goyek/goyek/issues/62

    What

    • Rename Task.Command field to Action to avoid confusion with exec.Command and TF.Cmd.
    • Rename Taskflow type to Flow to make the name simpler.
    • Rename TF type to Progress to name the type more meaningful.
    • Remove DefaultOutput global variable to reduce the API surface.
    • Remove TF.Exec method to reduce the API surface.

    Checklist

    • [x] CHANGELOG.md is updated.
    • [x] README.md is updated.
    • [x] The code changes follow Effective Go.
    opened by pellared 6
  • Update wrapper scripts in README.md

    Update wrapper scripts in README.md

    Why

    Why this pull request was created? Explain the problem that this pull request is trying to solve.

    README.md has not up-to-date wrapper scripts.

    What

    What does this pull request contain? Explain what is this pull request changing.

    Update wrapper scripts in README.md.

    Checklist

    • [x] CHANGELOG.md is updated.
    • [x] README.md is updated.
    • [x] The code changes follow Effective Go.
    opened by pellared 6
  • Test against Go 1.17

    Test against Go 1.17

    Why

    Go 1.17 is released

    What

    Add Go 1.17 to GH workflows.

    Checklist

    • [x] CHANGELOG.md is updated.
    • [x] README.md is updated.
    • [x] The code changes follow Effective Go.
    opened by pellared 5
  • Alternative method signature for Exec / Simplified Error handling

    Alternative method signature for Exec / Simplified Error handling

    Hi,

    I became aware of this by the Golang Weekly newsletter and I like the idea of goyek. I started to play around with it and I would like to start a discussion about two potential changes:

    Signature change for goyek.Exec

    Change the signature for goyek.Exec from

    func Exec(name string, args ...string) func(*TF)
    

    to

    func Exec(cmdLine string) func(*TF)
    

    This is inspired by github.com/bitfield/script (see https://github.com/bitfield/script/blob/9847d19e69d72d73b191e1f3c65d5dba94605a1c/filters.go#L123).

    The main advantage of this change to me is, that it increases readability and it simplifies the process of copying back and forth of the shell commands between goyek and an actual shell (or a Makefile/Shell script one is migrating). By using the backticks for the string in Go, in most cases there would be no need to modify the command while copying back and forth.

    Error handling

    I could imagine, that a task often has a pattern like this:

    res, err := someFunc()
    if err != nil {
    	tf.Error(err)
    }
    ...
    

    Since goyek is somewhat inspired by testing.T (and its relatives), I though about how I nowadays handle errors in tests. I highly prefere the very simple https://github.com/matryer/is package, which provides me a method for error handling like this (https://pkg.go.dev/github.com/matryer/is#I.NoErr):

    is.NoErr(err)
    

    If the error is nil, the normal flow continues. If the error is non nil, the tests are ended with t.Fail().

    For goyek I could imagine two methods with the following signature:

    func (*goyek.TF).IfErr(err error, args ...interface{})
    func (*goyek.TF).IfErrf(err error, format string, args ...interface{})
    

    For the exact naming there are several options:

    • tf.IfErr
    • tf.IfError
    • tf.IsErr
    • tf.IsNoErr
    • ...

    Thanks again for your initiative, keep up the good work.

    documentation 
    opened by breml 5
  • Move documentation to the docs directory

    Move documentation to the docs directory

    Why

    Why this pull request was created? Explain the problem that this pull request is trying to solve.

    Avoid clutter in the "frontpage" of the repository

    Reference: https://github.com/goyek/goyek/discussions/58#discussioncomment-782455

    What

    What does this pull request contain? Explain what is this pull request changing.

    • Move documentation to the docs directory. See: https://github.com/pellared/goyek/tree/docs
    • Pin versions in build GH workflow (could be done as a separate PR in theory)

    Checklist

    • [x] CHANGELOG.md is updated.
    • [x] README.md is updated.
    • [x] The code changes follow Effective Go.
    opened by pellared 4
  • Run tasks concurrently

    Run tasks concurrently

    Why

    I'm looking at goyek as an alternative to Mage - thanks for open sourcing this, so far I am loving the API as it feels very similar to the built in testing library and to libraries like urfav/cli.

    Something that I liked in Mage is that it was very easy to run tasks concurrently. For example, if task all depends on tasks a and b, tasks a and b would be executed concurrently in separate Goroutines. I tried modelling this in goyek as follows:

    package main
    
    import (
    	"time"
    
    	"github.com/goyek/goyek"
    )
    
    func main() {
    	flow := &goyek.Flow{}
    
    	a := flow.Register(goyek.Task{
    		Name: "a",
    		Action: func(tf *goyek.TF) {
    			time.Sleep(time.Second * 1)
    			tf.Log("a")
    			time.Sleep(time.Second * 1)
    			tf.Log("a")
    			time.Sleep(time.Second * 1)
    			tf.Log("a")
    		},
    	})
    
    	b := flow.Register(goyek.Task{
    		Name: "b",
    		Action: func(tf *goyek.TF) {
    			time.Sleep(time.Second * 1)
    			tf.Log("b")
    			time.Sleep(time.Second * 1)
    			tf.Log("b")
    			time.Sleep(time.Second * 1)
    			tf.Log("b")
    		},
    	})
    
    	flow.Register(goyek.Task{
    		Name: "all",
    		Deps: goyek.Deps{a, b},
    	})
    
    	flow.Main()
    }
    

    It appears that the tasks run sequentially, even though a and b don't depend on one another.

    go run ./build all -v
    ===== TASK  a
    a
    a
    a
    ----- PASS: a (3.00s)
    ===== TASK  b
    b
    b
    b
    ----- PASS: b (3.00s)
    ok      6.006s
    

    What

    Would you be open to a contribution to support running tasks concurrently?

    Some approaches to the API could be to add a flag at the Flow level:

    flow := &goyek.Flow{
    	Concurrent: true,
    }
    
    or to allow concurrency to be specified at a task level:
    
    ```go
    flow.Register(goyek.Task{
    	Name: "all",
    	Deps: goyek.ConcurrentDeps{a, b},
    })
    
    enhancement help wanted 
    opened by chrnorm 1
  • Add TF.Run() for running sub-tasks

    Add TF.Run() for running sub-tasks

    Why

    Like testing package it would be very helpful to create something like sub-tasks.

    Example use case: run go test for each Go Module within a repository.

    How

    Implement TF.Run() method which works similarly to the one in the testing package.

    We can also dogfood it e.g. to run golangci-lint for all Go modules within this repository.

    We could also support running a concrete subtask e.g ./goyek Task/Sub1 (separate PR).

    We could also try to support running tasks and subtasks using regex/wildcards (separate PR). go test -run regex takes regex . If we decide to take the same approach we wrap the regex with ^ $.

    enhancement help wanted 
    opened by pellared 5
  • TF Log(f), Error(f), Fatal(f), Skip(f), methods add file and line information

    TF Log(f), Error(f), Fatal(f), Skip(f), methods add file and line information

    Why

    Like testing package it would be very helpful if the "printing" methods add file and line information.

    For example, it will help to find where exactly a task is failing.

    How

    1. The TF's Log(f), Error(f), Fatal(f), Skip(f) add file and line information
    2. Add TF.Helper() method which works like the one for testing package.

      Helper marks the calling function as a test helper function. When printing file and line information, that function will be skipped. Helper may be called simultaneously from multiple goroutines.

    Remarks:

    1. Printing via tf.Output() MUST NOT print file and line information
    2. Reference implementation: https://github.com/golang/go/blob/master/src/testing/testing.go

    We can consider also adding a field like Taskflow.DisableCodeline to provide a possibility to disable such functionality. However, I would rather not add it until someone really needs it. One can always use tf.Output() to print without file and line information.

    enhancement help wanted 
    opened by pellared 6
Releases(v1.0.0)
  • v1.0.0(Sep 8, 2022)

  • v0.6.3(Feb 23, 2022)

    Added

    • TaskNamePattern and ParamNamePattern has been changed so that the plus (+) and colon (:) characters can be used in the task and parameter name (but not as the first character).
    Source code(tar.gz)
    Source code(zip)
  • v0.6.2(Jan 25, 2022)

  • v0.6.1(Dec 27, 2021)

    This release adds the possibility to change the defaults of the global parameters.

    Added

    • Add RegisterVerboseParam and RegisterWorkDirParam to overwrite default values for verbose and work dir parameters.
    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(Aug 2, 2021)

    This release contains multiple breaking changes in the Go API. It is supposed to make it cleaner.

    Added

    • Tf.Cmd also sets Stdin to os.Stdin.
    • Add Flow.Tasks and Flow.Params to allow introspection of the registered tasks and parameters.

    Changed

    • Rename Taskflow type to Flow to simplify the naming.
    • Rename Task.Command field to Action to avoid confusion with exec.Command and TF.Cmd.

    Removed

    • Remove DefaultOutput global variable.
    • Remove TF.Exec method.
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Jun 21, 2021)

    Added

    • Add the stack trace when a task panics.

    Changed

    • Names of parameters now have to follow goyek.ParamNamePattern. They were allowed to start with an underscore (_), and now no longer are.
    • The PowerShell wrapper scripts goyek.ps1 better handles stderr redirection.
    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(May 26, 2021)

    Added

    • Add Bash and PowerShell wrapper scripts.
    • Add -wd global parameter allowing to change the working directory. The new Taskflow.WorkDirParam method can be used to get its value in a task's command.
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(May 3, 2021)

    The repository has migrated to https://github.com/goyek/goyek.

    This release contains multiple breaking changes for both the CLI and the Go API.

    The biggest change is a redesign of the parameters API, so they have to be explicitly registered. It makes the usage of the parameters more controlled and provides a better help output. Moreover, the parameters are set via CLI using the flag syntax.

    Added

    • Help is printed when -h, --help or help is passed.
    • Help contains parameters' information.
    • The tasks and parameters can be passed via CLI in any order.
    • Taskflow.Run handles nil passed as context.Context argument.
    • Taskflow.Run panics when a registered parameter is not assigned to any task.

    Changed

    • Module path changed to github.com/goyek/goyek.
    • Rename package task to goyek.
    • Use the flag syntax for setting parameters via CLI.
    • Rename Task.Description field to Usage.
    • Rename Task.Dependencies field to Deps.
    • Rename CodeFailure constant to CodeFail.
    • Rename Taskflow.MustRegister method to Register and remove previous Taskflow.Register implementation.
    • Remove Taskflow.Params field and TF.Params method, add Taskflow.Register*Param methods and Task.Params field instead.
    • Remove TF.Verbose, add Taskflow.VerboseParam instead.
    • Remove New function, create instance using &Taskflow{} instead.
    • Unexport Runner type, use Taskflow in tests instead.
    • Enforce patterns for task names (TaskNamePattern) and parameter names (ParamNamePattern).
    • Drop official support for Go 1.10.
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Mar 14, 2021)

  • v0.1.1(Feb 28, 2021)

  • v0.1.0(Jan 14, 2021)

Owner
Robert Pająk
Backend developer https://www.linkedin.com/in/rpajak/
Robert Pająk
Colorize (highlight) `go build` command output

colorgo colorgo is a wrapper to go command that colorizes output from go build and go test. Installation go get -u github.com/songgao/colorgo Usage c

Song Gao 110 Sep 13, 2022
a build tool for Go, with a focus on cross-compiling, packaging and deployment

goxc NOTE: goxc has long been in maintenance mode. Ever since Go1.5 supported simple cross-compilation, this tool lost much of its value. There are st

Am Laher 1.7k Sep 19, 2022
EGo lets you build, debug und run Go apps on Intel SGX - as simple as conventional Go programming!

EGo is a framework for building confidential apps in Go. Confidential apps run in always-encrypted and verifiable enclaves on Intel SGX-enabled ha

Edgeless Systems GmbH 339 Sep 21, 2022
Build system and task runner for Go projects

Gilbert is task runner that aims to provide declarative way to define and run tasks like in other projects like Gradle, Maven and etc.

Gilbert 103 Sep 14, 2022
Build and (re)start go web apps after saving/creating/deleting source files.

unmaintained Fresh Fresh is a command line tool that builds and (re)starts your web application everytime you save a Go or template file. If the web f

Andrea Franz 3.4k Sep 22, 2022
KintoHub is an open source build and deployment platform designed with a developer-friendly interface for Kubernetes.

What is Kintohub? KintoHub is an open source build and deployment platform designed with a developer-friendly interface for Kubernetes. Build your cod

KintoHub 30 Jun 7, 2022
Build systems with Go examples

What is this? This is a repository containing all the examples from the book BUILD SYSTEMS with GO (and save the world). This book is written to help

Juan M. Tirado 92 Sep 27, 2022
🌍 Earthly is a build automation tool for the container era

?? Earthly is a build automation tool for the container era. It allows you to execute all your builds in containers. This makes them self-contained, repeatable, portable and parallel. You can use Earthly to create Docker images and artifacts (eg binaries, packages, arbitrary files).

Earthly 8k Sep 24, 2022
An experimental way to apply patches to the Go runtime at build time.

go-patch-overlay An experimental way to apply patches to the Go runtime at build time. Assuming you have a directory of patches to apply to the Go sou

Felix Geisendörfer 16 Feb 9, 2022
Please is a cross-language high-performance extensible build system for reproducible multi-language builds.

Please is a cross-language build system with an emphasis on high performance, extensibility and reproducibility. It supports a number of popular languages and can automate nearly any aspect of your build process.

Thought Machine 2.1k Sep 23, 2022
Blueprint Build System For Golang

Blueprint Build System Blueprint is being archived on 2021 May 3. On 2021 May 3, we will be archiving the Blueprint project. This means it will not be

GengKapak WIP 0 Nov 20, 2021
goyek is used to create build pipelines in Go

goyek is used to create build pipelines in Go. As opposed to many other tools, it is just a Go library.

null 308 Sep 26, 2022
A library to help you create pipelines in Golang

pipeline Pipeline is a go library that helps you build pipelines without worrying about channel management and concurrency. It contains common fan-in

Delivery Hero SE 113 Sep 18, 2022
Build powerful pipelines in any programming language.

Gaia is an open source automation platform which makes it easy and fun to build powerful pipelines in any programming language. Based on HashiCorp's g

Gaia 4.8k Sep 27, 2022
Pipelines using goroutines

pipeline This package provides a simplistic implementation of Go pipelines as outlined in Go Concurrency Patterns: Pipelines and cancellation. Docs Go

Nikhil Mungel 43 Aug 19, 2022
elPrep: a high-performance tool for analyzing sequence alignment/map files in sequencing pipelines.

Overview elPrep is a high-performance tool for analyzing .sam/.bam files (up to and including variant calling) in sequencing pipelines. The key advant

null 260 Sep 8, 2022
Robust, flexible and resource-efficient pipelines using Go and the commandline

Robust, flexible and resource-efficient pipelines using Go and the commandline Project links: Documentation & Main Website | Issue Tracker | Chat Why

SciPipe 959 Sep 23, 2022
🐺 Deploy Databases and Services Easily for Development and Testing Pipelines.

Peanut provides an API and a command line tool to deploy and configure the commonly used services like databases, message brokers, graphing tools ... etc. It perfectly suited for development, manual testing, automated testing pipelines where mocking is not possible and test drives.

Ahmed 667 Sep 2, 2022
🏯 Monitor your (gitlab/github) CI/CD pipelines via command line interface with fortress

__ _ / _| | | | |_ ___ _ __| |_ _ __ ___ ___ ___ | _/ _ \| '__| __| '__/ _ \/ __/ _

MrJosh 6 Mar 31, 2022
Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

Optimus Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality

Open Data Platform 649 Sep 27, 2022