Structured Concurrency in Go

Overview

nursery: structured concurrency in Go

GoDoc GoReportCard CircleCI gopherbadger-tag-do-not-edit

RunConcurrently(
    // Job 1
    func(context.Context, chan error) {
        time.Sleep(time.Millisecond * 10)
        log.Println("Job 1 done...")
    },
    // Job 2
    func(context.Context, chan error) {
        time.Sleep(time.Millisecond * 5)
        log.Println("Job 2 done...")
    },
)
log.Println("All jobs done...")

Installation

go get -u github.com/arunsworld/nursery

Notes on structured concurrency, or: Go statement considered harmful is an article that compares the dangers of goto with the go statement.

While I don't necessarily agree with the entire content I can appreciate that even with Go's high-level abstraction of concurrency using Goroutines, Channels & the select statement it is possible to end up with unreadable code, deadlocks, leaked goroutines, race conditions and poor error handling.

Implementing a higher-level abstraction for the use-cases mentioned is very straightforward in Go and this simple package provides just that.

The following functions are provided:

  • RunConcurrently(jobs ...ConcurrentJob) error: takes an array of ConcurrentJobs and runs them concurrently ensuring that all jobs are completed before the call terminates. If all jobs terminate cleanly error is nil; otherwise the first non-nil error is returned.
  • RunConcurrentlyWithContext(parentCtx context.Context, jobs ...ConcurrentJob) error: is the RunConcurrently behavior but additionally wraps a context that's passed in allowing cancellations of the parentCtx to get propagated.
  • RunMultipleCopiesConcurrently(copies int, job ConcurrentJob) error: makes copies of the given job and runs them concurrently. This is useful for cases where we want to execute multiple slow consumers taking jobs from a channel until the job is finished. The channel itself can be fed by a producer that is run concurrently with the job running the consumers. Each job's context is also passed an unique index with key nursery.JobID - a 0 based int - that maybe used as a job identity if required.
  • RunMultipleCopiesConcurrentlyWithContext(ctx context.Context, copies int, job ConcurrentJob) error: is the RunMultipleCopiesConcurrently behavior with a context that allows cancellation to be propagated to the jobs.
  • RunUntilFirstCompletion(jobs ...ConcurrentJob) error: takes an array of ConcurrentJobs and runs them concurrently but terminates after the completion of the earliest completing job. A key point here is that despite early termination it blocks until all jobs have terminated (ie. released any used resources). If all jobs terminate cleanly error is nil; otherwise the first non-nil error is returned.
  • RunUntilFirstCompletionWithContext(parentCtx context.Context, jobs ...ConcurrentJob) error: is the RunUntilFirstCompletion behavior but additionally wraps a context that's passed in allowing cancellations of the parentCtx to get propagated.
  • RunConcurrentlyWithTimeout(timeout time.Duration, jobs ...ConcurrentJob) error: is similar in behavior to RunConcurrently except it also takes a timeout and can cause the function to terminate earlier if timeout has expired. As before we wait for all jobs to have cleanly terminated.
  • RunUntilFirstCompletionWithTimeout(timeout time.Duration, jobs ...ConcurrentJob) error: is similar in behavior to RunUntilFirstCompletion with an additional timeout clause.

ConcurrentJob is a simple function that takes a context and error channel. We need to ensure that we're listening to the Done() channel on context and if invoked to clean-up resources and bail out. Errors are to be published to the error channel for proper handling.

Note: while this package simplifies the semantics of defining and executing concurrent code it cannot protect against bad concurrent programming such as using shared resources across jobs leading to data corruption or panics due to race conditions.

You may also be interested in reading Structured Concurrency in Go.

The library includes a utility function: IsContextDone(context.Context) to check if the passed in context is done or not. This can be used as a guard clause in a for loop within a ConcurrentJob using the passed in context to decide whether to stop processing and return or continue.

You might also like...
Code Generation for Functional Programming, Concurrency and Generics in Golang

goderive goderive derives mundane golang functions that you do not want to maintain and keeps them up to date. It does this by parsing your go code fo

A sync.WaitGroup with error handling and concurrency control

go-waitgroup How to use An package that allows you to use the constructs of a sync.WaitGroup to create a pool of goroutines and control the concurrenc

golang worker pool , Concurrency limiting goroutine pool

golang worker pool 中文说明 Concurrency limiting goroutine pool. Limits the concurrency of task execution, not the number of tasks queued. Never blocks su

gpool - a generic context-aware resizable goroutines pool to bound concurrency based on semaphore.

gpool - a generic context-aware resizable goroutines pool to bound concurrency. Installation $ go get github.com/sherifabdlnaby/gpool import "github.c

Concurrency limiting goroutine pool

workerpool Concurrency limiting goroutine pool. Limits the concurrency of task execution, not the number of tasks queued. Never blocks submitting task

concurrency utilities

concurrent concurrent.Map: backport sync.Map for go below 1.9 concurrent.Executor: goroutine with explicit ownership and cancellable concurrent.Map be

Pholcus is a distributed high-concurrency crawler software written in pure golang
Pholcus is a distributed high-concurrency crawler software written in pure golang

Pholcus Pholcus(幽灵蛛)是一款纯 Go 语言编写的支持分布式的高并发爬虫软件,仅用于编程学习与研究。 它支持单机、服务端、客户端三种运行模式,拥有Web、GUI、命令行三种操作界面;规则简单灵活、批量任务并发、输出方式丰富(mysql/mongodb/kafka/csv/excel等

Helper library for full uint64 randomness, pool backed for efficient concurrency

fastrand64-go Helper library for full uint64 randomness, pool backed for efficient concurrency Inspired by https://github.com/valyala/fastrand which i

dagger is a fast, concurrency safe, mutable, in-memory directed graph library with zero dependencies
dagger is a fast, concurrency safe, mutable, in-memory directed graph library with zero dependencies

dagger is a blazing fast, concurrency safe, mutable, in-memory directed graph implementation with zero dependencies

Concurrency  in Go video course with in depth explanations & examples
Concurrency in Go video course with in depth explanations & examples

Concurrency in Go Summary Coding Examples Introduction to Concurrency Go Routines Channels Select Concurrency Patterns Atomics Wait Groups - sync.Wait

yakv is a simple, in-memory, concurrency-safe key-value store for hobbyists.
yakv is a simple, in-memory, concurrency-safe key-value store for hobbyists.

yakv (yak-v. (originally intended to be "yet-another-key-value store")) is a simple, in-memory, concurrency-safe key-value store for hobbyists. yakv provides persistence by appending transactions to a transaction log and restoring data from the transaction log on startup.

This repository collects common concurrency patterns in Golang

Go Concurrency Patterns This repository collects common concurrency patterns in Golang Materials Concurrency is not parallelism Go Concurrency Pattern

Concurrency patterns in Go

Concurrency patterns in Go

 File Processor in Concurrency Pattern using Golang goroutine.
File Processor in Concurrency Pattern using Golang goroutine.

File Processor in Concurrency Pattern Implement a file processor solution in concurrency pattern using Golang goroutine. Get Started Run docker-compos

Concurrency Lab 2 Go Example
Concurrency Lab 2 Go Example

Concurrency Lab 2 If you're stuck look at examples on Go by Example Using the lab sheet There are two ways to use the lab sheet, you can either: creat

Concurrency Lab 1 Go Example
Concurrency Lab 1 Go Example

Concurrency Lab 1 If you're stuck look at examples on Go by Example Using the lab sheet There are two ways to use the lab sheet, you can either: creat

Concurrency Lab examples on Go
Concurrency Lab examples on Go

Concurrency Lab 1 If you're stuck look at examples on Go by Example Using the lab sheet There are two ways to use the lab sheet, you can either: creat

Go lang concurrency with channels and worker pools using sync.Pools.

Go Concurrency Go lang concurrency with channels and worker pools using sync.Pools. Concurrency Concurrency is an ability of a program to do multiple

A CLI based Downloader utilising Golang's concurrency feature.

Downloader CLI based downloader utilising the concurrency feature of Golang. Takes the download URL of the file from the user. Splits the files into s

Comments
  • Renaming proposal.

    Renaming proposal.

    Hi.

    Having read the very same article and after some thinking I thought of implementing the very same library but luckily googled first. Congrats on making it before me :)

    But I would suggest making the names shorter as the Go as a language tries to make everything concise and even the variable names are encouraged to be single letter in certain cases.

    I observe the following problems (from my humble point of view) with the current naming (nursery.RunConcurrently()).

    1. The nursery term usually refers to raising children, but since we run our goroutines to their very death, it's probably not perfect.
    2. The RunConcurrently is already two words and just from the sound of it it's not clear, if the goroutines would be still running from after that
    3. The "nursery run parallel" is not really a phrase.

    My ideas on how to name it would be:

    1. sync.Do(), though sync is taken so might cause problems for some people and still fails point (3)
    2. await.Run() - that seems to work around problem (2) due to package name choice
    3. scoped.Run() - feels a little better than previous one, tough the workaround for (2) is not there already.
    4. ...
    opened by snakeru 1
Owner
null
golang worker pool , Concurrency limiting goroutine pool

golang worker pool 中文说明 Concurrency limiting goroutine pool. Limits the concurrency of task execution, not the number of tasks queued. Never blocks su

xxj 424 Nov 16, 2022
gpool - a generic context-aware resizable goroutines pool to bound concurrency based on semaphore.

gpool - a generic context-aware resizable goroutines pool to bound concurrency. Installation $ go get github.com/sherifabdlnaby/gpool import "github.c

Sherif Abdel-Naby 86 Oct 31, 2022
Concurrency limiting goroutine pool

workerpool Concurrency limiting goroutine pool. Limits the concurrency of task execution, not the number of tasks queued. Never blocks submitting task

Andrew Gillis 952 Nov 21, 2022
This repository collects common concurrency patterns in Golang

Go Concurrency Patterns This repository collects common concurrency patterns in Golang Materials Concurrency is not parallelism Go Concurrency Pattern

Kha Nguyen 2k Nov 27, 2022
Pengenalan Concurrency dan Parallel Programming

Golang Goroutine Sumber Tutorial: Udemy Slide Pengenalan Concurrency dan Parallel Programming Pengenalan Parallel Programming Saat ini kita hidup dima

Akbarputra 0 Nov 5, 2021
Go-concurrency-patterns - Sample concurrency patterns with Goroutines

About This sample project provides some concurrency pattern examples in Go using

George Batagiannis ⚡ 1 Feb 21, 2022
Structured Concurrency in Go

nursery: structured concurrency in Go RunConcurrently( // Job 1 func(context.Context, chan error) { time.Sleep(time.Millisecond * 10)

null 52 Sep 26, 2022
Concurrency-safe Go caching library with expiration capabilities and access counters

cache2go Concurrency-safe golang caching library with expiration capabilities. Installation Make sure you have a working Go environment (Go 1.2 or hig

Christian Muehlhaeuser 1.9k Nov 23, 2022
A simple logging interface that supports cross-platform color and concurrency.

WLog Package wlog creates simple to use UI structure. The UI is used to simply print to the screen. There a wrappers that will wrap each other to crea

Will Dixon 59 Sep 26, 2022
Concurrency-safe Go caching library with expiration capabilities and access counters

cache2go Concurrency-safe golang caching library with expiration capabilities. Installation Make sure you have a working Go environment (Go 1.2 or hig

Christian Muehlhaeuser 1.9k Nov 23, 2022