parallel: a Go Parallel Processing Library

Related tags

parallel
Overview

parallel: a Go Parallel Processing Library

Build Status codecov Go Report Card GoDoc

Concurrency is hard. This library doesn't aim to make it easy, but it will hopefully make it a little less painful.

Install

This library should be compatible with all recent and future versions of Go, and has no third party dependencies.

go get -u github.com/ryanskidmore/parallel

You can then import the library

import "github.com/ryanskidmore/parallel"

Testing

This library uses the standard Go testing tools, and doesn't use any third party testing libraries.

go test

Quick Start

package main

import (
    "log"
    "fmt"

    "github.com/ryanskidmore/parallel"
)

func main() {
    p := parallel.New() // Create a new instance of parallel
    worker, err := p.NewWorker("worker1", &parallel.WorkerConfig{Parallelism: 1}) // Create a new worker
    if err != nil {
        log.Fatalf("FATAL: Failed to create new worker: %v", err)
    }
    worker.SetExecution(func(wh *parallel.WorkerHelper, args interface{}) { // Set the worker execution function
        fmt.Println(args)
        wh.Done()
    })
    worker.Start(interface{}("Test String")) // Start with worker with specified args
    worker.Wait() // Wait for the worker to complete
}

Docs

The best source of reference is the GoDocs for this library. Noted below are parts of the library that may not be immediately obvious from the docs or otherwise.

WorkerHelper

The WorkerHelper struct gives you access to a WaitGroup via WorkerHelper.Done(). When you call Worker.Wait() this waits on the WorkGroup for that worker and will block until every instance of the worker has called WorkerHelper.Done().

The WorkerHelper can also be used to consume and publish data to/from other workers/goroutines and this is done using DataChannels.

DataChannels

DataChannels are intended to be a method of publishing and consuming data between different workers and goroutines.

Before using a DataChannel in an execution function, it must first be initialised using:

err := p.NewDataChannel("name")

Once the DataChannel has been initialised, it can be published to by calling:

err := WorkerHelper.PublishData("name", data)

This is an asynchronous operation, so will not block execution.

Data can be consumed from the DataChannel via either calling:

data, err := WorkerHelper.ConsumeData("name")

or

data, err := WorkerHelper.ConsumeDataInBatches("name", 20)

These functions will return an error when the DataChannel doesn't exist or when the channel is closed.

Examples

Examples of usage can be found in the examples directory.

Releases(v0.0.1)
Owner
Ryan Skidmore
Ryan Skidmore
gpool - a generic context-aware resizable goroutines pool to bound concurrency based on semaphore.

gpool - a generic context-aware resizable goroutines pool to bound concurrency. Installation $ go get github.com/sherifabdlnaby/gpool import "github.c

Sherif Abdel-Naby 81 Aug 20, 2021
🚦 Semaphore pattern implementation with timeout of lock/unlock operations.

?? semaphore Semaphore pattern implementation with timeout of lock/unlock operations. ?? Idea The semaphore provides API to control access to a shared

Kamil Samigullin 88 Sep 15, 2021
Unlimited job queue for go, using a pool of concurrent workers processing the job queue entries

kyoo: A Go library providing an unlimited job queue and concurrent worker pools About kyoo is the phonetic transcription of the word queue. It provide

Dirk Pahl 34 Sep 11, 2021
A goroutine pool for Go

Tunny is a Golang library for spawning and managing a goroutine pool, allowing you to limit work coming from any number of goroutines with a synchrono

Ashley Jeffs 2.6k Sep 14, 2021
:speedboat: a limited consumer goroutine or unlimited goroutine pool for easier goroutine handling and cancellation

Package pool Package pool implements a limited consumer goroutine or unlimited goroutine pool for easier goroutine handling and cancellation. Features

Go Playgound 634 Sep 5, 2021
Set up tasks to be executed in parallel.

A simple Go library to set up tasks to be executed in parallel. package main import ( "context" "log" "github.com/bep/workers" ) func main() {

Bjørn Erik Pedersen 8 Aug 19, 2021
Reactive Extensions for the Go language.

RxGo Reactive Extensions for the Go Language ReactiveX ReactiveX, or Rx for short, is an API for programming with Observable streams. This is the offi

ReactiveX 3.7k Sep 10, 2021
go routine control, abstraction of the Main and some useful Executors.如果你不会管理Goroutine的话,用它

routine Routine Architecture Quick Start package main import ( "log" "context" "github.com/x-mod/routine" ) func main(){ if err := routine.Main

null 44 Sep 8, 2021
Floc: Orchestrate goroutines with ease.

go-floc Floc: Orchestrate goroutines with ease. The goal of the project is to make the process of running goroutines in parallel and synchronizing the

Andrew Bashkatov 208 Sep 10, 2021
Concurrency limiting goroutine pool

workerpool Concurrency limiting goroutine pool. Limits the concurrency of task execution, not the number of tasks queued. Never blocks submitting task

Andrew Gillis 630 Sep 7, 2021
Structured Concurrency in Go

nursery: structured concurrency in Go RunConcurrently( // Job 1 func(context.Context, chan error) { time.Sleep(time.Millisecond * 10)

null 34 Aug 30, 2021
Routines was a fixed number thread pool to process the user task, and it would respawn a corresponding new thread when panic

Routines Routines was a fixed number thread pool to process the user task, and it would respawn a corresponding new thread when panic. It supports the

hulk 10 Sep 1, 2021
goworker is a Go-based background worker that runs 10 to 100,000* times faster than Ruby-based workers.

goworker goworker is a Resque-compatible, Go-based background worker. It allows you to push jobs into a queue using an expressive language like Ruby w

Benjamin Manns 2.6k Sep 14, 2021
Lightweight Goroutine pool

grpool Lightweight Goroutine pool Clients can submit jobs. Dispatcher takes job, and sends it to first available worker. When worker is done with proc

Ivan Pusic 653 Sep 10, 2021
Run tasks concurrently with limits

Workerpool Package workerpool implements a concurrency limiting worker pool. Worker routines are spawned on demand as tasks are submitted. This packag

Cilium 12 Jun 9, 2021
errgroup with goroutine worker limits

neilotoole/errgroup neilotoole/errgroup is a drop-in alternative to Go's wonderful sync/errgroup but limited to N goroutines. This is useful for inter

Neil O'Toole 90 Sep 5, 2021
Minimalistic and High-performance goroutine worker pool written in Go

pond Minimalistic and High-performance goroutine worker pool written in Go Motivation This library is meant to provide a simple way to limit concurren

Alejandro Durante 275 Sep 16, 2021
Provides some convenient API, includes Goid(), AllGoid(), and LocalStorage, which is a goroutine's local storage, just like ThreadLocal in other languages.

routine 中文版 routine encapsulates and provides some easy-to-use, high-performance goroutine context access interfaces, which can help you access corout

null 23 Sep 13, 2021
Alternative sync library for Go

Alternative sync library for Go. Overview Future - A placeholder object for a value that may not yet exist. Promise - While futures are defined as a t

Eugene R. 24 Aug 4, 2021