Go worker pool

Related tags

Miscellaneous pool
Overview

description

This is a generic worker pool for the Go language. It's useful when you want to limit the number of goroutines running in parallel.

installation

go install github.com/stefantalpalaru/pool

compile and run the examples

go run examples/pool_example.go
go run examples/web_crawler.go

The last example is actually an exercise from the Go tour modified to use a worker pool for fetching and processing the URLs. The need to limit the number of concurrent requests in real web scraping scenarios was what prompted the creation of this package.

documentation

You can see the godoc-generated documentation online at godoc.org/github.com/stefantalpalaru/pool but it might be a bit dry.

The best documentation is in the form of an example: pool_example.go.

license

MPL-2.0

credits

Issues
  • Case for non-determistic number of jobs

    Case for non-determistic number of jobs

    hi - It doesn't look this project is not in active development, but I will ask anyway. I have a use case where the number of jobs is non-deterministic: in the sense that each Job may or may not generate new Jobs. The termination condition is no new jobs are generated by all submitted jobs. Can this pool structure support such use case? TIA.

    opened by python152 2
  • Any way to get the id of the worker

    Any way to get the id of the worker

    Hi Firstly, this is a nice utility. Thanx Qs: Im creating 5 workers using pool.New. How do I pass in the id lets say ids are 1 to 5. Or how do I get the worker ids in the Job result?

    opened by mg03 2
  •  use atomic instead of mutex

    use atomic instead of mutex

    Its a good idea that rename Next_job_id to next_job_id, public is not thread safe. But i still think atomic is better, simplify and effective, is this proposal acceptable to you?

    opened by MaruHyl 0
Owner
Ștefan Talpalaru
software architect & developer, system administrator
Ștefan Talpalaru
Algorand offchain worker with golang

Algorand offchain worker with golang

Warun C. ⚡ 0 Feb 10, 2022
a generic object pool for golang

Go Commons Pool The Go Commons Pool is a generic object pool for Golang, direct rewrite from Apache Commons Pool. Features Support custom PooledObject

jolestar 1.1k Aug 17, 2022
Gowl is a process management and process monitoring tool at once. An infinite worker pool gives you the ability to control the pool and processes and monitor their status.

Gowl is a process management and process monitoring tool at once. An infinite worker pool gives you the ability to control the pool and processes and monitor their status.

Hamed Yousefi 33 Aug 11, 2022
Golang Implementation of Worker Pool/ Thread Pool

Golang Implementation of Worker Pool/ Thread Pool

Telkom DEV 1 Jun 18, 2022
Work pool channlege - An url hash retriever worker pool for getting hash digest for a collection of urls

Code challenge The aim of this project is to provide an url hash retriever worke

null 0 Feb 16, 2022
Worker - A Golang library that provides worker pools

Worker A Golang library that provides worker pools. Usage See *_test.go files. T

Fatih Cetinkaya 2 Apr 15, 2022
Minimalistic and High-performance goroutine worker pool written in Go

pond Minimalistic and High-performance goroutine worker pool written in Go Motivation This library is meant to provide a simple way to limit concurren

Alejandro Durante 599 Aug 12, 2022
Go simple async worker pool

??‍?? worker-pool Go simple async worker pool. ?? ABOUT Worker pool is a software design pattern for achieving concurrency of task execution. Maintain

Rafał Lorenz 82 Feb 21, 2022
Go worker pool

description This is a generic worker pool for the Go language. It's useful when you want to limit the number of goroutines running in parallel. instal

Ștefan Talpalaru 62 Jul 4, 2021
Worker-Pool written in GO

go-workerpool Worker-Pool written in GO Installation go get github.com/agungsid/go-workerpool Usage package main type SampleSeeder struct{} func (s

Agung Kurniawan 2 Jun 10, 2022
Deadly simple worker pool

go-worker-pool Deadly simple worker pool Usage package main import ( "errors" workerpool "github.com/zelenin/go-worker-pool" "log" "time" ) func

Aleksandr Zelenin 1 Dec 10, 2021
Go-async - Worker pool (fan-in/fan-out)

Worker pool (fan-in/fan-out) func main() { pool := worker.NewPool(2) ctx := co

Ilya Tsyganov 0 Jan 3, 2022
Goworkers - Zero dependency Golang worker pool

Golang Worker Pool Zero dependency golang goroutines pool library. It is useful

Madhav Bhargava 0 Apr 28, 2022
Worker pool library with auto-scaling, backpressure, and easy composability of pools into pipelines

workerpool Worker pool library with auto-scaling, backpressure, and easy composability of pools into pipelines. Uses Go 1.18 generics. Notable differe

Charalampos Mitsakis 43 Jul 23, 2022
A better Generic Pool (sync.Pool)

This package is a thin wrapper over the Pool provided by the sync package. The Pool is an essential package to obtain maximum performance by reducing the number of memory allocations.

null 34 Jul 2, 2022
A mining pool proxy tool, support BTC, ETH, ETC, XMR mining pool, etc.

Tier2Pool A mining pool proxy tool, support BTC, ETH, ETC, XMR mining pool, etc. Build I use Ubuntu as a demo. sudo update sudo apt install git make s

Tier2Pool 6 Jul 29, 2022
Poweth - Etchash go module intended for use by core-pool (and open-ethereum-pool)

go-etchash Etchash go module intended for use by core-pool (and open-ethereum-po

2zo 0 Jan 20, 2022
Go-miningcore-pool - Miningcore Pool written in GOlang

Go-Miningcore-Pool (COMING SOON) Miningcore Pool written in GOlang 0x01 Configur

miningcore.com 2 Apr 24, 2022
Go-ldap-pool - A simple connection pool for go-ldap

Basic connection pool for go-ldap This little library use the go-ldap library an

Vincent 2 May 9, 2022
Simple in-memory job queue for Golang using worker-based dispatching

artifex Simple in-memory job queue for Golang using worker-based dispatching Documentation here: https://godoc.org/github.com/mborders/artifex Cron jo

Matthew Borders 134 Aug 3, 2022
goworker is a Go-based background worker that runs 10 to 100,000* times faster than Ruby-based workers.

goworker goworker is a Resque-compatible, Go-based background worker. It allows you to push jobs into a queue using an expressive language like Ruby w

Benjamin Manns 2.7k Aug 3, 2022
errgroup with goroutine worker limits

neilotoole/errgroup neilotoole/errgroup is a drop-in alternative to Go's wonderful sync/errgroup but limited to N goroutines. This is useful for inter

Neil O'Toole 131 Aug 14, 2022
Job worker service that provides an API to run arbitrary Linux processes.

Job Scheduler Summary Prototype job worker service that provides an API to run arbitrary Linux processes. Overview Library The library (Worker) is a r

Renato Guimarães 8 May 26, 2022
Worker failover support for PostgreSQL Citus extension using pg_auto_failover.

citus-failover Worker failover support for citus community version using pg_auto_failover. What is this? This is a simple service to monitor changes i

Navid Zarepak 22 Aug 3, 2022
Kubei is a flexible Kubernetes runtime scanner, scanning images of worker and Kubernetes nodes providing accurate vulnerabilities assessment, for more information checkout:

Kubei is a vulnerabilities scanning and CIS Docker benchmark tool that allows users to get an accurate and immediate risk assessment of their kubernet

Portshift 692 Aug 9, 2022
Go worker library

gorkers Examples Quickstart Multiple Go Workers Passing Fields Getting Started Import import ( "github.com/guilhem/gorkers" ) Create a worker func

Guilhem Lettron 3 Jan 7, 2022
Go lang concurrency with channels and worker pools using sync.Pools.

Go Concurrency Go lang concurrency with channels and worker pools using sync.Pools. Concurrency Concurrency is an ability of a program to do multiple

Kushal Khadka 0 Oct 31, 2021
Support variable parameters task send worker

go-worker-pool Support variable parameters task send worker 调用方式 go get github.com/214200196/gwp/v1 import ( "fmt" "github.com/214200196/gwp/v1" "

null 1 Dec 16, 2021
Snowflake algorithm generation worker Id sequence

sequence snowflake algorithm generation worker Id sequence 使用雪花算法生成ID,生成100万个只需要

null 1 Jan 21, 2022