Lightweight Goroutine pool

Overview

grpool

Build Status

Lightweight Goroutine pool

Clients can submit jobs. Dispatcher takes job, and sends it to first available worker. When worker is done with processing job, will be returned back to worker pool.

Number of workers and Job queue size is configurable.

Docs

https://godoc.org/github.com/ivpusic/grpool

Installation

go get github.com/ivpusic/grpool

Simple example

package main

import (
  "fmt"
  "runtime"
  "time"

  "github.com/ivpusic/grpool"
)

func main() {
  // number of workers, and size of job queue
  pool := grpool.NewPool(100, 50)

  // release resources used by pool
  defer pool.Release()

  // submit one or more jobs to pool
  for i := 0; i < 10; i++ {
    count := i

    pool.JobQueue <- func() {
      fmt.Printf("I am worker! Number %d\n", count)
    }
  }

  // dummy wait until jobs are finished
  time.Sleep(1 * time.Second)
}

Example with waiting jobs to finish

package main

import (
  "fmt"
  "runtime"

  "github.com/ivpusic/grpool"
)

func main() {
  // number of workers, and size of job queue
  pool := grpool.NewPool(100, 50)
  defer pool.Release()

  // how many jobs we should wait
  pool.WaitCount(10)

  // submit one or more jobs to pool
  for i := 0; i < 10; i++ {
    count := i

    pool.JobQueue <- func() {
      // say that job is done, so we can know how many jobs are finished
      defer pool.JobDone()

      fmt.Printf("hello %d\n", count)
    }
  }

  // wait until we call JobDone for all jobs
  pool.WaitAll()
}

License

MIT

Comments
  • tagged releases?

    tagged releases?

    hi,

    first, thank you for this software, it looks quite useful and is used in a tool I am packaging for debian (dmarc-cat).

    one problem with this dependency is that it doesn't have official releases. it makes it hard to find out when changes happen in the code base. it also makes tracking module dependencies with the new golang module system harder.

    would you consider using official, tagged releases for your software?

    again thank you for your time.

    opened by anarcat 2
  • Add support for future mechanism

    Add support for future mechanism

    Maybe we can adopt java's future mechanism, which allows client to fetch job result like this:

    pool := NewPool(2, 10)
    
    	f := pool.Submit(func() interface{} {
    		time.Sleep(time.Second * 3)
    		return "ok"
    	})
    
    	fmt.Println(<-f.ResultChan)
    

    I have added this feature in my forked repo

    opened by wanghongfei 2
  • need fixes to run examples/first.go and examples/second.go

    need fixes to run examples/first.go and examples/second.go

    @ivpusic I have to change "package grpool" to "package main" and "func first() {" to "func main() {" in examples/first.go to run command go run src/github.com/ivpusic/grpool/examples/first.go

    I also have to fix in the same way for examples/second.go

    opened by progamer71 1
  • Optimize completion signal and add benchmarks

    Optimize completion signal and add benchmarks

    Optimize the completion signal

    • Using an empty struct instead of a boolean flag. Takes zero memory and makes the code more elegant. Further reference here - https://dave.cheney.net/2014/03/25/the-empty-struct https://dave.cheney.net/2013/04/30/curious-channels

    • Also removes the need to check for the value passed as we are only concerned about receiving the value.

    Added a benchmark function

    • Benchmarks with the current master show an average improvement of 20-30ns

    Current master-

    go test -run=XXX -bench=. -benchmem .
    using MAXPROC
    BenchmarkPool-4   	  500000	      2375 ns/op	      56 B/op	       3 allocs/op
    

    With optimization-

    go test -run=XXX -bench=. -benchmem .
    using MAXPROC
    BenchmarkPool-4   	  500000	      2341 ns/op	      56 B/op	       3 allocs/op
    
    opened by agnivade 1
  • if a  pool instance used cross go routine, wg.Add()  wg.Done() not work correctly as purpose

    if a pool instance used cross go routine, wg.Add() wg.Done() not work correctly as purpose

    example: routine 1: wg.Add(10) then emit 10 go func wg.Done() routine 2: wg.Add(2) then emit 2 func wg.Done() when pool is Wait() ready, the result maybe composed by 8 routine 1 + 2 routine 2

    opened by rabbitHX 0
  • i think it's better to add panic recover for each worker go routine.

    i think it's better to add panic recover for each worker go routine.

    i think it's better to add panic recover for each worker go routine && i see #1 has add this feature, but i cannot see this code in the master branch.

    is there any reason for this?

    opened by WanliTian 2
Releases(v1.0.0)
golang worker pool , Concurrency limiting goroutine pool

golang worker pool 中文说明 Concurrency limiting goroutine pool. Limits the concurrency of task execution, not the number of tasks queued. Never blocks su

xxj 424 Nov 16, 2022
Lightweight Goroutine pool

grpool Lightweight Goroutine pool Clients can submit jobs. Dispatcher takes job, and sends it to first available worker. When worker is done with proc

Ivan Pusic 719 Nov 21, 2022
🐝 A Highly Performant and easy to use goroutine pool for Go

gohive Package gohive implements a simple and easy to use goroutine pool for Go Features Pool can be created with a specific size as per the requireme

Lovelesh 43 Sep 26, 2022
Minimalistic and High-performance goroutine worker pool written in Go

pond Minimalistic and High-performance goroutine worker pool written in Go Motivation This library is meant to provide a simple way to limit concurren

Alejandro Durante 678 Dec 1, 2022
A goroutine pool for Go

Tunny is a Golang library for spawning and managing a goroutine pool, allowing you to limit work coming from any number of goroutines with a synchrono

Ashley Jeffs 3.4k Dec 8, 2022
Concurrency limiting goroutine pool

workerpool Concurrency limiting goroutine pool. Limits the concurrency of task execution, not the number of tasks queued. Never blocks submitting task

Andrew Gillis 968 Dec 2, 2022
Queue is a Golang library for spawning and managing a Goroutine pool

Queue is a Golang library for spawning and managing a Goroutine pool, Alloowing you to create multiple worker according to limit CPU number of machine.

Bo-Yi Wu 244 Nov 23, 2022
Queue is a Golang library for spawning and managing a Goroutine pool

Queue is a Golang library for spawning and managing a Goroutine pool, Alloowing you to create multiple worker according to limit CPU number of machine.

golang-queue 244 Nov 23, 2022
goroutine pool in golang

goroutine pool in golang

wksw 1 Nov 1, 2021
Golang Implementation of Worker Pool/ Thread Pool

Golang Implementation of Worker Pool/ Thread Pool

Telkom DEV 1 Jun 18, 2022
Go-miningcore-pool - Miningcore Pool written in GOlang

Go-Miningcore-Pool (COMING SOON) Miningcore Pool written in GOlang 0x01 Configur

miningcore.com 2 Apr 24, 2022
Go-ldap-pool - A simple connection pool for go-ldap

Basic connection pool for go-ldap This little library use the go-ldap library an

Vincent 2 May 9, 2022
Work pool channlege - An url hash retriever worker pool for getting hash digest for a collection of urls

Code challenge The aim of this project is to provide an url hash retriever worke

null 0 Feb 16, 2022
errgroup with goroutine worker limits

neilotoole/errgroup neilotoole/errgroup is a drop-in alternative to Go's wonderful sync/errgroup but limited to N goroutines. This is useful for inter

Neil O'Toole 141 Nov 9, 2022
Waiting group for collecting goroutine information.

在go语言waitGroup和errGroup都是用来控制goroutine的并发的方式,前者只能等待所有goroutine执行完成之后再执行Wait()函数后面的代码并且不

Jarvib Ding 111 Dec 3, 2021
A simple and useful goroutine concurrent library.

Taskgroup A simple and useful goroutine concurrent library. Installation go get github.com/anthhub/taskgroup

Tangqy 4 May 19, 2021
Provides some convenient API, includes Goid(), AllGoid(), and LocalStorage, which is a goroutine's local storage, just like ThreadLocal in other languages.

routine 中文版 routine encapsulates and provides some easy-to-use, high-performance goroutine context access interfaces, which can help you access corout

null 92 Nov 20, 2022
A universal mechanism to manage goroutine lifecycles

A universal mechanism to manage goroutine lifecycles

OK Log 1.2k Dec 7, 2022
A cross goroutine storage tool with very simple implementation and function.

Simple-goroutine-local is a cross goroutine storage tool with very simple implementation and function (the concept is similar to Java ThreadLocal). Ge

null 0 Jan 13, 2022