Simple in-memory job queue for Golang using worker-based dispatching

Overview

GoDoc Build Status Go Report Card codecov

artifex

Simple in-memory job queue for Golang using worker-based dispatching

Documentation here: https://godoc.org/github.com/mborders/artifex

Cron jobs use the robfig/cron library: https://godoc.org/github.com/robfig/cron

Example Usage

// 10 workers, 100 max in job queue
d := artifex.NewDispatcher(10, 100)
d.Start()

d.Dispatch(func() {
  // do something
})

err := d.DispatchIn(func() {
  // do something in 500ms
}, time.Millisecond*500)

// Returns a DispatchTicker
dt, err := d.DispatchEvery(func() {
  // do something every 250ms
}, time.Millisecond*250)

// Stop a given DispatchTicker
dt.Stop()

// Returns a DispatchCron
dc, err := d.DispatchCron(func() {
  // do something every 1s
}, "*/1 * * * * *")

// Stop a given DispatchCron
dc.Stop()

// Stop a dispatcher and all its workers/tickers
d.Stop()
You might also like...
Go-async - Worker pool (fan-in/fan-out)

Worker pool (fan-in/fan-out) func main() { pool := worker.NewPool(2) ctx := co

Work pool channlege - An url hash retriever worker pool for getting hash digest for a collection of urls

Code challenge The aim of this project is to provide an url hash retriever worke

Worker pool library with auto-scaling, backpressure, and easy composability of pools into pipelines

workerpool Worker pool library with auto-scaling, backpressure, and easy composability of pools into pipelines. Uses Go 1.18 generics. Notable differe

Queue is a Golang library for spawning and managing a Goroutine pool

Queue is a Golang library for spawning and managing a Goroutine pool, Alloowing you to create multiple worker according to limit CPU number of machine.

Queue is a Golang library for spawning and managing a Goroutine pool

Queue is a Golang library for spawning and managing a Goroutine pool, Alloowing you to create multiple worker according to limit CPU number of machine.

Off heap golang memory pool

Stealthpool stealthpool provides a memory pool that allocates blocks off-heap that will NOT be tracked by the garbage collector. The name stealthpool

Simply way to control goroutines execution order based on dependencies
Simply way to control goroutines execution order based on dependencies

Goflow Goflow is a simply package to control goroutines execution order based on dependencies. It works similar to async.auto from node.js async packa

gpool - a generic context-aware resizable goroutines pool to bound concurrency based on semaphore.

gpool - a generic context-aware resizable goroutines pool to bound concurrency. Installation $ go get github.com/sherifabdlnaby/gpool import "github.c

Golang simple thread pool implementation

Golang Threadpool implementation Scalable threadpool implementation using Go to handle the huge network trafic. Install go get github.com/shettyh/thre

Comments
  • Module path should be

    Module path should be "github.com/mborders/artifex", not "github.com/borderstech/artifex"

    Background

    Module path is inconsistent with go import path. GO111MODULE=on, as doc said, import "github.com/mborders/artifex", then get this error:

    go: finding module for package github.com/mborders/artifex
    go: downloading github.com/mborders/artifex v0.0.0-20191113201913-8ea03c710bd4
    go: found github.com/mborders/artifex in github.com/mborders/artifex v0.0.0-20191113201913-8ea03c710bd4
    go: test1 imports
            github.com/mborders/artifex: github.com/mborders/[email protected]: parsing go.mod:
            module declares its path as: github.com/borderstech/artifex
                    but was required as: github.com/mborders/artifex 
    

    Solution

    Fix the module path:

    1. Rename the module path to "github.com/mborders/artifex": https://github.com/mborders/artifex/blob/master/go.mod#L1
    module github.com/mborders/artifex
    require (
    	…
    ) 
    
    1. Change the doc document to use import "github.com/borderstech/artifex".
    opened by KateGo520 3
  • Semver tag

    Semver tag

    Hi,

    looks like a very useful module. I tried to pull the latest version but it failed.

    go get -v -u github.com/mborders/[email protected]
    go get github.com/mborders/[email protected]: no matching versions for query "v0.4"
    

    My understanding is that go.mod expects a semver tag and v0.4 doesn't have the patch version. I would assume that tagging v0.4.0 would work without having to change anything.

    For now I can only get v0.3.1 which incidentally follows semver.

    opened by pims 2
  • Throttling

    Throttling

    Awesome library, pretty much exactly what I was looking for! There is one feature I'd still like to have, though.

    I'd like to have the ability to create throttled dispatchers, that is, dispatchers which run no more than X jobs per time (e.g. 5 per minute), regardless of which rate new jobs are created at.

    This could be helpful when sending out large amounts of e-mails, for example. Say, I want to send an e-mail to each of 1,000 users every Friday night, but don't want to send them all at once, but over a period of 30 minutes, to relieve the mail server a bit.

    opened by muety 0
Releases(v0.4.0)
Owner
Matthew Borders
Matthew Borders
Worker - A Golang library that provides worker pools

Worker A Golang library that provides worker pools. Usage See *_test.go files. T

Fatih Cetinkaya 2 Apr 15, 2022
goworker is a Go-based background worker that runs 10 to 100,000* times faster than Ruby-based workers.

goworker goworker is a Resque-compatible, Go-based background worker. It allows you to push jobs into a queue using an expressive language like Ruby w

Benjamin Manns 2.7k Nov 22, 2022
Go simple async worker pool

??‍?? worker-pool Go simple async worker pool. ?? ABOUT Worker pool is a software design pattern for achieving concurrency of task execution. Maintain

Rafał Lorenz 85 Sep 26, 2022
Deadly simple worker pool

go-worker-pool Deadly simple worker pool Usage package main import ( "errors" workerpool "github.com/zelenin/go-worker-pool" "log" "time" ) func

Aleksandr Zelenin 1 Dec 10, 2021
golang worker pool , Concurrency limiting goroutine pool

golang worker pool 中文说明 Concurrency limiting goroutine pool. Limits the concurrency of task execution, not the number of tasks queued. Never blocks su

xxj 424 Nov 16, 2022
Golang Implementation of Worker Pool/ Thread Pool

Golang Implementation of Worker Pool/ Thread Pool

Telkom DEV 1 Jun 18, 2022
Goworkers - Zero dependency Golang worker pool

Golang Worker Pool Zero dependency golang goroutines pool library. It is useful

Madhav Bhargava 0 Apr 28, 2022
errgroup with goroutine worker limits

neilotoole/errgroup neilotoole/errgroup is a drop-in alternative to Go's wonderful sync/errgroup but limited to N goroutines. This is useful for inter

Neil O'Toole 141 Nov 9, 2022
Minimalistic and High-performance goroutine worker pool written in Go

pond Minimalistic and High-performance goroutine worker pool written in Go Motivation This library is meant to provide a simple way to limit concurren

Alejandro Durante 668 Nov 23, 2022
Worker-Pool written in GO

go-workerpool Worker-Pool written in GO Installation go get github.com/agungsid/go-workerpool Usage package main type SampleSeeder struct{} func (s

Agung Kurniawan 2 Jun 10, 2022