High-performance minimalist queue implemented using a stripped-down lock-free ringbuffer, written in Go (golang.org)

Overview

This project is no longer maintained - feel free to fork the project!

gringo

A high-performance minimalist queue implemented using a stripped-down lock-free ringbuffer, written in Go (golang.org)

When operating with 2 or more goroutines, GOMAXPROCS >= number of goroutines and sufficient CPU cores to service the goroutines in parallel, this gives approximately 6 times the throughput of an equivalent pipeline built using channels.

Issues
  • Blocking Reads?

    Blocking Reads?

    Hey,

    I wanted to replace a worker/dispatcher logic orchestrated with channels with this but I notice a high CPU usage so the lib is probably not adapted for such usage. I'd like the have workers waiting for more work to process such as:

     for {
            work := gringo.Read()
            // ... process the work
        }
    

    but if I understand it well, it'll release the thread and continue to loop, hence a 100% CPU usage. Can you think of a way to block instead, waiting for more work to process? Many thanks!

    opened by MarcMagnin 0
  • Possible Race

    Possible Race

    https://github.com/textnode/gringo/blob/abc139f2e03e9b747a5da1a2e27e5619cd6d771d/gringo.go#L99

    lastCommittedIndex should be guarded by an atomic lock.

    opened by r0l1 3
  • Some feedback and performance tests with CPU heavy pipeline

    Some feedback and performance tests with CPU heavy pipeline

    Hi @textnode and thanks so much sharing this code!

    I also got really interested in the disruptor pattern after reading about it somewhere, and was happy to find this Go implementation.

    I've been thinking it might speed up pipelines created with my experimental flowbase and scipipe libraries (perhaps mostly relevant to flowbase), so I set out to experiment a little, with a slightly modified version of gringo, results of which is available here.

    It is a pretty CPU heavy pipeline, but I still manage to get speedups, although mostly for 1 or 2 CPUs, as can be seen in the example output (the times vary a bit, so one should really do some averaging).

    If you are interested in having a look whether I do any silly mistakes, my slightly adapted version of gringo is available on these lines in the disruptor version of the pipeline.

    opened by samuell 3
Owner
Darren Elwood
Darren Elwood
Fast golang queue using ring-buffer

Queue A fast Golang queue using a ring-buffer, based on the version suggested by Dariusz Górecki. Using this instead of other, simpler, queue implemen

Evan Huus 470 Aug 5, 2022
Package ring provides a high performance and thread safe Go implementation of a bloom filter.

ring - high performance bloom filter Package ring provides a high performance and thread safe Go implementation of a bloom filter. Usage Please see th

Tanner Ryan 126 May 10, 2022
skipmap is a high-performance concurrent sorted map based on skip list. Up to 3x ~ 10x faster than sync.Map in the typical pattern.

Introduction skipmap is a high-performance concurrent map based on skip list. In typical pattern(one million operations, 90%LOAD 9%STORE 1%DELETE), th

ZhangYunHao 73 Jul 31, 2022
A feature complete and high performance multi-group Raft library in Go.

Dragonboat - A Multi-Group Raft library in Go / 中文版 News 2021-01-20 Dragonboat v3.3 has been released, please check CHANGELOG for all changes. 2020-03

lni 4.3k Jul 30, 2022
go-fasttld is a high performance top level domains (TLD) extraction module.

go-fasttld go-fasttld is a high performance top level domains (TLD) extraction module implemented with compressed tries. This module is a port of the

Wu Tingfeng 9 Jul 20, 2022
A Go library for an efficient implementation of a skip list: https://godoc.org/github.com/MauriceGit/skiplist

Fast Skiplist Implementation This Go-library implements a very fast and efficient Skiplist that can be used as direct substitute for a balanced tree o

Maurice Tollmien 212 Aug 7, 2022
A highly optimized double-ended queue

Overview Deque is a highly optimized double-ended queue. Benchmark Benchmark_PushBack/Deque<harden> 100000000 10.3 ns/op 9 B/op

Edwin 54 Aug 3, 2022
Fast ring-buffer deque (double-ended queue)

deque Fast ring-buffer deque (double-ended queue) implementation. For a pictorial description, see the Deque diagram Installation $ go get github.com/

Andrew Gillis 361 Jul 29, 2022
Go concurrent-safe, goroutine-safe, thread-safe queue

goconcurrentqueue - Concurrent safe queues The package goconcurrentqueue offers a public interface Queue with methods for a queue. It comes with multi

Enrique Bris 200 Aug 9, 2022
go.fifo provides a simple fifo thread-safe queue for the Go programming language

go.fifo Description go.fifo provides a simple FIFO thread-safe queue. *fifo.Queue supports pushing an item at the end with Add(), and popping an item

Foize 41 Apr 15, 2022
Simple priority queue in Go

Priority Queue in Go ==================== This package provides a priority queue implementation and scaffold interfaces. Installation ------------ U

Kris Kovalik 14 Apr 5, 2022
A Go queue manager on top of Redis

Queue A Go library for managing queues on top of Redis. It is based on a hiring exercise but later I found it useful for myself in a custom task proce

Kaveh Mousavi Zamani 73 Apr 5, 2022
Cross-platform beanstalkd queue server admin console.

Overview aurora is a web-based Beanstalkd queue server console written in Go and works on macOS, Linux, and Windows machines. The main idea behind usi

null 564 Jul 31, 2022
Go implementation of the van Emde Boas tree data structure: Priority queue for positive whole numbers in O(log log u) time.

vEB Go implementation of the van Emde Boas tree data structure: Priority queue for positive whole numbers in O(log log u) time. Supports the following

null 4 Mar 7, 2022
Adaptive Radix Trees implemented in Go

An Adaptive Radix Tree Implementation in Go This library provides a Go implementation of the Adaptive Radix Tree (ART). Features: Lookup performance s

Pavel Larkin 240 Aug 1, 2022
fim is a collection of some popular frequent itemset mining algorithms implemented in Go.

fim fim is a collection of some popular frequent itemset mining algorithms implemented in Go. fim contains the implementations of the following algori

Paul Fedorow 7 Jul 14, 2022
Data structure,Algorithms implemented in Go (for education)

Data structure,Algorithms implemented in Go (for education) List of Content : 1. Math - 2. String - 3. Conversions - 4. Sort - 5. Search - 6. Data str

SkShahriarAhmedRaka 4 Jun 10, 2022
Package mafsa implements Minimal Acyclic Finite State Automata in Go, essentially a high-speed, memory-efficient, Unicode-friendly set of strings.

MA-FSA for Go Package mafsa implements Minimal Acyclic Finite State Automata (MA-FSA) with Minimal Perfect Hashing (MPH). Basically, it's a set of str

SmartyStreets (Archives) 291 Jul 2, 2022
A simple Set data structure implementation in Go (Golang) using LinkedHashMap.

Set Set is a simple Set data structure implementation in Go (Golang) using LinkedHashMap. This library allow you to get a set of int64 or string witho

Studio Sol Comunicação Digital Ltda 21 Jul 22, 2022