Collection of high performance, thread-safe, lock-free go data structures

Overview

Garr - Go libs in a Jar

Go Reference CI Go Report Card

Collection of high performance, thread-safe, lock-free go data structures.

  • adder - Data structure to perform highly-performant sum under high contention. Inspired by OpenJDK LongAdder
  • circuit-breaker - Data structure to implement circuit breaker pattern to detect remote service failure/alive status.
  • queue - Queue data structure, go implementation of JDKLinkedQueue and MutexLinkedQueue from OpenJDK.
  • retry - Controls backoff between attempts in a retry operation.
  • worker-pool - Worker pool implementation in go to help perform multiple tasks concurrently with a fixed-but-expandable amount of workers.

Usage

Getting started

go get -u go.linecorp.com/garr

Examples

Please find detailed examples in each sub-package.

Adder

package main

import (
	"fmt"
	"time"

	ga "go.linecorp.com/garr/adder"
)

func main() {
	// or ga.DefaultAdder() which uses jdk long-adder as default
	adder := ga.NewLongAdder(ga.JDKAdderType) 

	for i := 0; i < 100; i++ {
		go func() {
			adder.Add(123)
		}()
	}

	time.Sleep(3 * time.Second)

	// get total added value
	fmt.Println(adder.Sum()) 
}

Build your own Prometheus counter with Adder

package prom

import (
	ga "go.linecorp.com/garr/adder"

	"github.com/prometheus/client_golang/prometheus"
	dto "github.com/prometheus/client_model/go"
)

// NewCounterI64 creates a new CounterI64 based on the provided prometheus.CounterOpts.
func NewCounterI64(opts prometheus.CounterOpts) CounterI64 {
	return CounterI64{counter: prometheus.NewCounter(opts)}
}

// CounterI64 is optimized Prometheus Counter for int64 value type.
type CounterI64 struct {
	val     ga.JDKAdder
	counter prometheus.Counter
}

// Value returns current value.
func (c *CounterI64) Value() int64 {
	return c.val.Sum()
}

// Reset value.
func (c *CounterI64) Reset() {
	c.val.Reset()
}

// Desc returns metric desc.
func (c *CounterI64) Desc() *prometheus.Desc {
	return c.counter.Desc()
}

// Inc by 1.
func (c *CounterI64) Inc() {
	c.val.Add(1)
}

// Add by variant.
func (c *CounterI64) Add(val int64) {
	if val > 0 {
		c.val.Add(val)
	}
}

// Write implements prometheus.Metric interface.
func (c *CounterI64) Write(out *dto.Metric) (err error) {
	if err = c.counter.Write(out); err == nil {
		value := float64(c.val.Sum())
		out.Counter.Value = &value
	}
	return
}

// Collect implements prometheus.Collector interface.
func (c *CounterI64) Collect(ch chan<- prometheus.Metric) {
	ch <- c
}

// Describe implements prometheus.Collector interface.
func (c *CounterI64) Describe(ch chan<- *prometheus.Desc) {
	ch <- c.counter.Desc()
}

Queue

package main

import (
    "fmt"

    "go.linecorp.com/garr/queue"
)

func main() {
    q := queue.DefaultQueue() // default using jdk linked queue

    // push
    q.Offer(123)

    // return head queue but not remove
    head := q.Peak()
    fmt.Println(head)

    // remove and return head queue
    polled := q.Poll()
    fmt.Println(polled)
}

Circuit Breaker

package main

import (
    cbreaker "go.linecorp.com/garr/circuit-breaker"
)

func makeRequest() error {
	return nil
}

func main() {
    cb := cbreaker.NewCircuitBreakerBuilder().
                        SetTicker(cbreaker.SystemTicker).
                        SetFailureRateThreshold(validFailureRateThreshold).
                        Build()

    if cb.CanRequest() {
        err := makeRequest()
        if err != nil {
            cb.OnFailure()
        } else {
            cb.OnSuccess()
        }
    }
}
You might also like...
Simple, fast and scalable golang rpc library for high load

gorpc Simple, fast and scalable golang RPC library for high load and microservices. Gorpc provides the following features useful for highly loaded pro

distributed data sync with operational transformation/transforms

DOT The DOT project is a blend of operational transformation, CmRDT, persistent/immutable datastructures and reactive stream processing. This is an im

Take control of your data, connect with anything, and expose it anywhere through protocols such as HTTP, GraphQL, and gRPC.
Take control of your data, connect with anything, and expose it anywhere through protocols such as HTTP, GraphQL, and gRPC.

Semaphore Chat: Discord Documentation: Github pages Go package documentation: GoDev Take control of your data, connect with anything, and expose it an

Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel

SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.
A Golang lock-free thread-safe HashMap optimized for fastest read access.

hashmap Overview A Golang lock-free thread-safe HashMap optimized for fastest read access. Usage Set a value for a key in the map: m := &HashMap{} m.S

ZenQ - A low-latency thread-safe queue in golang implemented using a lock-free ringbuffer

ZenQ A low-latency thread-safe queue in golang implemented using a lock-free ringbuffer Features Much faster than native channels in both SPSC (single

Go concurrent-safe, goroutine-safe, thread-safe queue
Go concurrent-safe, goroutine-safe, thread-safe queue

goconcurrentqueue - Concurrent safe queues The package goconcurrentqueue offers a public interface Queue with methods for a queue. It comes with multi

High-performance minimalist queue implemented using a stripped-down lock-free ringbuffer, written in Go (golang.org)

This project is no longer maintained - feel free to fork the project! gringo A high-performance minimalist queue implemented using a stripped-down loc

Package ring provides a high performance and thread safe Go implementation of a bloom filter.

ring - high performance bloom filter Package ring provides a high performance and thread safe Go implementation of a bloom filter. Usage Please see th

solution lock for golang, locallock and remote lock base on redis.

solution lock for golang, locallock and remote lock base on redis.

Go-redisson: A redisson like distributed redis lock, support watchdog、reentrant lock

go-redisson a Redisson like distributed locking implementation using Redis. Installation go get github.com/cheerego/go-redisson Lock Category Mutex Ex

go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).

中文 English 愿景:为用户提供最简单、可靠、高效的分布式文件系统。 go-fastdfs是一个基于http协议的分布式文件系统,它基于大道至简的设计理念,一切从简设计,使得它的运维及扩展变得更加简单,它具有高性能、高可靠、无中心、免维护等优点。 大家担心的是这么简单的文件系统,靠不靠谱,可不

Rotating cache for small data with lock-free access.

Rotating cache Byte cache implementation with lock-free access. Designed to work with small data under high pressure. Lock-free access (both read and

Routines was a fixed number thread pool to process the user task, and it would respawn a corresponding new thread when panic

Routines Routines was a fixed number thread pool to process the user task, and it would respawn a corresponding new thread when panic. It supports the

🔑A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout (LSM+WAL) similar to Riak.

bitcask A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout

the pluto is a gateway new time, high performance, high stable, high availability, easy to use

pluto the pluto is a gateway new time, high performance, high stable, high availability, easy to use Acknowledgments thanks nbio for providing low lev

Distributed, lock-free, self-hosted health checks and status pages
Distributed, lock-free, self-hosted health checks and status pages

Checkup is distributed, lock-free, self-hosted health checks and status pages, written in Go. It features an elegant, minimalistic CLI and an idiomati

❄ An Lock Free ID Generator for Golang based on Snowflake Algorithm (Twitter announced).
❄ An Lock Free ID Generator for Golang based on Snowflake Algorithm (Twitter announced).

❄ An Lock Free ID Generator for Golang based on Snowflake Algorithm (Twitter announced).

Comments
  • Open source a generic http client module to garr

    Open source a generic http client module to garr

    • [x] Copy module to garr.
    • [x] Adapt changes to open source.
      • [x] Add apache license.
      • [x] Update module name to a generic httpclient.
      • [x] Update some imports to go.linecorp.com/garr.
      • [x] Update tests to use only go standard test package.
    • [ ] Fix data race.
    • [ ] Create readme and examples.
    enhancement wip 
    opened by Permagate 0
Releases(v0.1.1)
Owner
LINE
LINE
Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.

What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are

Sergey Shepelev 25 Dec 24, 2019
Lockgate is a cross-platform locking library for Go with distributed locks using Kubernetes or lockgate HTTP lock server as well as the OS file locks support.

Lockgate Lockgate is a locking library for Go. Classical interface: 2 types of locks: shared and exclusive; 2 modes of locking: blocking and non-block

werf 232 Dec 16, 2022
A simple but powerful distributed lock

nlock A simple but powerful distributed lock Get Started Download go get github.com/inuggets/nlock Usage Redis lock import lock "github.com/inuggets/

Nuggets 1 Nov 14, 2021
A feature complete and high performance multi-group Raft library in Go.

Dragonboat - A Multi-Group Raft library in Go / 中文版 News 2021-01-20 Dragonboat v3.3 has been released, please check CHANGELOG for all changes. 2020-03

lni 4.5k Dec 30, 2022
High performance, distributed and low latency publish-subscribe platform.

Emitter: Distributed Publish-Subscribe Platform Emitter is a distributed, scalable and fault-tolerant publish-subscribe platform built with MQTT proto

emitter 3.4k Jan 2, 2023
High-Performance server for NATS, the cloud native messaging system.

NATS is a simple, secure and performant communications system for digital systems, services and devices. NATS is part of the Cloud Native Computing Fo

NATS - The Cloud Native Messaging System 12k Jan 8, 2023
💡 A Distributed and High-Performance Monitoring System. The next generation of Open-Falcon

夜莺简介 夜莺是一套分布式高可用的运维监控系统,最大的特点是混合云支持,既可以支持传统物理机虚拟机的场景,也可以支持K8S容器的场景。同时,夜莺也不只是监控,还有一部分CMDB的能力、自动化运维的能力,很多公司都基于夜莺开发自己公司的运维平台。开源的这部分功能模块也是商业版本的一部分,所以可靠性有保

DiDi 5.7k Jan 5, 2023
short-url distributed and high-performance

durl 是一个分布式的高性能短链服务,逻辑简单,并提供了相关api接口,开发人员可以快速接入,也可以作为go初学者练手项目.

宋昂 549 Jan 2, 2023
N-tuple structures for Go with generics

tuple N-tuple structures for Go with generics Types 2-tuple Couple Pair Dyad 3-tuple Triple Triplet Triad 4-tuple Quadruple Quartet Quad Tetrad 5-tupl

MakKi (makki_d) 3 Mar 27, 2022
A distributed and coördination-free log management system

OK Log is archived I hoped to find the opportunity to continue developing OK Log after the spike of its creation. Unfortunately, despite effort, no su

OK Log 3k Dec 26, 2022