A distributed lock service in Go using etcd

Overview

locker

GoDoc

A distributed lock service client for etcd.

What? Why?

A distributed lock service is somewhat self-explanatory. Locking (mutexes) as a service that's distributed across a cluster.

What a lock service gives you is a key-value store with locking semantics, and a mechanism for watching state changes. The distributed part of a distributed lock service is thanks to the etcd underpinnings, and it simply means your locks are persisted across a cluster.

A couple of examples of what you can build with one of these services:

Name server

You could use a lock service just like a DNS, except a DNS that's populated by the hosts it fronts rather than being an independent store.

A service would lock its hostname record and set the value to its IP address:

client.Lock("www.example.com", "10.1.1.1", nil, nil)

Then clients wanting to resolve the address could client.Get("www.example.com") and recieve 10.1.1.1.

This gets a bit more interesting when clients start watching the records and are notified of changes when they occur.

valueChanges := make(chan string)
go client.Watch("www.example.com", valueChanges, nil)

select {
case newIp := <-valueChanges:
	redirectRequestsToNewIp(newIp)
}

Mesh/grid health monitoring

Imagine a grid or mesh of services which are all polling each-other for updates. When one service goes down, we don't want the others continuing to hammer it until it's in a healthy state again. You can do this quite easily with a lock service.

Each service locks a well-known record of its own when it's healthy.

// in service-a
client.Lock("service-a", "ok", nil, quit)

// in service-b
client.Lock("service-b", "ok", nil, quit)

When it becomes unhealthy, it kills its own lock.

// in service-b
quit <- true

Meanwhile, other services are watching the records of the services they depend on, and are notified when the lock dies. They can then respond to that service becoming unavailable.

// in service-a
client.Watch("service-b", changes)

select {
case change := <-changes:
	if change == "" {
		handleServiceBGoingOffline()
	}
}

You can take this further by making service-a then kill its own lock, leading to a cascading blackout of sorts (if that's what you want). An example of this could be when someone cuts your internet pipe, all your services gracefully shut down until the pipe comes back and then they restart.

Usage

Creating a lock

import "github.com/jagregory/locker"

// Create the locker client.
// Etcd is needed as a backend, create your etcd client elsewhere and pass it in.
client := locker.New(etcdclient)

go client.Lock("key", "value", nil, nil)

client.Get("key") // "value"

Listening for state changes

The third argument to Lock is a owned channel. When a transition of ownership occurs a bool will be dumped into this channel indicating whether you own the lock.

owned := make(chan bool)
go client.Lock("key", "value", owned, nil)

select {
case change := <-owned:
	fmt.Printf("Do we own the lock? %b", change)
}

Pass nil for this channel if you don't intend to read from it. Not doing so will block the lock from refreshing.

Releasing the lock

The fourth argument to Lock is a quit channel. Push anything into this channel to kill the locking, which will let the lock expire.

quit := make(chan bool)
go client.Lock("key", "value", nil, quit)

quit <- true

Watching a lock

An interesting aspect of lock services is the ability to watch a lock that isn't owned by you. A service can alter its behaviour depending on the value of a lock. You can use the Watch function to watch the value of a lock.

valueChanges := make(chan string)
go client.Watch("key", valueChanges, nil)

select {
case change := <-valueChanges
	fmt.Printf("Lock value changed: %s", change)
}

Quitting works the same way as Lock.

Gotchas

The Lock call is blocking, use a goroutine.

The Lock method takes a owned channel. Pass nil for this if you don't intend to read from it. Passing a channel and not reading from it will block the refreshing of the lock and you'll lose it.

Issues
  • Key not found

    Key not found

    Occasionally etcd and locker seem to get into a state where a lock can't be created because the key doesn't exist. Doesn't make much sense.

    Unable to lock foo with value http://bar, retrying. 100: Key not found (/locker/foo)
    
    opened by jagregory 0
  • Change value of lock without releasing it

    Change value of lock without releasing it

    It might be useful to be able to change the value of a lock without releasing it.

    I don't have the particular need for this right now, but I could see it being useful in health monitoring scenarios.

    opened by jagregory 0
Owner
James Gregory
James Gregory
Rink is a "distributed sticky ranked ring" using etcd.

Rink is a "distributed sticky ranked ring" using etcd. A rink provides role scheduling across distributed processes, with each role only assigned

Luno 6 Apr 12, 2022
Lockgate is a cross-platform locking library for Go with distributed locks using Kubernetes or lockgate HTTP lock server as well as the OS file locks support.

Lockgate Lockgate is a locking library for Go. Classical interface: 2 types of locks: shared and exclusive; 2 modes of locking: blocking and non-block

werf 230 Jun 16, 2022
Distributed-Services - Distributed Systems with Golang to consequently build a fully-fletched distributed service

Distributed-Services This project is essentially a result of my attempt to under

Hamza Yusuff 6 Jun 1, 2022
A simple but powerful distributed lock

nlock A simple but powerful distributed lock Get Started Download go get github.com/inuggets/nlock Usage Redis lock import lock "github.com/inuggets/

Nuggets 1 Nov 14, 2021
Leader-follower-pattern - Build leader-follower system pattern with etcd election

主备系统模式 原理 使用分布式锁实现主备节点系统。通过对分布式锁进行续期,保持长期锁, 从而使当前服务节点处于主服务节点 无法获取分布式锁的服务节点,则作为备选

null 0 Jan 24, 2022
Collection of high performance, thread-safe, lock-free go data structures

Garr - Go libs in a Jar Collection of high performance, thread-safe, lock-free go data structures. adder - Data structure to perform highly-performant

LINE 311 Jun 26, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel

etcd-io 40.4k Jun 30, 2022
AppsFlyer 483 Jun 23, 2022
Simplified distributed locking implementation using Redis

redislock Simplified distributed locking implementation using Redis. For more information, please see examples. Examples import ( "fmt" "time"

Black Square Media 721 Jun 20, 2022
A Distributed Content Licensing Framework (DCLF) using Hyperledger Fabric permissioned blockchain.

A Distributed Content Licensing Framework (DCLF) using Hyperledger Fabric permissioned blockchain.

Tal Derei 3 May 3, 2022
distributed data sync with operational transformation/transforms

DOT The DOT project is a blend of operational transformation, CmRDT, persistent/immutable datastructures and reactive stream processing. This is an im

DOT & Chain 69 May 11, 2022
High performance, distributed and low latency publish-subscribe platform.

Emitter: Distributed Publish-Subscribe Platform Emitter is a distributed, scalable and fault-tolerant publish-subscribe platform built with MQTT proto

emitter 3.3k Jun 26, 2022
Fast, efficient, and scalable distributed map/reduce system, DAG execution, in memory or on disk, written in pure Go, runs standalone or distributedly.

Gleam Gleam is a high performance and efficient distributed execution system, and also simple, generic, flexible and easy to customize. Gleam is built

Chris Lu 3.1k Jun 22, 2022
Go Micro is a framework for distributed systems development

Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop

Asim Aslam 18.6k Jun 25, 2022
Skynet is a framework for distributed services in Go.

##Introduction Skynet is a communication protocol for building massively distributed apps in Go. It is not constrained to Go, so it will lend itself n

null 2k Jun 12, 2022
Go Open Source, Distributed, Simple and efficient Search Engine

Go Open Source, Distributed, Simple and efficient full text search engine.

ego 6.1k Jun 25, 2022
Dapr is a portable, event-driven, runtime for building distributed applications across cloud and edge.

Dapr is a portable, serverless, event-driven runtime that makes it easy for developers to build resilient, stateless and stateful microservices that run on the cloud and edge and embraces the diversity of languages and developer frameworks.

Dapr 18.4k Jun 28, 2022
A distributed, proof of stake blockchain designed for the financial services industry.

Provenance Blockchain Provenance is a distributed, proof of stake blockchain designed for the financial services industry.

Provenance Blockchain, Inc. 55 Jun 23, 2022