A distributed unique ID generator inspired by Twitter's Snowflake

Overview

Sonyflake

GoDoc Build Status Coverage Status Go Report Card

Sonyflake is a distributed unique ID generator inspired by Twitter's Snowflake.

Sonyflake focuses on lifetime and performance on many host/core environment. So it has a different bit assignment from Snowflake. A Sonyflake ID is composed of

39 bits for time in units of 10 msec
 8 bits for a sequence number
16 bits for a machine id

As a result, Sonyflake has the following advantages and disadvantages:

  • The lifetime (174 years) is longer than that of Snowflake (69 years)
  • It can work in more distributed machines (2^16) than Snowflake (2^10)
  • It can generate 2^8 IDs per 10 msec at most in a single machine/thread (slower than Snowflake)

However, if you want more generation rate in a single host, you can easily run multiple Sonyflake ID generators concurrently using goroutines.

Installation

go get github.com/sony/sonyflake

Usage

The function NewSonyflake creates a new Sonyflake instance.

func NewSonyflake(st Settings) *Sonyflake

You can configure Sonyflake by the struct Settings:

type Settings struct {
	StartTime      time.Time
	MachineID      func() (uint16, error)
	CheckMachineID func(uint16) bool
}
  • StartTime is the time since which the Sonyflake time is defined as the elapsed time. If StartTime is 0, the start time of the Sonyflake is set to "2014-09-01 00:00:00 +0000 UTC". If StartTime is ahead of the current time, Sonyflake is not created.

  • MachineID returns the unique ID of the Sonyflake instance. If MachineID returns an error, Sonyflake is not created. If MachineID is nil, default MachineID is used. Default MachineID returns the lower 16 bits of the private IP address.

  • CheckMachineID validates the uniqueness of the machine ID. If CheckMachineID returns false, Sonyflake is not created. If CheckMachineID is nil, no validation is done.

In order to get a new unique ID, you just have to call the method NextID.

func (sf *Sonyflake) NextID() (uint64, error)

NextID can continue to generate IDs for about 174 years from StartTime. But after the Sonyflake time is over the limit, NextID returns an error.

AWS VPC and Docker

The awsutil package provides the function AmazonEC2MachineID that returns the lower 16-bit private IP address of the Amazon EC2 instance. It also works correctly on Docker by retrieving instance metadata.

AWS VPC is assigned a single CIDR with a netmask between /28 and /16. So if each EC2 instance has a unique private IP address in AWS VPC, the lower 16 bits of the address is also unique. In this common case, you can use AmazonEC2MachineID as Settings.MachineID.

See example that runs Sonyflake on AWS Elastic Beanstalk.

License

The MIT License (MIT)

See LICENSE for details.

Issues
  • how to ensure time synchronization In the cluster environment

    how to ensure time synchronization In the cluster environment

    how to ensure time synchronization In the cluster environment, have any good solution

    opened by zhaohuatai 6
  • Is it practical to implement sonyflake in RDBMS like MySQL, Postgres?

    Is it practical to implement sonyflake in RDBMS like MySQL, Postgres?

    Just curious to know, how practical is it to limit generation of 2^8 (256) unique IDs per 10ms on databases. Can someone please explain me with example like if I have 4GB RAM to 16GB RAM database instance, how many insert load I can expect to be handled by that server?

    It will help me use sonyflake as id generation at database level.

    opened by AmreeshTyagi 3
  • Are Sonyflakes strings always guaranteed to be in order and the same length?

    Are Sonyflakes strings always guaranteed to be in order and the same length?

    Hi!

    We would like to know if sonyflakes generated from now are a guaranteed sort order and if not then when is the next time that length will change. We have noticed an issue in our database where our sort order is being ill-computed based on that we sometimes store sonyflakes as strings. Could you kindly affirm whether the length of a sonyflake string can change? Or is there a guaranteed length.

    In particular the following algorithm...

    l = [
        "1",
        "9999",
        "2",
        "10000",
        "3",
        "4",
        "999",
        "6",
        "7",
        "8",
        "9",
        "5",
        "10",
        "100",
        "1000",
        "90",
    ]
    
    print(l)
    l.sort()
    print(l)
    

    Shows that say the number 2 > 1000000000000000 when doing string comparison. I would like to know how Sonyflakes generated reliably are affected by this as the impact of this could affect the sort ordering for some production data (may require migration). Due to the algorithmic complexity of the Sonyflake and how its generated (sonyflakes are always the same length I believe) I'm not sure if we would be impacted by this in the near future.

    Go example

    package main
    
    import (
        "fmt"
        "sort"
    )
    
    func main() {
        sl := []string{"1", "9999", "2", "10000", "3", "4", "999", "6", "7", "8", "9", "5", "10", "100", "1000", "90"}
        fmt.Println(sl)
        sort.Strings(sl)
        fmt.Println(sl)
    }
    

    I appreciate that this is not your concern and represents a mistake made in our systems, however I would appreciate any help on the matter. It would be helpful if I can run unit tests locally to generate snowflake now to the range generate one years in the future, but I have no idea how to adjust the snowflake without changing my system time like this. Appreciate the easiest way to diagnose the impact!

    opened by mmacheerpuppy 3
  • Is it possible to have a method which returns the timestamp/metadata used to create the sonyflake?

    Is it possible to have a method which returns the timestamp/metadata used to create the sonyflake?

    Hi,

    Thanks for the library. It would be nice to get the time{} struct the sonyflake is based off so that implementing services can prevent drift between recording current time and time allocated to the sonyFlake.

    Please let me know if this is something you can do.

    Cheers!

    opened by mmacheerpuppy 3
  • add benchmarks

    add benchmarks

    $ go test  -bench=.
    sonyflake id: 838861161
    decompose: map[id:838861161 machine-id:361 msb:0 sequence:0 time:50]
    max sequence: 255
    number of id: 256000
    number of cpu: 8
    number of id: 100000
    goos: linux
    goarch: amd64
    pkg: github.com/sony/sonyflake
    BenchmarkNextID-8            	   50000	     38984 ns/op
    --- FAIL: BenchmarkNextID_validate
        sonyflake_test.go:205: NextID returned err:over the time limit
    FAIL
    exit status 1
    FAIL	github.com/sony/sonyflake	16.740s
    
    opened by cupen 3
  • Fix ineffassign error

    Fix ineffassign error

    Fix ineffassign error. And, add a badge image.


    % pwd
    /path/to/sonyflake
    
    % ineffassign .
    /path/to/sonyflake/awsutil/awsutil.go:59:5: err assigned and not used
    
    opened by yuokada 3
  • Hard coded IP address should be dynamic?

    Hard coded IP address should be dynamic?

    Hi,

    Thanks for open sourcing this. I've read the source code and it seems that here https://github.com/sony/sonyflake/blob/master/awsutil/awsutil.go#L16 the IP address / URL is hard coded and can't be changed by the user. Shouldn't this be part of some sort of parameter one initializes the package with?

    I can work on a PR to do the following changes:

    • change the URL to a variable
    • create an exported package function called awsutil.Init(url string) which should receive the URL as an argument.

    However this would be a breaking change and I'd like to know your thoughts on this one first.

    opened by dlsniper 2
  • is it safe for PostgreSQL bigint

    is it safe for PostgreSQL bigint

    I see that it leaves most significant bit unused. That shall make it safe for use with 64 bit signed integer, because it will never go into the negative range from DB perspective (and will stay sortable).

    Am I reading it right?

    opened by kulak 2
  • Set Go stable versions to .travis.yml

    Set Go stable versions to .travis.yml

    The title said all.

    opened by osamingo 2
  • Make testing simple

    Make testing simple

    The title said all.

    opened by osamingo 2
  • Add DecomposeToBuffer() API to avoid allocating a map on hot paths

    Add DecomposeToBuffer() API to avoid allocating a map on hot paths

    Allocating a map[string]uint64 can be very expensive if called from a hot loop. Add a new function DecomposeToBuffer() that writes the ID components into a struct. The new function is implicitly covered by the existing tests because Decompose() uses it under the covers.

    opened by kamstrup 0
  • Add NewReproducibleID()

    Add NewReproducibleID()

    For build systems, unit tests, research, and a range of other contexts it is useful to be able to generate reproducible IDs. Similar to controlling the random seed of a RNG. This branch adds a new method NextReproducibleID() with this feature.

    Thanks for an awesome library :-)

    opened by kamstrup 0
  • Introduce mocking framework and increase coverage

    Introduce mocking framework and increase coverage

    This change introduces three common Golang patterns to the SonyFlake project test framework:

    • types: this will allow fine-tuned control over imported types by defining how they will be used, project-wide

    • mock: this allows the generation of mock constructors, which lends itself to testing any individual path in a function by "injecting" a mock object which matches the expected type

    • array-based-testing: privateIPv4 and lower16BitPrivateIP were lacking tests for their error paths, but using types and mocks, I was able add two array-based tests that cover all test cases for these methods

    This change also increases test coverage to 💯, while adding no additional time for testing ⌛ .

    This is a really cool project, and I'd be happy to contribute more!

    opened by BradleyBoutcher 3
  • Add sortable test

    Add sortable test

    TestSortableID will test if the generated ID is always sortable even when you run in Go routine. So, if you have a system that needs to generate id in application level in a thread, the generated ID can be sorted by those generated ids. Sorted by ID will resemble like sorted by time. And SonyFlake id resembles increment id in the database when you want to sort it.

    opened by yusufsyaifudin 1
  • Update sonyflake.go

    Update sonyflake.go

    根据sonyflake算法介绍,最后一位应为序列位

    另外关于sleep那一块代码,我认为不许休眠,不管如何,时间会一直向前,总会赶上损失的时间差

    opened by SongLiangChen 0
Owner
Sony
Sony Corporation
Sony
✨ Generate unique IDs (Port of Node package "generate-snowflake" to Golang)

✨ Generate Snowflake Generate unique IDs. Inspired by Twitter's Snowflake system. ?? Installation Initialize your project (go mod init example.com/exa

Barış DEMİRCİ 6 Sep 19, 2021
A distributed unique ID generator of using Sonyflake and encoded by Base58

Indigo About A distributed unique ID generator of using Sonyflake and encoded by Base58. ID max length is 11 characters by unsigned int64 max value. A

Osamu TONOMORI 91 Nov 24, 2021
❄ An Lock Free ID Generator for Golang based on Snowflake Algorithm (Twitter announced).

❄ An Lock Free ID Generator for Golang based on Snowflake Algorithm (Twitter announced).

徐连波 150 Nov 26, 2021
A tiny and fast Go unique string generator

Nano ID A tiny and fast Go unique string generator Safe. It uses cryptographically strong random APIs and tests distribution of symbols. Compact. It u

Dair Aidarkhanov 39 Nov 17, 2021
High performance unique number generator powered by Go

SEQSVR High performance unique number generator powered by Go 中文 README Features Distributed: Can be scaled horizontally High performance: Allocation

xin zhao 59 Aug 6, 2021
Tarmac is a unique framework designed for the next generation of distributed systems

Framework for building distributed services with Web Assembly

Benjamin Cane 54 Nov 28, 2021
A simple to use Go (golang) package to generate or parse Twitter snowflake IDs

snowflake snowflake is a Go package that provides A very simple Twitter snowflake generator. Methods to parse existing snowflake IDs. Methods to conve

null 1.7k Dec 3, 2021
Golang wrapper for the Snowflake Api.

GoSnowflakeApi GoSnowflakeApi is an api wrapper for the snowflake api written in golang for golang developers. Example package main import ( "fmt

SnowflakeDev Community 4 Jul 25, 2021
Snowflake By Go.

snow Snowflake by Go. package main import "github.com/czasg/snow" func main() { // default WorkID & DataCenterID is 0. snowID := snow.Next()

cza 1 Nov 24, 2021
Snowflake implemented in GO (Golang)

snowflake Snowflake is a fast, goroutine-safe unique ID generator built for distributed systems Key concepts Snowflake Snowflakes are int64s. uint64 i

Jacques Amsel 1 Nov 28, 2021
Super short, fully unique, non-sequential and URL friendly Ids

Generator of unique non-sequential short Ids The package shortidenables the generation of short, fully unique, non-sequential and by default URL frien

teris.io 720 Dec 5, 2021
Compact, sortable and fast unique IDs with embedded metadata.

A spec for unique IDs in distributed systems based on the Snowflake design, i.e. a coordination-based ID variant. It aims to be friendly to both machi

null 54 Nov 22, 2021
Universally Unique Lexicographically Sortable Identifier (ULID) in Go

Universally Unique Lexicographically Sortable Identifier A Go port of alizain/ulid with binary format implemented. Background A GUID/UUID can be subop

OK Log 2.5k Dec 3, 2021
Gerasimos (Makis) Maropoulos 21.5k Dec 1, 2021
make slice items unique in go

make slice items unique in go

Dariush Abbasi 2 Oct 27, 2021
PKSUID is a small extension for KSUID (K-Sortable Globally Unique IDs) which allows prefixing it with arbitrary up to 16 bytes strings

PKSUID is a small extension for KSUID (K-Sortable Globally Unique IDs) which allows prefixing it with arbitrary up to 16 bytes strings

Velmie 1 Nov 2, 2021
K-Sortable Globally Unique IDs

ksuid ksuid is an efficient, comprehensive, battle-tested Go library for generating and parsing a specific kind of globally unique identifier called a

Segment 3.2k Dec 3, 2021
Filling a matrix with unique, random, int numbers

randmatrix The task is to fill a matrix (m × n) with random, int, unique numbers from a certain range (for example for the range 1-3 possible numbers

Alex 0 Dec 4, 2021
Distributed hyperparameter optimization framework, inspired by Optuna.

Goptuna Distributed hyperparameter optimization framework, inspired by Optuna [1]. This library is particularly designed for machine learning, but eve

Masashi SHIBATA 192 Nov 30, 2021
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 38.1k Dec 3, 2021
Distributed lock manager. Warning: very hard to use it properly. Not because it's broken, but because distributed systems are hard. If in doubt, do not use this.

What Dlock is a distributed lock manager [1]. It is designed after flock utility but for multiple machines. When client disconnects, all his locks are

Sergey Shepelev 25 Dec 24, 2019
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 38.1k Dec 7, 2021
open source training courses about distributed database and distributed systemes

Welcome to learn Talent Plan Courses! Talent Plan is an open source training program initiated by PingCAP. It aims to create or combine some open sour

PingCAP 6.9k Dec 5, 2021
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 38.1k Dec 7, 2021
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel

etcd-io 38.1k Dec 6, 2021
An open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developersAn open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developers

Developer-oriented Continuous Delivery Product ⁣ English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use? How to

null 0 Oct 19, 2021
Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.

Consul Website: https://www.consul.io Tutorials: HashiCorp Learn Forum: Discuss Consul is a distributed, highly available, and data center aware solut

HashiCorp 23.6k Dec 4, 2021
Pure Go line editor with history, inspired by linenoise

Liner Liner is a command line editor with history. It was inspired by linenoise; everything Unix-like is a VT100 (or is trying very hard to be). If yo

Peter Harris 835 Nov 30, 2021
:art: Contextual fmt inspired by bootstrap color classes

Cfmt Contextual fmt It provides contextual formatting functions that have nearly identical usage of the fmt package. The ideas were borrowed from boot

MinJae Kwon 81 Sep 10, 2021