Distributed, fault-tolerant key-value storage written in go.

Overview

Godown

Build Status Go Report Card GitHub codecov

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.

asciicast

How to install

Install via binaries

You can find binaries on the Github releases page.

/application/godown/godown-server -dir=/application/godown/data01 -id=01 -listen=127.0.0.1:14001 -raft=127.0.0.1:24001
/application/godown/godown-server -dir=/application/godown/data02 -id=02 -listen=127.0.0.1:14002 -raft=127.0.0.1:24002 -join=127.0.0.1:14001
/application/godown/godown-server -dir=/application/godown/data03 -id=03 -listen=127.0.0.1:14003 -raft=127.0.0.1:24003 -join=127.0.0.1:14001

Install via docker

# creating a network
docker network create godown 

# creating a volume
docker volume create godown 

# bootstrap a cluster with a single node
docker run -it --rm -v godown:/var/lib/godown --name=godown_1 --net=godown -p 5000:5000 \
    namreg/godown-server -id 1 -listen godown_1:5000 -raft godown_1:6000

# join the second node to the cluster
docker run -it --rm -v godown:/var/lib/godown --name=godown_2 --net=godown -p 5001:5001 \
    namreg/godown-server -id 2 -listen godown_2:5001 -join godown_1:5000 -raft godown_2:6001

# join the third node to the cluster
docker run -it --rm -v godown:/var/lib/godown --name=godown_3 --net=godown -p 5002:5002  \
    namreg/godown-server -id 3 -listen godown_3:5001 -join godown_1:5000 -raft godown_3:6002

Available options to run a server:

  -dir string
        Directory where data is stored.
  -gc duration
        Garbage collector interval.
  -id string
        Server unique id.
  -join string
        Server address to join.
  -listen string
        Server address to listen.
  -raft string
        Raft protocol listen address.
  -resp string
        Redis Serialization Protocol listen address.      
  -version
        Show version.

How to connect

You can connect to any godown node. All modifications will be replicated to all nodes.

Connect via any redis client

If you have specified resp address while starting a node, you can connect to the one by any redis client.

package main

import (
	"fmt"
	"net"
	"time"

	"github.com/garyburd/redigo/redis"
)

const connectionTimeout = 100 * time.Millisecond

func main() {
	conn, err := net.Dial("tcp", "6380")
	if err != nil {
		panic(err)
	}
	rconn := redis.NewConn(conn, connectionTimeout, connectionTimeout)

	reply, err := rconn.Do("LRANGE", "list", 0, 100)
	vals, err := redis.Strings(reply, err)

	if err != nil {
		panic(err)
	}

	fmt.Println(vals)
}

Connect via CLI

godown-cli

Available options:

  -host string
    	Host to connect to a server (default "127.0.0.1")
  -port string
    	Port to connect to a server (default "4000")
  -version
    	Show godown version.

Supported commands:

Command Description
HELP command Show the usage of the given command.
TYPE key Returns the type stored at key.
KEYS pattern Find all keys matching the given pattern.
PING [message] Returns PONG if no argument is provided, otherwise return a copy of the argument as a bulk.
EXPIRE key seconds Set a timeout on key. After the timeout has expired, the key will automatically be deleted.
TTL key Returns the remaining time to live of a key. -1 returns if key does not have timeout.
--- ---
SET key value Set key to hold the string value. If key already holds a value, it is overwritten.
GET key Get the value by key. If provided key does not exist NIL will be returned.
STRLEN key Returns length of the given key. If key does not exists, 0 will be returned.
DEL key Delete the given key.
--- ---
SETBIT key offset value Sets or clears the bit at offset in the string value stored at key.
GETBIT key offset Returns the bit value at offset in the string value stored at key.
--- ---
LPUSH key value [value ...] Prepend one or multiple values to a list.
LPOP key Removes and returns the first element of the list stored at key.
RPUSH key value [value ...] Append one or multiple values to a list.
RPOP key Removes and returns the last element of the list stored at key.
LLEN key Returns the length of the list stored at key. If key does not exist, it is interpreted as an empty list and 0 is returned.
LINDEX key index Returns the element at index index in the list stored at key.
The index is zero-based, so 0 means the first element, 1 the second element and so on. Negative indices can be used to designate elements starting at the tail of the list.
LRANGE key start stop Returns the specified elements of the list stored at key.
The offsets start and stop are zero-based indexes, with 0 being the first element of the list (the head of the list), 1 being the next element and so on.
LREM key value Removes all occurrences of elements equal to value from the list stored at key.
--- ---
HSET key field value Sets field in the hash stored at key to value.
HGET key field Returns the value associated with field in the hash stored at key.
HKEYS key Returns all field names in the hash stored at key. Order of fields is not guaranteed.
HVALS key Returns all values in the hash stored at key.
HDEL key field [field ...] Removes the specified fields from the hash stored at key. Returns the number of fields that were removed.

Connect via go client

package main

import (
	"fmt"

	"github.com/namreg/godown/client"
)

func main() {
	c, err := client.New("127.0.0.1:4000")
	if err != nil {
		panic(err)
	}
	defer c.Close()

	res := c.Get("key")
	if res.Err() != nil {
		panic(res.Err())
	}

	if res.IsNil() {
		fmt.Print("key does not exist")
	} else {
		fmt.Println(res.Int64())
	}
}

Client documentation available at godoc

TODO

  • Write more docs
  • Write more tests
Issues
  • test(marshaler): add tests and benches on MarshalJSON

    test(marshaler): add tests and benches on MarshalJSON

    I added some tests on MarshalJSON to responds the TODO list in the README.md.

    I detected something weird during the marshaling of a BitMap Value.

    this value object:

    &Value{
      dataType: BitMapDataType,
      data: []uint64{1, 2, 3, 4}
    }
    

    is marshaled as:

    {
      "ttl": 0,
      "type": "bitmap",
      "value": [1, 12, 123, 1234]
    }
    

    But I was thinking that this simple BitMap Value would be marshaled as:

    {
      "ttl": 0,
      "type": "bitmap",
      "value": [1, 2, 3, 4]
    }
    

    am I missing something ?

    opened by lhauspie 2
  • Add gRPC-based API (?)

    Add gRPC-based API (?)

    Hi,

    I'm really impressed with this effort. I understand the main goal is to provide a transparent REDIS client API but would you consider an addition of a gRPC API in addition to the REDIS API? The benefits would be improved RPC semantics, speed and throughput (binary protocol).

    Would you accept contributions to that effect? (maintaining the API cleanly separated, of course)

    opened by acacio 1
  • lpush, rpush api inconsistent.

    lpush, rpush api inconsistent.

    I found lpush, rpush api inconsistent.

    // RPush appends a new value(s) to the list stored at the given key.
    func (c *Client) RPush(key, value string, values ...string) StatusResult
    
    //LPush prepends a new value to the list stored at the given key.
    func (c *Client) LPush(key, value string) StatusResult
    

    I think setting the api this way will make it more comfortable to use.

    func (c *Client) RPush(key, values ...string) StatusResult

    Originally posted by @fangpianqi in https://github.com/namreg/godown/issues/5#issuecomment-443554022

    enhancement 
    opened by namreg 5
  • func (c *Client) RPush(key, value string, values ...string) StatusResult

    func (c *Client) RPush(key, value string, values ...string) StatusResult

    I seem to have found a new problem. In my application scenario, I need to use the rpush command to send 9000000 data entries. I started three instances to form a cluster, but in the end I found that the data in the list became 27000000 after the program was completed.

    Here is my test code

    package main
    
    import (
    	"fmt"
    	"github.com/namreg/godown/client"
    	"time"
    )
    
    const data = `
    	ow9sif3W+Of+dqe2xoII1QJ2OoZFS5zjz4NtdQ49BhPOUdSlQtHVfStNlyLBxD7kxU/BV9h6PjDen2VSIbbV69jEAamcQtT4r8MTtghMC+9H/dgfNUQJQmtvHkJPqHTho1rBSRFseOpmhR9SW2CJL+0tfKGK6hziwTlbdi40qUDy905zbteLCxf2RiNDLkmwQTx+KfPVFvNPHzDOBrgvA52ZgivGlIur6r2djaAqp0XbLQN+3pw5bR1AtB/GL9x56MU+BVDg/IOQH15klpmR+EEsABF3vNo6FBL7i+JDu6ri9z9jEjMxL+UZ/lFvTkzQSiyNsigps8GnDtjuyq2ZWmdh02HMgCfNZyz9BvVZlU9PE4SgFPBtW49V080iX5t8/qlppMcRRBmOiShjmo54uTT4r+48fKMu3qX2p6F9n4jxMBbQxUDPzCq7xmNS5g4f7sH7XLTKc/W7XI2V4TNYXXxqH0YATMjz7Jyo3fOkQtUN50WPBW5fNj8soW+52q4gQqdfJcEa34HiAEo3axX5Q5LO0giLdBF/zNiM0K0leQEfgEjoGviD9s1+t214FyoKnHBQgrrFf2HxGZnuMfiDVTtUEG5UqN+dRL5VD0ZTMzY/YEJ1Fn1NIy1R7vBfhKfZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAA=
    	`
    
    func main() {
    
    	c, err := client.New("10.2.1.246:14001")
    
    	if err != nil {
    		fmt.Println(err)
    		return
    	}
    
    	start := time.Now()
    	for i := 0; i < 90; i ++ {
    		x := make([]string, 999)
    		for j := 0; j < 999; j++ {
    			x[j] = data
    		}
    		c.RPush("t13", data, x...)
    	}
    	fmt.Println(time.Since(start))
    }
    
    

    The amount of data has tripled, which is equivalent to the amount of data multiplied by the number of clusters.

    bug 
    opened by fangpianqi 1
Releases(1.2.0)
  • 1.2.0(Dec 1, 2018)

  • 1.1.3(Dec 1, 2018)

  • 1.1.2(Nov 30, 2018)

  • 1.1.0(Nov 8, 2018)

    Changelog

    6c64360 Make linter happy 1004357 Add RPOP command 164eaf6 Add RPUSH command 6620a43 fix(remove comments): remove comments after the fix in MarshalJSON of BitMap b8b4224 test(marshaler): add tests and benches on MarshalJSON 129ad09 Fix bug with bitmap marshaler 7faf827 Update README.md d80c3ea Fix type in readme

    Docker images

    • docker pull namreg/godown-server:1.1.0
    • docker pull namreg/godown-server:latest
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(394 bytes)
    godown_1.1.0_Darwin_i386.tar.gz(6.56 MB)
    godown_1.1.0_Darwin_x86_64.tar.gz(6.77 MB)
    godown_1.1.0_Linux_i386.tar.gz(6.18 MB)
    godown_1.1.0_Linux_x86_64.tar.gz(6.37 MB)
  • 1.0.1(Oct 30, 2018)

    Changelog

    d64f906 Exclude test from linter b8b4224 test(marshaler): add tests and benches on MarshalJSON 129ad09 Fix bug with bitmap marshaler 7faf827 Update README.md d80c3ea Fix type in readme

    Docker images

    • docker pull namreg/godown-server:1.0.1
    • docker pull namreg/godown-server:latest
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(394 bytes)
    godown_1.0.1_Darwin_i386.tar.gz(6.56 MB)
    godown_1.0.1_Darwin_x86_64.tar.gz(6.77 MB)
    godown_1.0.1_Linux_i386.tar.gz(6.18 MB)
    godown_1.0.1_Linux_x86_64.tar.gz(6.37 MB)
  • 1.0.0(Oct 24, 2018)

    Changelog

    87ea89f Add ci server 4dab88b Add HDEL command 24759fe Basic readme 9379e06 Add license 1d36728 Add Type method to the client 5ff9d69 Add TTL method to the client af39768 Add Strlen method to the client e9f575e Add SetBit method to the client 397b12f Add Ping method to the client 8988384 Add LRem to the client d92aba4 Add LRange method to the client 771531d Add LPush method to the client 3a98494 Add LPop method to the client 4843179 Add LLen method to the client f676e0b Add LIndex method to the client 84f9050 Add Keys method to the client de5ffd1 Add HVals method to the client ceac9ec Add HSet method to the client e533b8a Add HKeys method to the client 57d8f38 Add HGet method to the client b71c761 Add GetBit method to the client 0a70886 Small refactoring of GETBIT command 4ee862b Add Expire method to the client 05dcf18 Add Del command to the client d4b7c65 Write some tests for StatusResult and ScalarResult 3b0d6e0 Add Set method to the client 0f4b7f8 Add Get method to the client da0f1e0 Add client constructor e9b4cb4 Rename project 0c36eba Add docker integration to the goreleaser 7bbc633 Fix bug with leader connection. Small improvements 46bd7bd Add goreleaser to makefile 72382c0 Add goreleaser 14e4e57 Colorize cli 1e11768 Add version flag to the cli and to the server eece50f Add ascii logo e375b8f Persist/Restore snapshots d8dc07d Add custom marshaler to storage.Value d90e174 Add integration with raft protocol 279d9b8 Yet another refactoring 73b8071 Add PING command 4ad4dae Rename HelpResult to RawStringResult cc983b3 Remove unused code 66a9cbb Implent CLI printer 9a66bac Add godown-cli cmd c58abf9 Server now works via grpc 8b55eb5 Design protocol e57199a Change project structure 8e1beae Segrerate interfaces. Completely refactor code e0357d5 Run tests with -race flag 2fbd429 Refactor storage f69462f Use more linters 0274159 Refactor command.Results 7f3720d Integrate a golangci-lint linter and fix issues that found by the linter 8f6fbbe Use go:generate for mock genrating 896d2af Put mocks together with interfaces 92e281c Bitmap improvements: now bitmap can operates with unlimited (max uint64) offset fbffe0b Store bitmap as []uint64 5ce8f62 Store bitmap as uint64 insted of int64 e8f534c Write tests for server package 79fdc14 Write tests for server.conn c71499f Use logger in the server as dependecy bd939db Write tests for SETBIT command b8c7c8e Write tests for TYPE command 5b2a880 Introduce Clock interface. Write tests for TTL command. Refactor some other tests 23431dd Refactor tests for SET command 3daeb48 Write tests for STRLEN command 600a30c Write tests for results 42a5c46 Write tests for LREM command 68afaf0 Write tests for LRANGE command 9d5da3e Write tests for LPUSH command b9ee896 Write tests for LPOP command 08a889a Write tests for LLEN command 852b119 Write tests for LINDEX command 4cb3c1d Write tests for KEYS command 55b5e42 Write test for HVALS command 9ad8a75 Ignore expired keys in memory storage Keys method bc44f95 Write tests for HSET command d65f809 Write tests for HKEYS command 513932b Write tests for HGET command 774a792 Check expiration in storage.Get method d935413 Write tests for HELP command dd37a8f Write tests for GETBIT command fdcc145 Add EXPIRE tests. Use minimock for mocking instead of gomock 8d92995 Update memory storage tests b4861ac Remove ValidateArgs method from command 7a6f469 Write tests for command.Parse function 58f001c Write tests for DEL command. Refactor memory storage constructor 7834171 Write test for GET command d3dcca4 Write tests for SET command 478612d Write tests for memory package e7f5d4c Write tests for storage package 4731449 Optimize gc: storing values with TTL separately dbfda7f Add EXPIRE command 15ceae6 Add TTL command b6e7f62 Inroduce Garbage Collector that removes expired values 9ac3732 Add HVALS command 39bbe61 Add HKEYS command 138ce4a Add HSET command 2eaab63 Add HSET command bca1bc6 Add LRANGE command 217228b Add LREM command 63d8ca5 Add LPOP command ac50146 Add LINDEX command ca280e5 Add LLEN command 3475cdb Add LPUSH command b9bc95a Introduce ValidateArgs instead of ArgsNumber in Command interface c23924c Add STRLEN command 88d5f5a Add DEL command 50f68df Refactoring. Add SETBIT and GETBIT commands 9a02863 Use patterns in KEYS command 74c0285 Refactor results. Add simple KEYS implementation 8089c8a Add TYPE command b3ba303 Memory storage implementation. Add GET, SET commands. 06a33ae Base server with accepting command and printin it to log 6249a15 Initial commit

    Docker images

    • docker pull namreg/godown-server:1.0.0
    • docker pull namreg/godown-server:latest
    Source code(tar.gz)
    Source code(zip)
    checksums.txt(394 bytes)
    godown_1.0.0_Darwin_i386.tar.gz(6.56 MB)
    godown_1.0.0_Darwin_x86_64.tar.gz(6.77 MB)
    godown_1.0.0_Linux_i386.tar.gz(6.18 MB)
    godown_1.0.0_Linux_x86_64.tar.gz(6.37 MB)
Owner
Igor German
Backend developer (go)
Igor German
Golang-key-value-store - Key Value Store API Service with Go DDD Architecture

This document specifies the tools used in the Key-Value store and reorganizes how to use them. In this created service, In-Memory Key-Value Service was created and how to use the API is specified in the HTML file in the folder named "doc"

Kemal Emre Ballı 3 Mar 16, 2022
A key-value db api with multiple storage engines and key generation

Jet is a deadly-simple key-value api. The main goals of this project are : Making a simple KV tool for our other projects. Learn tests writing and git

null 12 Apr 5, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 40.3k Jun 24, 2022
Keyval - A simple key-value storage library written in Go

keyval keyval is a simple key-value storage library written in Go and its back c

sina safari 3 Mar 1, 2022
Simple Distributed key-value database (in-memory/disk) written with Golang.

Kallbaz DB Simple Distributed key-value store (in-memory/disk) written with Golang. Installation go get github.com/msam1r/kallbaz-db Usage API // Get

Mohamed Samir 5 Jan 18, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

null 7 Jan 4, 2022
A key-value storage transaction interpretator.

kv-txn-interpreter A key-value storage transaction interpreter, which provides an etcd-like transaction interface to help you build a transaction over

JmPotato 2 Feb 22, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

ytt 0 Dec 21, 2021
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.

rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask. Its on-disk files are organized as WAL(Write Ahead Log) in LSM trees, optimizing for write throughput.

roseduan 2.9k Jun 26, 2022
Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the R

null 0 May 30, 2022
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Burak Sezer 2.2k Jul 1, 2022
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jake Grogan 721 Jun 9, 2022
Distributed key-value store

Keva Distributed key-value store General Demo Start the server docker-compose up --build Insert data curl -XPOST http://localhost:5555/storage/test1

Yaroslav Gaponov 0 Nov 15, 2021
Implementation of distributed key-value system based on TiKV

Distributed_key-value_system A naive implementation of distributed key-value system based on TiKV Features Features of this work are listed below: Dis

null 2 Mar 7, 2022
Pogreb is an embedded key-value store for read-heavy workloads written in Go.

Embedded key-value store for read-heavy workloads written in Go

Artem Krylysov 900 Jun 26, 2022
ZedisDB - a key-value memory database written in Go

ZedisDB - a key-value memory database written in Go

Znkisoft 3 Feb 2, 2022
NutsDB a simple, fast, embeddable and persistent key/value store written in pure Go.

A simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set.

徐佳军 2.3k Jun 24, 2022
An embedded key/value database for Go.

Bolt Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable datab

BoltDB 12.9k Jun 25, 2022
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Peter Bourgon 1.2k Jun 26, 2022