FlashDB is an embeddable, in-memory key/value database in Go

Overview

FlashDB

flashdb

FlashDB is a simple, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a single writer. It supports redis like operations for data structures like SET, SORTED SET, HASH and STRING.

Features

Architecture

FlashDB is made of composable libraries that can be used independently and are easy to understand. The idea is to bridge the learning for anyone new on how to build a simple ACID database.

Getting Started

Installing

To start using FlashDB, install Go and run go get:

$ go get -u github.com/arriqaaq/flashdb

This will retrieve the library.

Opening a database

The primary object in FlashDB is a DB. To open or create your database, use the flashdb.New() function:

package main

import (
	"log"

	"github.com/arriqaaq/flashdb"
)

func main() {
	config := &flashdb.Config{Path:"/tmp", EvictionInterval: 10}
	db, err := flashdb.New(config)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	...
}

It's also possible to open a database that does not persist to disk by keeping the path in the config empty.

config := &flashdb.Config{Path:"", EvictionInterval: 10}
flashdb.New(config)

Transactions

All reads and writes must be performed from inside a transaction. FlashDB can have one write transaction opened at a time, but can have many concurrent read transactions. Each transaction maintains a stable view of the database. In other words, once a transaction has begun, the data for that transaction cannot be changed by other transactions.

When a transaction fails, it will roll back, and revert all changes that occurred to the database during that transaction. When a read/write transaction succeeds all changes are persisted to disk.

Read-only Transactions

A read-only transaction should be used when you don't need to make changes to the data. The advantage of a read-only transaction is that there can be many running concurrently.

err := db.View(func(tx *flashdb.Tx) error {
	...
	return nil
})

Read/write Transactions

A read/write transaction is used when you need to make changes to your data. There can only be one read/write transaction running at a time. So make sure you close it as soon as you are done with it.

err := db.Update(func(tx *flashdb.Tx) error {
	...
	return nil
})

Setting and getting key/values

To set a value you must open a read/write transaction:

err := db.Update(func(tx *flashdb.Tx) error {
	err := tx.Set("mykey", "myvalue")
	return err
})

To get the value:

err := db.View(func(tx *flashdb.Tx) error {
	val, err := tx.Get("mykey")
	if err != nil{
		return err
	}
	fmt.Printf("value is %s\n", val)
	return nil
})

Commands

String Hash Set ZSet
SET HSET SADD ZADD
GET HGET SISMEMBER ZSCORE
DELETE HGETALL SRANDMEMBER ZCARD
EXPIRE HDEL SREM ZRANK
TTL HEXISTS SMOVE ZREVRANK
HLEN SCARD ZRANGE
HKEYS SMEMBERS ZREVRANGE
HVALS SUNION ZREM
HCLEAR SDIFF ZGETBYRANK
SCLEAR ZREVGETBYRANK
ZSCORERANGE
ZREVSCORERANGE
ZCLEAR

Benchmarks

  • Go Version : go1.11.4 darwin/amd64
  • OS: Mac OS X 10.13.6
  • Architecture: x86_64
  • 16 GB 2133 MHz LPDDR3
  • CPU: 3.1 GHz Intel Core i7
badger 2022/03/09 14:04:44 INFO: All 0 tables opened in 0s
goos: darwin
goarch: amd64
pkg: github.com/arriqaaq/flashbench
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz

BenchmarkBadgerDBPutValue64B-16     	    9940	    141844 ns/op	    2208 B/op	      68 allocs/op
BenchmarkBadgerDBPutValue128B-16    	    7701	    192942 ns/op	    2337 B/op	      68 allocs/op
BenchmarkBadgerDBPutValue256B-16    	    7368	    142600 ns/op	    2637 B/op	      69 allocs/op
BenchmarkBadgerDBPutValue512B-16    	    6980	    148056 ns/op	    3149 B/op	      69 allocs/op
BenchmarkBadgerDBGet-16             	 1000000	      1388 ns/op	     408 B/op	       9 allocs/op

BenchmarkFlashDBPutValue64B-16     	  204318	      5129 ns/op	    1385 B/op	      19 allocs/op
BenchmarkFlashDBPutValue128B-16    	  231177	      5318 ns/op	    1976 B/op	      16 allocs/op
BenchmarkFlashDBPutValue256B-16    	  189516	      6202 ns/op	    3263 B/op	      15 allocs/op
BenchmarkFlashDBPutValue512B-16    	  165580	      8110 ns/op	    5866 B/op	      16 allocs/op
BenchmarkFlashDBGet-16             	 4053836	       295 ns/op	      32 B/op	       2 allocs/op

PASS
ok  	github.com/arriqaaq/flashbench	28.947s

With fsync enabled for every update transaction

badger 2022/03/09 14:04:44 INFO: All 0 tables opened in 0s
goos: darwin
goarch: amd64
pkg: github.com/arriqaaq/flashbench
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz

BenchmarkNutsDBPutValue64B-16     	      52	  20301019 ns/op	    1315 B/op	      17 allocs/op
BenchmarkNutsDBPutValue128B-16    	      63	  23496536 ns/op	    1059 B/op	      15 allocs/op
BenchmarkNutsDBPutValue256B-16    	      62	  20037952 ns/op	    1343 B/op	      15 allocs/op
BenchmarkNutsDBPutValue512B-16    	      62	  20090731 ns/op	    1754 B/op	      15 allocs/op

BenchmarkFlashDBPutValue64B-16     	      62	  18364330 ns/op	     692 B/op	      15 allocs/op
BenchmarkFlashDBPutValue128B-16    	      64	  18315903 ns/op	    1015 B/op	      15 allocs/op
BenchmarkFlashDBPutValue256B-16    	      64	  19250441 ns/op	    1694 B/op	      15 allocs/op
BenchmarkFlashDBPutValue512B-16    	      61	  18811900 ns/op	    2976 B/op	      15 allocs/op
BenchmarkFlashDBGet-16			    3599500	     340.7 ns/op	      32 B/op	       2 allocs/op

PASS
ok  	github.com/arriqaaq/flashbench	28.947s

The benchmark code can be found here flashdb-bench.

TODO

FlashDB is in early stages of development. A couple of to-do tasks listed:

  • Add more comprehensive unit test cases
  • Add explicit documentation on various usecases

References

FlashDB is inspired by NutsDB and BuntDB.

Contact

Farhan Khan @arriqaaq

License

FlashDB source code is available under the MIT License

Issues
  • Panic occurs after about 10 seconds of inactivity after initialization

    Panic occurs after about 10 seconds of inactivity after initialization

    When I try the setup in the README, panic occurs after about 10 seconds of inactivity after flashDB initialization.

    2022-03-21 0:19:24 ~/p/g/g/a/flashdb (git)-[main]
    % go build -o bin/flashdb-server cmd/server/main.go
    2022-03-21 0:19:39 ~/p/g/g/a/flashdb (git)-[main]
    % ./bin/flashdb-server
    2022/03/21 00:19:40 no config set, using the default config.
    2022/03/21 00:19:40 FlashDB is running, ready to accept connections.
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x10cfa5c]
    
    goroutine 20 [running]:
    github.com/arriqaaq/art.(*Node).IsLeaf(...)
            /Users/matsuyoshi/go/pkg/mod/github.com/arriqaaq/[email protected]/art.go:410
    github.com/arriqaaq/flashdb.(*strStore).Keys.func1(0x4b11888)
            /Users/matsuyoshi/projects/ghq/github.com/arriqaaq/flashdb/store.go:45 +0x1c
    github.com/arriqaaq/art.(*Tree).each(0x1, 0x0, 0xc00021fdf8)
            /Users/matsuyoshi/go/pkg/mod/github.com/arriqaaq/[email protected]/art.go:811 +0x32
    github.com/arriqaaq/art.(*Tree).Each(...)
            /Users/matsuyoshi/go/pkg/mod/github.com/arriqaaq/[email protected]/art.go:807
    github.com/arriqaaq/flashdb.(*strStore).Keys(0x101000000000060)
            /Users/matsuyoshi/projects/ghq/github.com/arriqaaq/flashdb/store.go:44 +0x50
    github.com/arriqaaq/flashdb.(*strStore).evict(0xc00009c000, 0xc0000a2018)
            /Users/matsuyoshi/projects/ghq/github.com/arriqaaq/flashdb/store.go:57 +0x96
    github.com/arriqaaq/flashdb.(*sweeper).run(0xc00009c080, 0x0)
            /Users/matsuyoshi/projects/ghq/github.com/arriqaaq/flashdb/evict.go:58 +0x65
    created by github.com/arriqaaq/flashdb.New
            /Users/matsuyoshi/projects/ghq/github.com/arriqaaq/flashdb/flashdb.go:62 +0x857
    

    The cause is tree.Each() is called for art.Tree (nil) in strStore.

    opened by matsuyoshi30 3
  • Add comment lines for documentation

    Add comment lines for documentation

    • Comment lines are added to each method in tx_str.go for godoc.

    @arriqaaq I can refactor the comment lines if you want. Just let me know.

    NOTE: I will continue to add comment lines for methods in other files.

    Signed-off-by: Gökhan Özeloğlu [email protected]

    opened by gozeloglu 2
  • Updated README to show correct usage of Set command

    Updated README to show correct usage of Set command

    Issue Set API doesn't returns 3 values as mentioned in the documentation. It just returns an error instance.

    func (tx *Tx) Set(key string, value string) error {
    	e := newRecord([]byte(key), []byte(value), StringRecord, StringSet)
    	tx.addRecord(e)
    
    	return nil
    }
    

    Changes Updating the documentation to reflect the correct usage of Set API

    opened by varunu28 1
  • Trying to open a file provided as an input in a variable

    Trying to open a file provided as an input in a variable

    https://github.com/arriqaaq/flashdb/blob/8ca13a6bdd99c4ce0cff0b8b7fafd21784bd6e4f/cmd/server/main.go#L52

    Trying to open a file provided as an input in a variable. The content of this variable might be controlled by an attacker who could change it to hold unauthorised file paths from the system. see: https://securego.io/docs/rules/g304.html

    opened by lianxmfor 1
Golang-key-value-store - Key Value Store API Service with Go DDD Architecture

This document specifies the tools used in the Key-Value store and reorganizes how to use them. In this created service, In-Memory Key-Value Service was created and how to use the API is specified in the HTML file in the folder named "doc"

Kemal Emre Ballı 2 Jul 31, 2022
Key-value database stored in memory with option of persistence

Easy and intuitive command line tool allows you to spin up a database avaliable from web or locally in a few seconds. Server can be run over a custom TCP protocol or over HTTP.

Mario Petričko 6 Aug 1, 2022
ZedisDB - a key-value memory database written in Go

ZedisDB - a key-value memory database written in Go

Znkisoft 5 Jul 28, 2022
Simple Distributed key-value database (in-memory/disk) written with Golang.

Kallbaz DB Simple Distributed key-value store (in-memory/disk) written with Golang. Installation go get github.com/msam1r/kallbaz-db Usage API // Get

Mohamed Samir 5 Jan 18, 2022
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Burak Sezer 2.3k Aug 12, 2022
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jake Grogan 725 Aug 13, 2022
yakv is a simple, in-memory, concurrency-safe key-value store for hobbyists.

yakv (yak-v. (originally intended to be "yet-another-key-value store")) is a simple, in-memory, concurrency-safe key-value store for hobbyists. yakv provides persistence by appending transactions to a transaction log and restoring data from the transaction log on startup.

Aadhav Vignesh 5 Feb 24, 2022
A rest-api that works with golang as an in-memory key value store

In Store A rest-api that works with golang as an in-memory key value store Usage Fist of all, clone the repo with the command below. You must have gol

Eyüp Arslan 0 Oct 24, 2021
Simple in memory key-value store.

Simple in memory key-value store. Development This project is written in Go. Make sure you have Go installed (download). Version 1.17 or higher is req

Mustafa Navruz 0 Nov 6, 2021
A simple in-memory key-value store application

vtec vtec, is a simple in-memory key-value store application. vtec provides persistence by appending transactions to a json file and restoring data fr

Ahmet Tek 3 Jun 22, 2022
An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.

go-cache go-cache is an in-memory key:value store/cache similar to memcached that is suitable for applications running on a single machine. Its major

Patrick Mylund Nielsen 6.4k Aug 6, 2022
KV - a toy in-memory key value store built primarily in an effort to write more go and check out grpc

KV KV is a toy in-memory key value store built primarily in an effort to write more go and check out grpc. This is still a work in progress. // downlo

Ali Mir 0 Dec 30, 2021
kvStore is a simple key/value in-memory store

kvStore is a simple key/value in-memory store. It is designed for the API. kvStore keeps records at /tmp/kvStore/dbName.db. You can specify server port, dbName and, file save interval in your RunServer(Addr, dbName) call.

Buğra Mengi 2 Feb 24, 2022
A key-value db api with multiple storage engines and key generation

Jet is a deadly-simple key-value api. The main goals of this project are : Making a simple KV tool for our other projects. Learn tests writing and git

null 12 Apr 5, 2022
An embedded key/value database for Go.

Bolt Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable datab

BoltDB 13.1k Aug 8, 2022
LevelDB key/value database in Go.

This is an implementation of the LevelDB key/value database in the Go programming language. Installation go get github.com/syndtr/goleveldb/leveldb R

Suryandaru Triandana 5.3k Aug 8, 2022
RocksDB/LevelDB inspired key-value database in Go

Pebble Nightly benchmarks Pebble is a LevelDB/RocksDB inspired key-value store focused on performance and internal usage by CockroachDB. Pebble inheri

CockroachDB 3k Aug 12, 2022
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Peter Bourgon 1.2k Aug 8, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 40.8k Aug 15, 2022