rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.

Overview

rosedb_ico.png

 Go Report Card GitHub top language GitHub stars codecov CodeFactor Go Reference Mentioned in Awesome Go

English | 简体中文

rosedb is not production-ready, we are doing more test now.

rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask. Its on-disk files are organized as WAL(Write Ahead Log) in LSM trees, optimizing for write throughput.

rosedb supports a variety of data structures such as string, list, hash, set and sorted set.

rosedb is written in pure Go, easy to understand and use.

Our vision is to build an efficient k-v storage engine for Go-based applications. Feel free to give us any advices, and also please give us a star ,that helps a lot, thanks!

Stargazers over time

Features

  • Supports all common data structures : string, list, hash, set, zset.
  • Easy to embed (import "github.com/roseduan/rosedb").
  • Low latency and high throughput (see Benchmark).
  • Has built-in parallel execution of data modification on many provided data structures.
  • Comes with rosedb-cli for command line access, that is compatible with redis-cli.
  • Supports TTL-based key eviction.
  • Supports prefix scan and range scan for string keys.
  • Support simple transaction, ACID features.
  • Merge operation can be stopped manually.

Usage

Cli example

Navigate to rosedb/cmd/server and run main.go

Xnip2021-04-14_14-33-11.png

Open a new terminal, navigate to rosedb/cmd/cli, and run main.go

Xnip2021-04-14_14-35-50.png

Or you can just use redis-cli or any other redis client:

2021-05-14 上午11.19.24.png

Embedding example

Import rosedb in the application:

import "github.com/roseduan/rosedb"

Open a connection to the database:

package main

import (
	"github.com/roseduan/rosedb"
	"log"
)

func main() {
	config := rosedb.DefaultConfig()
	db, err := rosedb.Open(config)
	
	if err != nil {
		log.Fatal(err)
	}
	
  // don`t forget to close!
	defer db.Close()
	
	//...
}

Deploy and run on Docker

set hello rosedb OK 127.0.0.1:5200>get hello rosedb ">
docker build -t="rosedb:v1.2.9" .
docker run --name=rosedb -itd -p 5200:5200 rosedb:v1.2.9
docker exec -it rosedb sh

$ rosedb-cli
127.0.0.1:5200>set hello rosedb
OK
127.0.0.1:5200>get hello
rosedb

Command

String

  • Set
  • SetNx
  • Get
  • GetSet
  • Append
  • StrLen
  • StrExists
  • StrRem
  • PrefixScan
  • RangeScan
  • Expire
  • Persist
  • TTL

List

  • LPush
  • RPush
  • LPop
  • RPop
  • LIndex
  • LRem
  • LInsert
  • LSet
  • LTrim
  • LRange
  • LLen
  • LKeyExists
  • LValExists

Hash

  • HSet
  • HSetNx
  • HGet
  • HGetAll
  • HDel
  • HExists
  • HLen
  • HKeys
  • HVals

Set

  • SAdd
  • SPop
  • SIsMember
  • SRandMember
  • SRem
  • SMove
  • SCard
  • SMembers
  • SUnion
  • SDiff

Zset

  • ZAdd
  • ZScore
  • ZCard
  • ZRank
  • ZRevRank
  • ZIncrBy
  • ZRange
  • ZRangeWithScores
  • ZRevRange
  • ZRevRangeWithScores
  • ZRem
  • ZGetByRank
  • ZRevGetByRank
  • ZScoreRange
  • ZRevScoreRange

TODO

see here.

Benchmark

Environment

  • Go version:1.14.4
  • OS: macOS Catalina 10.15.7
  • CPU: 2.6GHz 6-Core Intel Core i7
  • Memory: 16 GB 2667 MHz DDR4
  • Code: rosedb-bench
  • Other databases for comparison:
    • Badger
    • GoLevelDB
    • Pudge

Result

1,000,000 iterations

go test -bench=. -benchtime=1000000x
badger 2021/05/16 21:59:53 INFO: All 0 tables opened in 0s
badger 2021/05/16 21:59:53 INFO: Discard stats nextEmptySlot: 0
badger 2021/05/16 21:59:53 INFO: Set nextTxnTs to 0
goos: darwin
goarch: amd64
pkg: rosedb-bench
BenchmarkPutValue_BadgerDB-12                    1000000             11518 ns/op            2110 B/op         46 allocs/op
BenchmarkGetValue_BadgerDB-12                    1000000              3547 ns/op            1172 B/op         20 allocs/op
BenchmarkPutValue_GoLevelDB-12                   1000000              4659 ns/op             352 B/op          9 allocs/op
BenchmarkGetValue_GoLevelDB-12                   1000000              2838 ns/op             814 B/op         13 allocs/op
BenchmarkPutValue_Pudge-12                       1000000              8512 ns/op             791 B/op         22 allocs/op
BenchmarkGetValue_Pudge-12                       1000000              1253 ns/op             200 B/op          6 allocs/op
BenchmarkPutValue_RoseDB_KeyValRam-12            1000000              4371 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyValRam-12            1000000               481 ns/op              56 B/op          3 allocs/op
BenchmarkPutValue_RoseDB_KeyOnlyRam-12           1000000              4255 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyOnlyRam-12           1000000              2986 ns/op             312 B/op          8 allocs/op
PASS
ok      rosedb-bench    46.388s

2,500,000 iterations

go test -bench=. -benchtime=2500000x
badger 2021/05/16 22:06:08 INFO: All 0 tables opened in 0s
badger 2021/05/16 22:06:08 INFO: Discard stats nextEmptySlot: 0
badger 2021/05/16 22:06:08 INFO: Set nextTxnTs to 0
goos: darwin
goarch: amd64
pkg: rosedb-bench
BenchmarkPutValue_BadgerDB-12                    2500000             11660 ns/op            2150 B/op         46 allocs/op
BenchmarkGetValue_BadgerDB-12                    2500000              4180 ns/op            1222 B/op         21 allocs/op
BenchmarkPutValue_GoLevelDB-12                   2500000              4637 ns/op             336 B/op          9 allocs/op
BenchmarkGetValue_GoLevelDB-12                   2500000              2942 ns/op             817 B/op         14 allocs/op
BenchmarkPutValue_Pudge-12                       2500000              9238 ns/op             763 B/op         22 allocs/op
BenchmarkGetValue_Pudge-12                       2500000              1275 ns/op             200 B/op          6 allocs/op
BenchmarkPutValue_RoseDB_KeyValRam-12            2500000              4474 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyValRam-12            2500000               525 ns/op              56 B/op          3 allocs/op
BenchmarkPutValue_RoseDB_KeyOnlyRam-12           2500000              4294 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyOnlyRam-12           2500000              3038 ns/op             312 B/op          8 allocs/op
PASS
ok      rosedb-bench    119.529s

Conclusion

Badger

Stable read and write performance. Write: 11000+ ns/op. Read: 4000+ ns/op.

GoLevelDB

Write performance is almost 2.5x that of Badger. Read is almost 3000 ns/op, a little faster than that of Badger.

Pudge

Write performance is between that of GoLevelDB and Badger, almost 8500 ns/op. Read is very fast and stable, almost twice as fast as that of GoLevelDB.

RoseDB

Write performance is stable, almost the same as that of GoLevelDB, 2.5x that of Badger.

In KeyValueRamMode, since the values are all in memory, it is the fastest of all.

In KeyOnlyRamMode, it is almost the same as GoLevelDB.

Contributing

If you are intrested in contributing to rosedb, please see here: CONTRIBUTING

Contact me

If you have any questions, you can contact me by email: [email protected]

License

rosedb is licensed under the term of the MIT License

Comments
  • Add Decr, DecrBy, Incr, IncrBy commands

    Add Decr, DecrBy, Incr, IncrBy commands

    opened by gozeloglu 7
  • 修改切换ActiveFile的时机(Modify the timing of replacing activeFile)

    修改切换ActiveFile的时机(Modify the timing of replacing activeFile)

    修改更换ActiveFile时机(Modify the timing of replacing ActivaFile)

    当前实现(Now Impl)

    • 判断当前entry是否能够写入ActiveFile(Determine whether the current entry can be written to ActiveFile)
    • 如果不能则更换ActiveFile(If not, change ActiveFile)
    • 将Entry写入到新的ActiveFile中(Write Entry to new ActiveFile)

    修改(Modify)

    • 对于当前entry, 如果ActiveFile空间不足,将超出BlockSize限制将entry写入到当前ActiveFile中(For the current entry, if the ActiveFile space is insufficient, the entry will be written to the current ActiveFile beyond the BlockSize limit)
    • 采用类似懒加载方式。每次写时先检查上次写完之后ActiveFile文件是否已满, 如果满那么写之前更换ActiveFile。(Just like lazy loading, each time you write, first check whether the ActiveFile file is full after the last write is completed, if it is full, then replace the ActiveFile before writing)

    好处(Advantage)

    参照MySQL binlog处理方式,能够最大化利用空间(Imitate MySQL binlog processing method, can maximize the use of space)

    opened by guoyuanchao1202 6
  • Add LRange commad

    Add LRange commad

    Related issue: https://github.com/flower-corp/rosedb/issues/149

    • refactor listSequence to apply it in LRange function
    • add listSequence test cases
    • Add LRange function
    opened by Reid00 5
  • Add SDiff command

    Add SDiff command

    • SDiff command implemented. SMembers() logic part exported as private helper method in order to use it another method(e.g., inside of the SDiff() method).
    • Tests added for SDiff().

    Signed-off-by: Gökhan Özeloğlu [email protected]

    opened by gozeloglu 5
  • build problems

    build problems

    Hello,

    Using: go version go1.16 windows/amd64

    I just tried to build RoseDB, but have errors:

    C:\goprojects\src\RoseDB\test>go build cmd\server\main.go cmd\grpc_server.go:6:2: no required module provides package github.com/roseduan/rosedb/cmd/proto; to add it: go get github.com/roseduan/rosedb/cmd/proto

    C:\goprojects\src\RoseDB\test>go get github.com/roseduan/rosedb/cmd/proto go get github.com/roseduan/rosedb/cmd/proto: no matching versions for query "upgrade"

    opened by lonnietc 5
  • lpop or rpop命令每删除一个数据,发送两次discard数据到discard文件

    lpop or rpop命令每删除一个数据,发送两次discard数据到discard文件

    Describe the bug https://github.com/flower-corp/rosedb/blob/main/list.go#L381

    	if err = db.saveListMeta(idxTree, key, headSeq, tailSeq); err != nil {
    		return nil, err
    	}
    	// send discard
    	db.sendDiscard(oldVal, updated, List)                                                 // 第一次发送到discard
    	_, entrySize := logfile.EncodeEntry(ent)
    	node := &indexNode{fid: pos.fid, entrySize: entrySize}
    	select {
    	case db.discards[List].valChan <- node:                                           // 第二次发送到discard
    	default:
    		logger.Warn("send to discard chan fail")
    	}
    

    我认为删除一条entry数据,仅需要发送一次discard文件,因此我想仅保存db.sendDiscard(oldVal, updated, List),此处是否为一个bug? To Reproduce Steps to reproduce the behavior: NA

    Expected behavior NA

    Additional context NA

    opened by Yukino831143 4
  • why mu is defined as the pointer type *sync. RWMutex in listIndex structs ?

    why mu is defined as the pointer type *sync. RWMutex in listIndex structs ?

    For structs in the project, such as "listIndex or hashIndex", mu is defined as the pointer type *sync. RWMutex, in the project mu is defined as is there anything wrong with sync.RWMutex(e.g. sync. RWMutex copy problem)?

    opened by vectian 4
  • Sharding for P2P model?

    Sharding for P2P model?

    Hello,

    I have just come across RoseDB and think that it might be what I need for a project that I am working on.

    One thing that I am needing is to develop for a P2P solution for the database with high number of nodes and am wondering about what it might take to have simple sharding added to RoseDB so that it can do high node clustering and spread out the data across all nodes as needed?

    opened by lonnietc 4
  • 退出GC程序的signal信号问题

    退出GC程序的signal信号问题

    Describe the bug https://github.com/flower-corp/rosedb/blob/main/db.go#L490

    signal.Notify(quitSig, os.Interrupt, os.Kill, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
    

    问题:监听的信号os.Interrupt syscall.SIGINT存在重复

    To Reproduce NA

    Expected behavior NA

    Additional context NA

    bug 
    opened by Yukino831143 3
  • Why the func is legal

    Why the func is legal

    https://github.com/flower-corp/rosedb/blob/4a3aa6540cd8c043886aee5f9b425245860ad335/util/hash.go#L7 I have no idea why this func define is legal? there is no alias for it, how it work?

    opened by Reid00 3
  • Add MSetNX command

    Add MSetNX command

    Equivalent Redis command: MSetNX

    • Helper function added to check duplicate keys.
    • Tests added.
    • Example usage added.

    Signed-off-by: Gökhan Özeloğlu [email protected]

    opened by gozeloglu 3
  • Failed under ARM/x86 architecture

    Failed under ARM/x86 architecture

    Problem with alignment on ARM (32bit) systems

    runtime/internal/atomic.panicUnaligned()
    	.../runtime/internal/atomic/unaligned.go:8 +0x24
    runtime/internal/atomic.Store64(0x205416c, 0x0)
    	.../runtime/internal/atomic/atomic_arm.s:291 +0x14
    github.com/flower-corp/rosedb.(*RoseDB).loadIndexFromLogFiles.func1(0x0, 0x20ac630)
    	.../go/pkg/mod/github.com/flower-corp/[email protected]/index.go:208 +0x268
    created by github.com/flower-corp/rosedb.(*RoseDB).loadIndexFromLogFiles
    	.../go/pkg/mod/github.com/flower-corp/[email protected]/index.go:216 +0x88
    
    opened by gendalf 0
  • feature: feat(feature): Add incrByFloat ()

    feature: feat(feature): Add incrByFloat ()

    I've been around roseDB for a long time, but this is the first time I've contributed. I've provided an incrByFloat() function for self-increment ids, and I'll try to continue contributing to roseDB and LotusDB, thanks

    opened by azhsmesos 0
  • Add More Benchmark Test

    Add More Benchmark Test

    The performance of rosedb is not so clear. We don't know how it perform as a kv database, just add a benchmark like other kv db can help a lot to constract other kv database.

    enhancement 
    opened by qiuyuyin 1
  • add GetAllKeys

    add GetAllKeys

    A new feature: list all stored keys conatain all datastructure types.

    Here is my idea:

    • add GetKeys command in different data types respectively(GetStrsKeys, GetListKeys, GetHashKeys...)
    • add GetAllKeys command in db.go, retrieve all data types GetKeys concurrently

    The GetAllKeys method like this: func (db *RoseDB) GetAllKeys(dataTypes... DataType) (map[DataType][][]byte, error)

    enhancement medium 
    opened by roseduan 2
Releases(v1.1.1)
  • v1.1.1(Jun 18, 2022)

  • v1.1.0(Jun 16, 2022)

  • v1.0.0(May 8, 2022)

    🚀 Features

    • Support String, List, Hash, Set, and ZSet

    • Logfile garbage collection automatically

      • see option LogFileGCRatio and LogFileGCInterval
    • Support standard FileIO and MMap

    • Support different index mode

      • KeyOnlyMemMode: only store keys in memory and values are in disk
      • KeyValueMemMode: both keys and values will store in memory

    🎠 Commuity

    Thanks to all new contributors:

    @gozeloglu @icemint0828 @bigdaronlee163 @Roderland @chowc @itzmn @herrhu97

    Source code(tar.gz)
    Source code(zip)
Owner
roseduan
微信公众号:roseduan写字的地方,回复 Go,你懂的。
roseduan
simple kv store engine inspired by bitcask

tiny-bitcask simple kv store engine inspired by bitcask Inspired By BitCask BitCask System Design DB启动时,系统内存储着merged-data-file 与 hint-file 系统通过读取hint-

Zach 0 Nov 14, 2021
A key-value db api with multiple storage engines and key generation

Jet is a deadly-simple key-value api. The main goals of this project are : Making a simple KV tool for our other projects. Learn tests writing and git

null 12 Apr 5, 2022
Golang-key-value-store - Key Value Store API Service with Go DDD Architecture

This document specifies the tools used in the Key-Value store and reorganizes how to use them. In this created service, In-Memory Key-Value Service was created and how to use the API is specified in the HTML file in the folder named "doc"

Kemal Emre Ballı 2 Jul 31, 2022
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Burak Sezer 2.6k Nov 30, 2022
An embedded key/value database for Go.

Bolt Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable datab

BoltDB 13.3k Nov 27, 2022
Pogreb is an embedded key-value store for read-heavy workloads written in Go.

Embedded key-value store for read-heavy workloads written in Go

Artem Krylysov 963 Nov 24, 2022
CrankDB is an ultra fast and very lightweight Key Value based Document Store.

CrankDB is a ultra fast, extreme lightweight Key Value based Document Store.

Shrey Batra 30 Apr 12, 2022
Distributed, fault-tolerant key-value storage written in go.

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.

Igor German 360 Aug 4, 2022
Fault tolerant, sharded key value storage written in GoLang

Ravel is a sharded, fault-tolerant key-value store built using BadgerDB and hashicorp/raft. You can shard your data across multiple clusters with mult

Aditya Meharia 77 Nov 1, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

null 7 Jan 4, 2022
A key-value storage transaction interpretator.

kv-txn-interpreter A key-value storage transaction interpreter, which provides an etcd-like transaction interface to help you build a transaction over

JmPotato 2 Feb 22, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

ytt 0 Jul 26, 2022
Keyval - A simple key-value storage library written in Go

keyval keyval is a simple key-value storage library written in Go and its back c

sina safari 2 Sep 16, 2022
Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the R

null 0 May 30, 2022
NutsDB a simple, fast, embeddable and persistent key/value store written in pure Go.

A simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set.

徐佳军 2.6k Nov 28, 2022
Fast key-value DB in Go.

BadgerDB BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the underlying database for Dgraph, a fast,

Dgraph 11.6k Nov 30, 2022
moss - a simple, fast, ordered, persistable, key-val storage library for golang

moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library.

null 893 Oct 25, 2022
HA LDAP based key/value solution for projects configuration storing with multi master replication support

Recon is the simple solution for storing configs of you application. There are no specified instruments, no specified data protocols. For the full power of Recon you only need curl.

Mikhail Panfilov 12 Jun 15, 2022
Implementation of distributed key-value system based on TiKV

Distributed_key-value_system A naive implementation of distributed key-value system based on TiKV Features Features of this work are listed below: Dis

null 2 Mar 7, 2022