rosedb is an embedded and fast k-v database based on LSM + WAL

Overview

rosedb_ico.png

 Go Report Card GitHub top language GitHub stars codecov CodeFactor Go Reference Mentioned in Awesome Go

English | 简体中文

rosedb is an embedded and fast k-v database based on LSM + WAL, so it has pretty good write performance and high throughput. It also supports many kinds of data structures such as string, list, hash, set, zset,and the API name style is similar to Redis.

rosedb is in pure Go, simple and easy to understand for using or learning.

Feature

  • Support rich data structures : string, list, hash, set, zset.
  • Easy to embedded (import "github.com/roseduan/rosedb").
  • Low latency and high throughput.
  • Operations of various data types can be parallel.
  • Has builtin rosedb-cli for command line, also support redis-cli.
  • Support expiration and TTL.

Usage

Cli example

Change the directory to rosedb/cmd/server.

Run the main.go

Xnip2021-04-14_14-33-11.png

Open a new shell, and change the directory to rosedb/cmd/cli, and run the main.go

Xnip2021-04-14_14-35-50.png

Or you can just use redis-cli or any other redis client:

2021-05-14 上午11.19.24.png

Embedded example

Import rosedb in the application:

import "github.com/roseduan/rosedb"

And open a database:

package main

import (
	"github.com/roseduan/rosedb"
	"log"
)

func main() {
	config := rosedb.DefaultConfig()
	db, err := rosedb.Open(config)
	
	if err != nil {
		log.Fatal(err)
	}
	
  // don`t forget to close!
	defer db.Close()
	
	//...
}

Command

String

  • Set
  • SetNx
  • Get
  • GetSet
  • Append
  • StrLen
  • StrExists
  • StrRem
  • PrefixScan
  • RangeScan
  • Expire
  • Persist
  • TTL

List

  • LPush
  • RPush
  • LPop
  • RPop
  • LIndex
  • LRem
  • LInsert
  • LSet
  • LTrim
  • LRange
  • LLen

Hash

  • HSet
  • HSetNx
  • HGet
  • HGetAll
  • HDel
  • HExists
  • HLen
  • HKeys
  • HValues

Set

  • SAdd
  • SPop
  • SIsMember
  • SRandMember
  • SRem
  • SMove
  • SCard
  • SMembers
  • SUnion
  • SDiff

Zset

  • ZAdd
  • ZScore
  • ZCard
  • ZRank
  • ZRevRank
  • ZIncrBy
  • ZRange
  • ZRevRange
  • ZRem
  • ZGetByRank
  • ZRevGetByRank
  • ZScoreRange
  • ZRevScoreRange

TODO

  • Support expiration and TTL
  • Add prefix scan and range scan for string type
  • Cli for command line use
  • Improve the performance of reopening db.
  • Improve the performance of reclaim operation.
  • Support transaction, ACID features
  • Compress the written data
  • Add cache elimination strategy (Such as LRU, LFU, Random, etc...)
  • Improve related documents

Benchmark

Benchmark Environment

  • Go version:1.14.4

  • System: macOS Catalina 10.15.7

  • CPU: 2.6GHz 6-Core Intel Core i7

  • Memory: 16 GB 2667 MHz DDR4

  • The test databases I choose:

    • Badger
    • GoLevelDB
    • Pudge

Benchmark Result

execute 100w times

go test -bench=. -benchtime=1000000x
badger 2021/05/16 21:59:53 INFO: All 0 tables opened in 0s
badger 2021/05/16 21:59:53 INFO: Discard stats nextEmptySlot: 0
badger 2021/05/16 21:59:53 INFO: Set nextTxnTs to 0
goos: darwin
goarch: amd64
pkg: rosedb-bench
BenchmarkPutValue_BadgerDB-12                    1000000             11518 ns/op            2110 B/op         46 allocs/op
BenchmarkGetValue_BadgerDB-12                    1000000              3547 ns/op            1172 B/op         20 allocs/op
BenchmarkPutValue_GoLevelDB-12                   1000000              4659 ns/op             352 B/op          9 allocs/op
BenchmarkGetValue_GoLevelDB-12                   1000000              2838 ns/op             814 B/op         13 allocs/op
BenchmarkPutValue_Pudge-12                       1000000              8512 ns/op             791 B/op         22 allocs/op
BenchmarkGetValue_Pudge-12                       1000000              1253 ns/op             200 B/op          6 allocs/op
BenchmarkPutValue_RoseDB_KeyValRam-12            1000000              4371 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyValRam-12            1000000               481 ns/op              56 B/op          3 allocs/op
BenchmarkPutValue_RoseDB_KeyOnlyRam-12           1000000              4255 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyOnlyRam-12           1000000              2986 ns/op             312 B/op          8 allocs/op
PASS
ok      rosedb-bench    46.388s

execute 250w times

go test -bench=. -benchtime=2500000x
badger 2021/05/16 22:06:08 INFO: All 0 tables opened in 0s
badger 2021/05/16 22:06:08 INFO: Discard stats nextEmptySlot: 0
badger 2021/05/16 22:06:08 INFO: Set nextTxnTs to 0
goos: darwin
goarch: amd64
pkg: rosedb-bench
BenchmarkPutValue_BadgerDB-12                    2500000             11660 ns/op            2150 B/op         46 allocs/op
BenchmarkGetValue_BadgerDB-12                    2500000              4180 ns/op            1222 B/op         21 allocs/op
BenchmarkPutValue_GoLevelDB-12                   2500000              4637 ns/op             336 B/op          9 allocs/op
BenchmarkGetValue_GoLevelDB-12                   2500000              2942 ns/op             817 B/op         14 allocs/op
BenchmarkPutValue_Pudge-12                       2500000              9238 ns/op             763 B/op         22 allocs/op
BenchmarkGetValue_Pudge-12                       2500000              1275 ns/op             200 B/op          6 allocs/op
BenchmarkPutValue_RoseDB_KeyValRam-12            2500000              4474 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyValRam-12            2500000               525 ns/op              56 B/op          3 allocs/op
BenchmarkPutValue_RoseDB_KeyOnlyRam-12           2500000              4294 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyOnlyRam-12           2500000              3038 ns/op             312 B/op          8 allocs/op
PASS
ok      rosedb-bench    119.529s

Benchmark Conclusion

Badger

Its read and wirte performance are stable. Write: 11000+ ns/op. Read: 4000+ ns/op.

GoLevelDB

Its write performance is almost 2.5x faster than Badger, and its read performance is almost 3000 ns/op, a little faster than Badger.

Pudge

Its write performance is between GoLevelDB and Badger, almost 8500 ns/op, slower than GoLevelDB. Its read performance is very fast and stable, almost 2x faster than GoLevelDB.

RoseDB

Its write performance is stable, alomost the same as GoLevelDB, 2.5x faster than Badger.

In KeyValueRamMode, since the values are all in memory, so it is the fastest of all.

In KeyOnlyRamMode, it is almost the same as GoLevelDB.

Contributing

If you are intrested in contributing to rosedb, please see here: CONTRIBUTING

Contact me

If you have any questions, you can contact me by email: [email protected]

License

rosedb is licensed under the term of the MIT License

Comments
  • Add Decr, DecrBy, Incr, IncrBy commands

    Add Decr, DecrBy, Incr, IncrBy commands

    opened by gozeloglu 7
  • 修改切换ActiveFile的时机(Modify the timing of replacing activeFile)

    修改切换ActiveFile的时机(Modify the timing of replacing activeFile)

    修改更换ActiveFile时机(Modify the timing of replacing ActivaFile)

    当前实现(Now Impl)

    • 判断当前entry是否能够写入ActiveFile(Determine whether the current entry can be written to ActiveFile)
    • 如果不能则更换ActiveFile(If not, change ActiveFile)
    • 将Entry写入到新的ActiveFile中(Write Entry to new ActiveFile)

    修改(Modify)

    • 对于当前entry, 如果ActiveFile空间不足,将超出BlockSize限制将entry写入到当前ActiveFile中(For the current entry, if the ActiveFile space is insufficient, the entry will be written to the current ActiveFile beyond the BlockSize limit)
    • 采用类似懒加载方式。每次写时先检查上次写完之后ActiveFile文件是否已满, 如果满那么写之前更换ActiveFile。(Just like lazy loading, each time you write, first check whether the ActiveFile file is full after the last write is completed, if it is full, then replace the ActiveFile before writing)

    好处(Advantage)

    参照MySQL binlog处理方式,能够最大化利用空间(Imitate MySQL binlog processing method, can maximize the use of space)

    opened by guoyuanchao1202 6
  • Add LRange commad

    Add LRange commad

    Related issue: https://github.com/flower-corp/rosedb/issues/149

    • refactor listSequence to apply it in LRange function
    • add listSequence test cases
    • Add LRange function
    opened by Reid00 5
  • Add SDiff command

    Add SDiff command

    • SDiff command implemented. SMembers() logic part exported as private helper method in order to use it another method(e.g., inside of the SDiff() method).
    • Tests added for SDiff().

    Signed-off-by: Gökhan Özeloğlu [email protected]

    opened by gozeloglu 5
  • build problems

    build problems

    Hello,

    Using: go version go1.16 windows/amd64

    I just tried to build RoseDB, but have errors:

    C:\goprojects\src\RoseDB\test>go build cmd\server\main.go cmd\grpc_server.go:6:2: no required module provides package github.com/roseduan/rosedb/cmd/proto; to add it: go get github.com/roseduan/rosedb/cmd/proto

    C:\goprojects\src\RoseDB\test>go get github.com/roseduan/rosedb/cmd/proto go get github.com/roseduan/rosedb/cmd/proto: no matching versions for query "upgrade"

    opened by lonnietc 5
  • lpop or rpop命令每删除一个数据,发送两次discard数据到discard文件

    lpop or rpop命令每删除一个数据,发送两次discard数据到discard文件

    Describe the bug https://github.com/flower-corp/rosedb/blob/main/list.go#L381

    	if err = db.saveListMeta(idxTree, key, headSeq, tailSeq); err != nil {
    		return nil, err
    	}
    	// send discard
    	db.sendDiscard(oldVal, updated, List)                                                 // 第一次发送到discard
    	_, entrySize := logfile.EncodeEntry(ent)
    	node := &indexNode{fid: pos.fid, entrySize: entrySize}
    	select {
    	case db.discards[List].valChan <- node:                                           // 第二次发送到discard
    	default:
    		logger.Warn("send to discard chan fail")
    	}
    

    我认为删除一条entry数据,仅需要发送一次discard文件,因此我想仅保存db.sendDiscard(oldVal, updated, List),此处是否为一个bug? To Reproduce Steps to reproduce the behavior: NA

    Expected behavior NA

    Additional context NA

    opened by Yukino831143 4
  • why mu is defined as the pointer type *sync. RWMutex in listIndex structs ?

    why mu is defined as the pointer type *sync. RWMutex in listIndex structs ?

    For structs in the project, such as "listIndex or hashIndex", mu is defined as the pointer type *sync. RWMutex, in the project mu is defined as is there anything wrong with sync.RWMutex(e.g. sync. RWMutex copy problem)?

    opened by vectian 4
  • Sharding for P2P model?

    Sharding for P2P model?

    Hello,

    I have just come across RoseDB and think that it might be what I need for a project that I am working on.

    One thing that I am needing is to develop for a P2P solution for the database with high number of nodes and am wondering about what it might take to have simple sharding added to RoseDB so that it can do high node clustering and spread out the data across all nodes as needed?

    opened by lonnietc 4
  • 退出GC程序的signal信号问题

    退出GC程序的signal信号问题

    Describe the bug https://github.com/flower-corp/rosedb/blob/main/db.go#L490

    signal.Notify(quitSig, os.Interrupt, os.Kill, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
    

    问题:监听的信号os.Interrupt syscall.SIGINT存在重复

    To Reproduce NA

    Expected behavior NA

    Additional context NA

    bug 
    opened by Yukino831143 3
  • Why the func is legal

    Why the func is legal

    https://github.com/flower-corp/rosedb/blob/4a3aa6540cd8c043886aee5f9b425245860ad335/util/hash.go#L7 I have no idea why this func define is legal? there is no alias for it, how it work?

    opened by Reid00 3
  • Add MSetNX command

    Add MSetNX command

    Equivalent Redis command: MSetNX

    • Helper function added to check duplicate keys.
    • Tests added.
    • Example usage added.

    Signed-off-by: Gökhan Özeloğlu [email protected]

    opened by gozeloglu 3
  • feature: feat(feature): Add incrByFloat ()

    feature: feat(feature): Add incrByFloat ()

    I've been around roseDB for a long time, but this is the first time I've contributed. I've provided an incrByFloat() function for self-increment ids, and I'll try to continue contributing to roseDB and LotusDB, thanks

    opened by azhsmesos 0
  • Add More Benchmark Test

    Add More Benchmark Test

    The performance of rosedb is not so clear. We don't know how it perform as a kv database, just add a benchmark like other kv db can help a lot to constract other kv database.

    enhancement 
    opened by qiuyuyin 1
  • add GetAllKeys

    add GetAllKeys

    A new feature: list all stored keys conatain all datastructure types.

    Here is my idea:

    • add GetKeys command in different data types respectively(GetStrsKeys, GetListKeys, GetHashKeys...)
    • add GetAllKeys command in db.go, retrieve all data types GetKeys concurrently

    The GetAllKeys method like this: func (db *RoseDB) GetAllKeys(dataTypes... DataType) (map[DataType][][]byte, error)

    enhancement medium 
    opened by roseduan 2
  • Add more server commands

    Add more server commands

    rosedb is starting to support redis protocol, so you can add the commands compatible with redis protocol:

    • find variable supportedCommands in cmd/cli.go, which lists all supported commands and its handler function
    • find a method in rosedb not listed here
    • add the command and its handler function
    good first issue easy 
    opened by roseduan 7
Releases(v1.1.1)
  • v1.1.1(Jun 18, 2022)

  • v1.1.0(Jun 16, 2022)

  • v1.0.0(May 8, 2022)

    🚀 Features

    • Support String, List, Hash, Set, and ZSet

    • Logfile garbage collection automatically

      • see option LogFileGCRatio and LogFileGCInterval
    • Support standard FileIO and MMap

    • Support different index mode

      • KeyOnlyMemMode: only store keys in memory and values are in disk
      • KeyValueMemMode: both keys and values will store in memory

    🎠 Commuity

    Thanks to all new contributors:

    @gozeloglu @icemint0828 @bigdaronlee163 @Roderland @chowc @itzmn @herrhu97

    Source code(tar.gz)
    Source code(zip)
Owner
roseduan
微信公众号:roseduan写字的地方,回复 Go,你懂的。
roseduan
An embedded key/value database for Go.

bbolt bbolt is a fork of Ben Johnson's Bolt key/value store. The purpose of this fork is to provide the Go community with an active maintenance and de

etcd-io 5.9k Sep 22, 2022
An embedded, hardened key/value database for Go.

Bolt Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable datab

null 0 Nov 4, 2021
Hard Disk Database based on a former database

Hard Disk Database based on a former database

null 0 Nov 1, 2021
Embedded key-value store for read-heavy workloads written in Go

Pogreb Pogreb is an embedded key-value store for read-heavy workloads written in Go. Key characteristics 100% Go. Optimized for fast random lookups an

Artem Krylysov 936 Sep 18, 2022
Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on commands and key-regexes written by GO

Welcome to NIPO Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on command

Morteza Bashsiz 16 Jun 13, 2022
Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures.

Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures. capabilities which owl provides include Process approval、sql Audit、sql execute and execute as crontab、data backup and recover .

null 35 Jun 17, 2022
Beerus-DB: a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic operations

Beerus-DB · Beerus-DB is a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic

Beerus 6 Mar 5, 2022
Fast specialized time-series database for IoT, real-time internet connected devices and AI analytics.

unitdb Unitdb is blazing fast specialized time-series database for microservices, IoT, and realtime internet connected devices. As Unitdb satisfy the

Saffat Technologies 96 Sep 1, 2022
VictoriaMetrics: fast, cost-effective monitoring solution and time series database

VictoriaMetrics VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. It is available in binary release

VictoriaMetrics 7.2k Sep 25, 2022
BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go

BadgerDB BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the underlying database for Dgraph, a fast,

Blizard 1 Dec 10, 2021
This is a simple graph database in SQLite, inspired by "SQLite as a document database".

About This is a simple graph database in SQLite, inspired by "SQLite as a document database". Structure The schema consists of just two structures: No

Denis Papathanasiou 1.1k Sep 5, 2022
Simple key value database that use json files to store the database

KValDB Simple key value database that use json files to store the database, the key and the respective value. This simple database have two gRPC metho

Francisco Santos 0 Nov 13, 2021
Fast Database engine in Go.

gaeadb gaeadb is a pure Go Database engine designed by nnsgmsone. The goal of the project is to provide a database engine for table or other complex d

null 13 Oct 29, 2021
EliasDB a graph-based database.

EliasDB EliasDB is a graph-based database which aims to provide a lightweight solution for projects which want to store their data as a graph. Feature

Matthias Ladkau 945 Sep 18, 2022
Lightweight RESTful database engine based on stack data structures

piladb [pee-lah-dee-bee]. pila means stack or battery in Spanish. piladb is a lightweight RESTful database engine based on stack data structures. Crea

Fernando Álvarez 198 Sep 7, 2022
Time Series Database based on Cassandra with Prometheus remote read/write support

SquirrelDB SquirrelDB is a scalable high-available timeseries database (TSDB) compatible with Prometheus remote storage. SquirrelDB store data in Cass

Bleemeo 16 Jun 18, 2022
Rk-db - Enterprise level database bootstrapper with YAML based on rk-entry in Golang

rk-db Enterprise level database bootstrapper with YAML in golang. This belongs to rk-boot family. We suggest use this lib from rk-boot. Database Statu

PointGoal 3 Mar 7, 2022
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support

BuntDB is a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a sing

Josh Baker 3.9k Sep 21, 2022
The Prometheus monitoring system and time series database.

Prometheus Visit prometheus.io for the full documentation, examples and guides. Prometheus, a Cloud Native Computing Foundation project, is a systems

Prometheus 44.4k Sep 26, 2022