rosedb is an embedded and fast k-v database based on LSM + WAL

Overview

rosedb_ico.png

 Go Report Card GitHub top language GitHub stars codecov CodeFactor Go Reference Mentioned in Awesome Go

English | 简体中文

rosedb is an embedded and fast k-v database based on LSM + WAL, so it has pretty good write performance and high throughput. It also supports many kinds of data structures such as string, list, hash, set, zset,and the API name style is similar to Redis.

rosedb is in pure Go, simple and easy to understand for using or learning.

Feature

  • Support rich data structures : string, list, hash, set, zset.
  • Easy to embedded (import "github.com/roseduan/rosedb").
  • Low latency and high throughput.
  • Operations of various data types can be parallel.
  • Has builtin rosedb-cli for command line, also support redis-cli.
  • Support expiration and TTL.

Usage

Cli example

Change the directory to rosedb/cmd/server.

Run the main.go

Xnip2021-04-14_14-33-11.png

Open a new shell, and change the directory to rosedb/cmd/cli, and run the main.go

Xnip2021-04-14_14-35-50.png

Or you can just use redis-cli or any other redis client:

2021-05-14 上午11.19.24.png

Embedded example

Import rosedb in the application:

import "github.com/roseduan/rosedb"

And open a database:

package main

import (
	"github.com/roseduan/rosedb"
	"log"
)

func main() {
	config := rosedb.DefaultConfig()
	db, err := rosedb.Open(config)
	
	if err != nil {
		log.Fatal(err)
	}
	
  // don`t forget to close!
	defer db.Close()
	
	//...
}

Command

String

  • Set
  • SetNx
  • Get
  • GetSet
  • Append
  • StrLen
  • StrExists
  • StrRem
  • PrefixScan
  • RangeScan
  • Expire
  • Persist
  • TTL

List

  • LPush
  • RPush
  • LPop
  • RPop
  • LIndex
  • LRem
  • LInsert
  • LSet
  • LTrim
  • LRange
  • LLen

Hash

  • HSet
  • HSetNx
  • HGet
  • HGetAll
  • HDel
  • HExists
  • HLen
  • HKeys
  • HValues

Set

  • SAdd
  • SPop
  • SIsMember
  • SRandMember
  • SRem
  • SMove
  • SCard
  • SMembers
  • SUnion
  • SDiff

Zset

  • ZAdd
  • ZScore
  • ZCard
  • ZRank
  • ZRevRank
  • ZIncrBy
  • ZRange
  • ZRevRange
  • ZRem
  • ZGetByRank
  • ZRevGetByRank
  • ZScoreRange
  • ZRevScoreRange

TODO

  • Support expiration and TTL
  • Add prefix scan and range scan for string type
  • Cli for command line use
  • Improve the performance of reopening db.
  • Improve the performance of reclaim operation.
  • Support transaction, ACID features
  • Compress the written data
  • Add cache elimination strategy (Such as LRU, LFU, Random, etc...)
  • Improve related documents

Benchmark

Benchmark Environment

  • Go version:1.14.4

  • System: macOS Catalina 10.15.7

  • CPU: 2.6GHz 6-Core Intel Core i7

  • Memory: 16 GB 2667 MHz DDR4

  • The test databases I choose:

    • Badger
    • GoLevelDB
    • Pudge

Benchmark Result

execute 100w times

go test -bench=. -benchtime=1000000x
badger 2021/05/16 21:59:53 INFO: All 0 tables opened in 0s
badger 2021/05/16 21:59:53 INFO: Discard stats nextEmptySlot: 0
badger 2021/05/16 21:59:53 INFO: Set nextTxnTs to 0
goos: darwin
goarch: amd64
pkg: rosedb-bench
BenchmarkPutValue_BadgerDB-12                    1000000             11518 ns/op            2110 B/op         46 allocs/op
BenchmarkGetValue_BadgerDB-12                    1000000              3547 ns/op            1172 B/op         20 allocs/op
BenchmarkPutValue_GoLevelDB-12                   1000000              4659 ns/op             352 B/op          9 allocs/op
BenchmarkGetValue_GoLevelDB-12                   1000000              2838 ns/op             814 B/op         13 allocs/op
BenchmarkPutValue_Pudge-12                       1000000              8512 ns/op             791 B/op         22 allocs/op
BenchmarkGetValue_Pudge-12                       1000000              1253 ns/op             200 B/op          6 allocs/op
BenchmarkPutValue_RoseDB_KeyValRam-12            1000000              4371 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyValRam-12            1000000               481 ns/op              56 B/op          3 allocs/op
BenchmarkPutValue_RoseDB_KeyOnlyRam-12           1000000              4255 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyOnlyRam-12           1000000              2986 ns/op             312 B/op          8 allocs/op
PASS
ok      rosedb-bench    46.388s

execute 250w times

go test -bench=. -benchtime=2500000x
badger 2021/05/16 22:06:08 INFO: All 0 tables opened in 0s
badger 2021/05/16 22:06:08 INFO: Discard stats nextEmptySlot: 0
badger 2021/05/16 22:06:08 INFO: Set nextTxnTs to 0
goos: darwin
goarch: amd64
pkg: rosedb-bench
BenchmarkPutValue_BadgerDB-12                    2500000             11660 ns/op            2150 B/op         46 allocs/op
BenchmarkGetValue_BadgerDB-12                    2500000              4180 ns/op            1222 B/op         21 allocs/op
BenchmarkPutValue_GoLevelDB-12                   2500000              4637 ns/op             336 B/op          9 allocs/op
BenchmarkGetValue_GoLevelDB-12                   2500000              2942 ns/op             817 B/op         14 allocs/op
BenchmarkPutValue_Pudge-12                       2500000              9238 ns/op             763 B/op         22 allocs/op
BenchmarkGetValue_Pudge-12                       2500000              1275 ns/op             200 B/op          6 allocs/op
BenchmarkPutValue_RoseDB_KeyValRam-12            2500000              4474 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyValRam-12            2500000               525 ns/op              56 B/op          3 allocs/op
BenchmarkPutValue_RoseDB_KeyOnlyRam-12           2500000              4294 ns/op             566 B/op         11 allocs/op
BenchmarkGetValue_RoseDB_KeyOnlyRam-12           2500000              3038 ns/op             312 B/op          8 allocs/op
PASS
ok      rosedb-bench    119.529s

Benchmark Conclusion

Badger

Its read and wirte performance are stable. Write: 11000+ ns/op. Read: 4000+ ns/op.

GoLevelDB

Its write performance is almost 2.5x faster than Badger, and its read performance is almost 3000 ns/op, a little faster than Badger.

Pudge

Its write performance is between GoLevelDB and Badger, almost 8500 ns/op, slower than GoLevelDB. Its read performance is very fast and stable, almost 2x faster than GoLevelDB.

RoseDB

Its write performance is stable, alomost the same as GoLevelDB, 2.5x faster than Badger.

In KeyValueRamMode, since the values are all in memory, so it is the fastest of all.

In KeyOnlyRamMode, it is almost the same as GoLevelDB.

Contributing

If you are intrested in contributing to rosedb, please see here: CONTRIBUTING

Contact me

If you have any questions, you can contact me by email: [email protected]

License

rosedb is licensed under the term of the MIT License

Issues
  • Add Decr, DecrBy, Incr, IncrBy commands

    Add Decr, DecrBy, Incr, IncrBy commands

    opened by gozeloglu 7
  • 修改切换ActiveFile的时机(Modify the timing of replacing activeFile)

    修改切换ActiveFile的时机(Modify the timing of replacing activeFile)

    修改更换ActiveFile时机(Modify the timing of replacing ActivaFile)

    当前实现(Now Impl)

    • 判断当前entry是否能够写入ActiveFile(Determine whether the current entry can be written to ActiveFile)
    • 如果不能则更换ActiveFile(If not, change ActiveFile)
    • 将Entry写入到新的ActiveFile中(Write Entry to new ActiveFile)

    修改(Modify)

    • 对于当前entry, 如果ActiveFile空间不足,将超出BlockSize限制将entry写入到当前ActiveFile中(For the current entry, if the ActiveFile space is insufficient, the entry will be written to the current ActiveFile beyond the BlockSize limit)
    • 采用类似懒加载方式。每次写时先检查上次写完之后ActiveFile文件是否已满, 如果满那么写之前更换ActiveFile。(Just like lazy loading, each time you write, first check whether the ActiveFile file is full after the last write is completed, if it is full, then replace the ActiveFile before writing)

    好处(Advantage)

    参照MySQL binlog处理方式,能够最大化利用空间(Imitate MySQL binlog processing method, can maximize the use of space)

    opened by guoyuanchao1202 6
  • Add SDiff command

    Add SDiff command

    • SDiff command implemented. SMembers() logic part exported as private helper method in order to use it another method(e.g., inside of the SDiff() method).
    • Tests added for SDiff().

    Signed-off-by: Gökhan Özeloğlu [email protected]

    opened by gozeloglu 5
  • build problems

    build problems

    Hello,

    Using: go version go1.16 windows/amd64

    I just tried to build RoseDB, but have errors:

    C:\goprojects\src\RoseDB\test>go build cmd\server\main.go cmd\grpc_server.go:6:2: no required module provides package github.com/roseduan/rosedb/cmd/proto; to add it: go get github.com/roseduan/rosedb/cmd/proto

    C:\goprojects\src\RoseDB\test>go get github.com/roseduan/rosedb/cmd/proto go get github.com/roseduan/rosedb/cmd/proto: no matching versions for query "upgrade"

    opened by lonnietc 5
  • why mu is defined as the pointer type *sync. RWMutex in listIndex structs ?

    why mu is defined as the pointer type *sync. RWMutex in listIndex structs ?

    For structs in the project, such as "listIndex or hashIndex", mu is defined as the pointer type *sync. RWMutex, in the project mu is defined as is there anything wrong with sync.RWMutex(e.g. sync. RWMutex copy problem)?

    opened by eventian 4
  • Sharding for P2P model?

    Sharding for P2P model?

    Hello,

    I have just come across RoseDB and think that it might be what I need for a project that I am working on.

    One thing that I am needing is to develop for a P2P solution for the database with high number of nodes and am wondering about what it might take to have simple sharding added to RoseDB so that it can do high node clustering and spread out the data across all nodes as needed?

    opened by lonnietc 4
  • Add MSetNX command

    Add MSetNX command

    Equivalent Redis command: MSetNX

    • Helper function added to check duplicate keys.
    • Tests added.
    • Example usage added.

    Signed-off-by: Gökhan Özeloğlu [email protected]

    opened by gozeloglu 3
  • db_str的bug

    db_str的bug

    func TestRoseDB_GetSet(t *testing.T) {
    	t.Run("1", func(t *testing.T) {
    		err := roseDB.Set(123, 456)
    		assert.Empty(t, err)
    
    		var res int
    		err = roseDB.GetSet(123, 567, &res)
    		assert.Empty(t, err)
    		assert.Equal(t, res, 456)
    
    		var r2 int
    		err = roseDB.Get(123, &r2)
    		assert.Empty(t, err)
    		assert.Equal(t, r2, 567) // will raise a bug r2 = 456 not 567
    	})
    
    	t.Run("2", func(t *testing.T) {
    		var res interface{}
    		err := roseDB.GetSet(123, 222, &res)
    		assert.Equal(t, err, nil)
    	})
    }
    beacause for r2 get. it get data from cache. but 567 not write into cache
    this bug i can fit it.
    bug 
    opened by JayL-zxl 3
  • About support index mode[KeyValueMemMode / KeyOnlyMemMode] -> hash data structure support.

    About support index mode[KeyValueMemMode / KeyOnlyMemMode] -> hash data structure support.

    #60 关于这个issue提到的代码修改,此次更新关于hash部分的索引支持idxMode的设计:

    1. 如果是KeyValueMemMode, Value会被存储在索引部分,读取的时候直接从内存中读取,
    2. 如果是KeyOnlyMemMode, Value存储在磁盘中,内存中只有对应的fileId,offset等信息,可以通过这些信息从磁盘中读取对应的value。
    opened by eddievim 3
  • Add MSet command

    Add MSet command

    Related to https://github.com/flower-corp/rosedb/issues/98.

    • MSet() command added.
    • New error added. The error message is copied from the Redis MSet command error message.
    • Tests added.
    • Example usage added.

    Signed-off-by: Gökhan Özeloğlu [email protected]

    opened by gozeloglu 2
  • mmap写文件,每个文件只有一个entry

    mmap写文件,每个文件只有一个entry

    bug现象:

    使用mmap方式set一个kv,在0.data.str文件中会写入一个二进制entry;再次使用mmap set一个kv,该kv会写入1.data.str文件的开头,而不是继续在0.data.str中继续写入。 这导致16mb的0.data.str文件中,只有一个entry,剩余绝大部分空间都没有使用。并且每set一次不同的k,就会多一个文件。

    bug原因:

    文件offset数值错误。这发生在NewDBFile方法中的mmap逻辑中: 一开始我们让offset相当于文件的大小,但其实Truncate会将文件以'\0'填满,所以对于mmap来说,offset永远指的是文件固定大小,而不是实际entry要存储的位置。 由于存储逻辑中,会根据offset判断文件容量达到上限创建新文件存储entry,这就导致set新的k,会创建新的文件。

    stat, err := file.Stat()
    	if err != nil {
    		return nil, err
    	}
    
    	df := &DBFile{Id: fileId, Path: path, Offset: stat.Size(), method: method}
    
    	if method == FileIO {
    		df.File = file
    	} else {
    		if err = file.Truncate(blockSize); err != nil {
    			return nil, err
    		}
    		m, err := mmap.Map(file, os.O_RDWR, 0)
    		if err != nil {
    			return nil, err
    		}
    		df.mmap = m
    	}
    
    good first issue 
    opened by k-si 2
  • Add more server commands

    Add more server commands

    rosedb is starting to support redis protocol, so you can add the commands compatible with redis protocol:

    • find variable supportedCommands in cmd/cli.go, which lists all supported commands and its handler function
    • find a method in rosedb not listed here
    • add the command and its handler function
    good first issue easy 
    opened by roseduan 4
  • How to contribute?

    How to contribute?

    简体中文

    Familiar with Pull Request

    • add Test Case
    • add example code

    There are no Test methods for some API in rosedb, you can add test cases according to the existed.

    And you can also add some example code to show users how to use rosedb(under examples dir). https://github.com/flower-corp/rosedb/tree/main/examples

    This kind PRs only aims to let you deep into rosedb and understand more about how it works.

    Easy problems

    • add more Redis commands
    • good first issue
    • add Redis command

    The commands in rosedb are most similar to Redis, our goal is to support the most frequently used commands in Redis, and make sure the command does not exist in rosedb before you add it.

    Redis commands docs: https://redis.io/commands

    If an issue is marked with good first issue, it is suitable for newcomers, you can pick one which you are interested in or good at.

    Now rosedb has supported Redis protocol, but some commands are missed, so you can add the command:

    • find variable supportedCommands in cmd/cli.go, which lists all supported commands and its handler function
    • find a method in rosedb not listed here
    • add the command and its handler function

    Fix bugs or add/enhance features

    If an issue is marked with bug or enhancement, which means bug fix or add/enhance features, you might have to understand completely rosedb to solve it.

    Note

    every issue is marked with its difficulty(easy, medium, hard), you can pick them according to your understanding of rosedb, and if you have any problems, contact me as soon.

    opened by roseduan 0
  • 如何参与 rosedb ?

    如何参与 rosedb ?

    熟悉 Pr 流程

    • 添加 test 方法
    • 添加 example 代码

    目前 rosedb 当中的一些方法是没有对应的 Test 方法的,你可以按照已有的 Test,来为方法添加单元测测试。rosedb 下有一个 examples 目录,向用户展示 rosedb 的使用方法,你可以往其中添加一个方法。

    这类 Pr 没有太多的技巧可言,只是为了让你熟悉整个 Pr 的流程,并且能够对 rosedb 中的一些方法有所认识,对 rosedb 的数据读写流程有更进一步的理解。

    简单问题

    • 为 rosedb 添加更多 redis 相关的命令
    • good first issue
    • 添加 redis 协议中的方法

    rosedb 支持的命令目前大致是按照 redis 来添加的,我们做到能够支持 redis 大多数常用的命令即可,做之前要确认一下这个命令在 rosedb 当中是不存在的。

    尽量选择 redis 一些常用的命令,不常用的可以暂时不支持。

    如果一个 issue 被标识为了 good first issue,则说明这个 issue 也很适合新手参与,你可以按照自己的情况选择自己感兴趣或擅长的。

    目前 rosedb 支持了 redis 协议,但是对应的方法并不完整,你可以为 rosedb 添加一个对应的 redis 协议的方法,具体做法是:

    • 找到 cmd/server.go 中的 supportedCommands 变量,这里是所有支持的命令及其对应的处理方法
    • 选择一个在 rosedb 当中已经实现了而在这里又没有的方法
    • supportedCommands 中增加这个方法的名称,并添加对应的处理方法

    说明:redis 协议的方法的输出尽量和原生 redis 保持一致

    bug 修改或功能完善/增强

    • bug
    • enhancement

    如果有一些 bug 或者新的功能,或是对已有功能进行完善,这类 issue 会被标识为 bugenhancement,这类问题可能稍微复杂点,需要你对 rosedb 整体的流程更加熟悉。

    最后

    每个 issue 都标识了该问题的难易程度(easy、medium、hard),你可以根据自己对 rosedb 的了解程度,选择适合自己的,对于任何问题或者 issue,在做之前,或者做的过程当中有问题的话,都可以提出来和我交流。

    good first issue 
    opened by roseduan 0
Releases(v1.0.0)
  • v1.0.0(May 8, 2022)

    🚀 Features

    • Support String, List, Hash, Set, and ZSet

    • Logfile garbage collection automatically

      • see option LogFileGCRatio and LogFileGCInterval
    • Support standard FileIO and MMap

    • Support different index mode

      • KeyOnlyMemMode: only store keys in memory and values are in disk
      • KeyValueMemMode: both keys and values will store in memory

    🎠 Commuity

    Thanks to all new contributors:

    @gozeloglu @icemint0828 @bigdaronlee163 @Roderland @chowc @itzmn @herrhu97

    Source code(tar.gz)
    Source code(zip)
Owner
roseduan
微信公众号:roseduan写字的地方,回复 Go,你懂的。
roseduan
An embedded key/value database for Go.

bbolt bbolt is a fork of Ben Johnson's Bolt key/value store. The purpose of this fork is to provide the Go community with an active maintenance and de

etcd-io 5.5k May 10, 2022
An embedded, hardened key/value database for Go.

Bolt Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable datab

null 0 Nov 4, 2021
Hard Disk Database based on a former database

Hard Disk Database based on a former database

null 0 Nov 1, 2021
Embedded key-value store for read-heavy workloads written in Go

Pogreb Pogreb is an embedded key-value store for read-heavy workloads written in Go. Key characteristics 100% Go. Optimized for fast random lookups an

Artem Krylysov 895 May 10, 2022
Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on commands and key-regexes written by GO

Welcome to NIPO Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on command

Morteza Bashsiz 15 Apr 23, 2021
Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures.

Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures. capabilities which owl provides include Process approval、sql Audit、sql execute and execute as crontab、data backup and recover .

null 36 Feb 23, 2022
Beerus-DB: a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic operations

Beerus-DB · Beerus-DB is a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic

Beerus 6 Mar 5, 2022
Fast specialized time-series database for IoT, real-time internet connected devices and AI analytics.

unitdb Unitdb is blazing fast specialized time-series database for microservices, IoT, and realtime internet connected devices. As Unitdb satisfy the

Saffat Technologies 88 Mar 29, 2022
VictoriaMetrics: fast, cost-effective monitoring solution and time series database

VictoriaMetrics VictoriaMetrics is a fast, cost-effective and scalable monitoring solution and time series database. It is available in binary release

VictoriaMetrics 6.4k May 14, 2022
BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go

BadgerDB BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the underlying database for Dgraph, a fast,

Blizard 1 Dec 10, 2021
This is a simple graph database in SQLite, inspired by "SQLite as a document database".

About This is a simple graph database in SQLite, inspired by "SQLite as a document database". Structure The schema consists of just two structures: No

Denis Papathanasiou 1k May 9, 2022
Simple key value database that use json files to store the database

KValDB Simple key value database that use json files to store the database, the key and the respective value. This simple database have two gRPC metho

Francisco Santos 0 Nov 13, 2021
Fast Database engine in Go.

gaeadb gaeadb is a pure Go Database engine designed by nnsgmsone. The goal of the project is to provide a database engine for table or other complex d

null 13 Oct 29, 2021
EliasDB a graph-based database.

EliasDB EliasDB is a graph-based database which aims to provide a lightweight solution for projects which want to store their data as a graph. Feature

Matthias Ladkau 901 May 8, 2022
Lightweight RESTful database engine based on stack data structures

piladb [pee-lah-dee-bee]. pila means stack or battery in Spanish. piladb is a lightweight RESTful database engine based on stack data structures. Crea

Fernando Álvarez 194 Mar 21, 2022
Time Series Database based on Cassandra with Prometheus remote read/write support

SquirrelDB SquirrelDB is a scalable high-available timeseries database (TSDB) compatible with Prometheus remote storage. SquirrelDB store data in Cass

Bleemeo 14 Apr 25, 2022
Rk-db - Enterprise level database bootstrapper with YAML based on rk-entry in Golang

rk-db Enterprise level database bootstrapper with YAML in golang. This belongs to rk-boot family. We suggest use this lib from rk-boot. Database Statu

PointGoal 3 Mar 7, 2022
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support

BuntDB is a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a sing

Josh Baker 3.7k May 12, 2022
The Prometheus monitoring system and time series database.

Prometheus Visit prometheus.io for the full documentation, examples and guides. Prometheus, a Cloud Native Computing Foundation project, is a systems

Prometheus 42.4k May 14, 2022