NutsDB a simple, fast, embeddable and persistent key/value store written in pure Go.

Overview

   

NutsDB GoDoc Go Report Card Build Status Coverage Status License Mentioned in Awesome Go

English | 简体中文

NutsDB is a simple, fast, embeddable and persistent key/value store written in pure Go.

It supports fully serializable transactions and many data structures such as list、set、sorted set. All operations happen inside a Tx. Tx represents a transaction, which can be read-only or read-write. Read-only transactions can read values for a given bucket and a given key or iterate over a set of key-value pairs. Read-write transactions can read, update and delete keys from the DB.

Motivation

I wanted a simple, fast, embeddable and persistent key/value store written in pure Go. And if it supports more data structures such as list, set, sorted set,it will be better.

There are some options around the embeddable key/value store in Go:

BoltDB is based on B+ tree, has a good random read performance and awesome sequential scan performance, and it supports ACID transactions with serializable isolation, but it is terrible at random write performance and not supports more data structures such as list, etc.

GoLevelDB is based on a log-structured merge-tree (LSM tree), but it does not support more data structures.

Badger is based on LSM tree with value log. It designed for SSDs. It also supports transactions. But its write performance is not as good as I thought. And it also does not support more data structures.

Moreover, I was curious about how to implement a key/value database. The database can be said to be the core of the system, to understand the database kernel or their own implementation, better use of the same kind of database or the next time according to the business custom database is very helpful.

So I tried to build a key/value store by myself, I wanted to find a simple store engine model as a reference. Finally, I found the Bitcask model. It is simple and easy to implement. However, it has its limitation, like range or prefix queries, are not efficient. For example, you cannot easily scan over all keys between user000000 and user999999, you had to look up each key individually in the hashmap.

In order to break the limitation, I tried to optimize them. Finally, I did it and named NutsDB. NutsDB offers a high read/write performance and supports transactions. And it still has a lot of room for optimization. Welcome contributions to NutsDB.

Table of Contents

Getting Started

Installing

To start using NutsDB, first needs Go installed (version 1.11+ is required). and run go get:

go get -u github.com/xujiajun/nutsdb

Opening a database

To open your database, use the nutsdb.Open() function,with the appropriate options.The Dir , EntryIdxMode and SegmentSize options are must be specified by the client. About options see here for detail.

package main

import (
	"log"

	"github.com/xujiajun/nutsdb"
)

func main() {
	// Open the database located in the /tmp/nutsdb directory.
	// It will be created if it doesn't exist.
	opt := nutsdb.DefaultOptions
	opt.Dir = "/tmp/nutsdb"
	db, err := nutsdb.Open(opt)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	...
}

Options

  • Dir string

Dir represents Open the database located in which dir.

  • EntryIdxMode EntryIdxMode

EntryIdxMode represents using which mode to index the entries. EntryIdxMode includes three options: HintKeyValAndRAMIdxMode,HintKeyAndRAMIdxMode and HintBPTSparseIdxMode. HintKeyValAndRAMIdxMode represents ram index (key and value) mode, HintKeyAndRAMIdxMode represents ram index (only key) mode and HintBPTSparseIdxMode represents b+ tree sparse index mode.

  • RWMode RWMode

RWMode represents the read and write mode. RWMode includes two options: FileIO and MMap. FileIO represents the read and write mode using standard I/O. And MMap represents the read and write mode using mmap.

  • SegmentSize int64

NutsDB will truncate data file if the active file is larger than SegmentSize. Current verison default SegmentSize is 8MB,but you can custom it. Once set, it cannot be changed. see caveats--limitations for detail.

  • NodeNum int64

NodeNum represents the node number.Default NodeNum is 1. NodeNum range [1,1023] .

  • SyncEnable bool

SyncEnable represents if call Sync() function. if SyncEnable is false, high write performance but potential data loss likely. if SyncEnable is true, slower but persistent.

  • StartFileLoadingMode RWMode

StartFileLoadingMode represents when open a database which RWMode to load files.

Default Options

Recommend to use the DefaultOptions . Unless you know what you're doing.

var DefaultOptions = Options{
	EntryIdxMode:         HintKeyValAndRAMIdxMode,
	SegmentSize:          defaultSegmentSize,
	NodeNum:              1,
	RWMode:               FileIO,
	SyncEnable:           true,
	StartFileLoadingMode: MMap,
}

Transactions

NutsDB allows only one read-write transaction at a time but allows as many read-only transactions as you want at a time. Each transaction has a consistent view of the data as it existed when the transaction started.

When a transaction fails, it will roll back, and revert all changes that occurred to the database during that transaction. if set the option SyncEnable true When a read/write transaction succeeds all changes are persisted to disk.

Creating transaction from the DB is thread safe.

Read-write transactions

err := db.Update(
	func(tx *nutsdb.Tx) error {
	...
	return nil
})

Read-only transactions

err := db.View(
	func(tx *nutsdb.Tx) error {
	...
	return nil
})

Managing transactions manually

The DB.View() and DB.Update() functions are wrappers around the DB.Begin() function. These helper functions will start the transaction, execute a function, and then safely close your transaction if an error is returned. This is the recommended way to use NutsDB transactions.

However, sometimes you may want to manually start and end your transactions. You can use the DB.Begin() function directly but please be sure to close the transaction.

 // Start a write transaction.
tx, err := db.Begin(true)
if err != nil {
    return err
}

bucket := "bucket1"
key := []byte("foo")
val := []byte("bar")

// Use the transaction.
if err = tx.Put(bucket, key, val, Persistent); err != nil {
	// Rollback the transaction.
	tx.Rollback()
} else {
	// Commit the transaction and check for error.
	if err = tx.Commit(); err != nil {
		tx.Rollback()
		return err
	}
}

Using buckets

Buckets are collections of key/value pairs within the database. All keys in a bucket must be unique. Bucket can be interpreted as a table or namespace. So you can store the same key in different bucket.

key := []byte("key001")
val := []byte("val001")

bucket001 := "bucket001"
if err := db.Update(
	func(tx *nutsdb.Tx) error {
		if err := tx.Put(bucket001, key, val, 0); err != nil {
			return err
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

bucket002 := "bucket002"
if err := db.Update(
	func(tx *nutsdb.Tx) error {
		if err := tx.Put(bucket002, key, val, 0); err != nil {
			return err
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

Also, this bucket is related to the data structure you use. Different data index structures that use the same bucket are also different. For example, you define a bucket named bucket_foo, so you need to use the list data structure, use tx.RPush to add data, you must query or retrieve from this bucket_foo data structure, use tx.RPop, tx.LRange, etc. You cannot use tx.Get (same index type as tx.GetAll, tx.Put, tx.Delete, tx.RangeScan, etc.) to read the data in this bucket_foo, because the index structure is different. Other data structures such as Set, Sorted Set are the same.

Using key/value pairs

To save a key/value pair to a bucket, use the tx.Put method:

if err := db.Update(
	func(tx *nutsdb.Tx) error {
	key := []byte("name1")
	val := []byte("val1")
	bucket := "bucket1"
	if err := tx.Put(bucket, key, val, 0); err != nil {
		return err
	}
	return nil
}); err != nil {
	log.Fatal(err)
}

This will set the value of the "name1" key to "val1" in the bucket1 bucket.

To update the the value of the "name1" key,we can still use the tx.Put function:

if err := db.Update(
	func(tx *nutsdb.Tx) error {
	key := []byte("name1")
	val := []byte("val1-modify") // Update the value
	bucket := "bucket1"
	if err := tx.Put(bucket, key, val, 0); err != nil {
		return err
	}
	return nil
}); err != nil {
	log.Fatal(err)
}

To retrieve this value, we can use the tx.Get function:

if err := db.View(
func(tx *nutsdb.Tx) error {
	key := []byte("name1")
	bucket := "bucket1"
	if e, err := tx.Get(bucket, key); err != nil {
		return err
	} else {
		fmt.Println(string(e.Value)) // "val1-modify"
	}
	return nil
}); err != nil {
	log.Println(err)
}

Use the tx.Delete() function to delete a key from the bucket.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
	key := []byte("name1")
	bucket := "bucket1"
	if err := tx.Delete(bucket, key); err != nil {
		return err
	}
	return nil
}); err != nil {
	log.Fatal(err)
}

Using TTL(Time To Live)

NusDB supports TTL(Time to Live) for keys, you can use tx.Put function with a ttl parameter.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
	key := []byte("name1")
	val := []byte("val1")
	bucket := "bucket1"
	
	// If set ttl = 0 or Persistent, this key will nerver expired.
	// Set ttl = 60 , after 60 seconds, this key will expired.
	if err := tx.Put(bucket, key, val, 60); err != nil {
		return err
	}
	return nil
}); err != nil {
	log.Fatal(err)
}

Iterating over keys

NutsDB stores its keys in byte-sorted order within a bucket. This makes sequential iteration over these keys extremely fast.

Prefix scans

To iterate over a key prefix, we can use PrefixScan function, and the parameters offsetNum and limitNum constrain the number of entries returned :

if err := db.View(
	func(tx *nutsdb.Tx) error {
		prefix := []byte("user_")
		bucket := "user_list"
		// Constrain 100 entries returned 
		if entries, _, err := tx.PrefixScan(bucket, prefix, 25, 100); err != nil {
			return err
		} else {
			for _, entry := range entries {
				fmt.Println(string(entry.Key), string(entry.Value))
			}
		}
		return nil
	}); err != nil {
		log.Fatal(err)
}

Prefix search scans

To iterate over a key prefix with search by regular expression on a second part of key without prefix, we can use PrefixSearchScan function, and the parameters offsetNum, limitNum constrain the number of entries returned :

if err := db.View(
	func(tx *nutsdb.Tx) error {
		prefix := []byte("user_")
		reg := "username"
		bucket := "user_list"
		// Constrain 100 entries returned 
		if entries, _, err := tx.PrefixSearchScan(bucket, prefix, reg, 25, 100); err != nil {
			return err
		} else {
			for _, entry := range entries {
				fmt.Println(string(entry.Key), string(entry.Value))
			}
		}
		return nil
	}); err != nil {
		log.Fatal(err)
}

Range scans

To scan over a range, we can use RangeScan function. For example:

if err := db.View(
	func(tx *nutsdb.Tx) error {
		// Assume key from user_0000000 to user_9999999.
		// Query a specific user key range like this.
		start := []byte("user_0010001")
		end := []byte("user_0010010")
		bucket := "user_list"
		if entries, err := tx.RangeScan(bucket, start, end); err != nil {
			return err
		} else {
			for _, entry := range entries {
				fmt.Println(string(entry.Key), string(entry.Value))
			}
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

Get all

To scan all keys and values of the bucket stored, we can use GetAll function. For example:

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "user_list"
		entries, err := tx.GetAll(bucket)
		if err != nil {
			return err
		}

		for _, entry := range entries {
			fmt.Println(string(entry.Key),string(entry.Value))
		}

		return nil
	}); err != nil {
	log.Println(err)
}

Merge Operation

NutsDB supports merge operation. you can use db.Merge() function removes dirty data and reduce data redundancy. Call this function from a read-write transaction. It will effect other write request. So you can execute it at the appropriate time.

err := db.Merge()
if err != nil {
    ...
}

Notice: the HintBPTSparseIdxMode mode does not support the merge operation of the current version.

Database backup

NutsDB is easy to backup. You can use the db.Backup() function at given dir, call this function from a read-only transaction, it will perform a hot backup and not block your other database reads and writes.

err = db.Backup(dir)
if err != nil {
   ...
}

Using other data structures

The syntax here is modeled after Redis commands

List

RPush

Inserts the values at the tail of the list stored in the bucket at given bucket,key and values.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "bucketForList"
		key := []byte("myList")
		val := []byte("val1")
		return tx.RPush(bucket, key, val)
	}); err != nil {
	log.Fatal(err)
}
LPush

Inserts the values at the head of the list stored in the bucket at given bucket,key and values.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
	        bucket := "bucketForList"
		key := []byte("myList")
		val := []byte("val2")
		return tx.LPush(bucket, key, val)
	}); err != nil {
	log.Fatal(err)
}
LPop

Removes and returns the first element of the list stored in the bucket at given bucket and key.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
	        bucket := "bucketForList"
		key := []byte("myList")
		if item, err := tx.LPop(bucket, key); err != nil {
			return err
		} else {
			fmt.Println("LPop item:", string(item))
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
LPeek

Returns the first element of the list stored in the bucket at given bucket and key.

if err := db.View(
	func(tx *nutsdb.Tx) error {
	        bucket := "bucketForList"
		key := []byte("myList")
		if item, err := tx.LPeek(bucket, key); err != nil {
			return err
		} else {
			fmt.Println("LPeek item:", string(item)) //val11
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
RPop

Removes and returns the last element of the list stored in the bucket at given bucket and key.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
	        bucket := "bucketForList"
		key := []byte("myList")
		if item, err := tx.RPop(bucket, key); err != nil {
			return err
		} else {
			fmt.Println("RPop item:", string(item))
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
RPeek

Returns the last element of the list stored in the bucket at given bucket and key.

if err := db.View(
	func(tx *nutsdb.Tx) error {
	        bucket := "bucketForList"
		key := []byte("myList")
		if item, err := tx.RPeek(bucket, key); err != nil {
			return err
		} else {
			fmt.Println("RPeek item:", string(item))
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
LRange

Returns the specified elements of the list stored in the bucket at given bucket,key, start and end. The offsets start and stop are zero-based indexes 0 being the first element of the list (the head of the list), 1 being the next element and so on. Start and end can also be negative numbers indicating offsets from the end of the list, where -1 is the last element of the list, -2 the penultimate element and so on.

if err := db.View(
	func(tx *nutsdb.Tx) error {
	        bucket := "bucketForList"
		key := []byte("myList")
		if items, err := tx.LRange(bucket, key, 0, -1); err != nil {
			return err
		} else {
			//fmt.Println(items)
			for _, item := range items {
				fmt.Println(string(item))
			}
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
LRem

Note: This feature can be used starting from v0.6.0

Removes the first count occurrences of elements equal to value from the list stored in the bucket at given bucket,key,count. The count argument influences the operation in the following ways:

  • count > 0: Remove elements equal to value moving from head to tail.
  • count < 0: Remove elements equal to value moving from tail to head.
  • count = 0: Remove all elements equal to value.
if err := db.Update(
	func(tx *nutsdb.Tx) error {
	        bucket := "bucketForList"
		key := []byte("myList")
		return tx.LRem(bucket, key, 1, []byte("value11))
	}); err != nil {
	log.Fatal(err)
}
LSet

Sets the list element at index to value.

val11") } return nil }); err != nil { log.Fatal(err) } ">
if err := db.Update(
	func(tx *nutsdb.Tx) error {
	        bucket := "bucketForList"
		key := []byte("myList")
		if err := tx.LSet(bucket, key, 0, []byte("val11")); err != nil {
			return err
		} else {
			fmt.Println("LSet ok, index 0 item value => val11")
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
Ltrim

Trims an existing list so that it will contain only the specified range of elements specified. the offsets start and stop are zero-based indexes 0 being the first element of the list (the head of the list), 1 being the next element and so on.Start and end can also be negative numbers indicating offsets from the end of the list, where -1 is the last element of the list, -2 the penultimate element and so on.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
	        bucket := "bucketForList"
		key := []byte("myList")
		return tx.LTrim(bucket, key, 0, 1)
	}); err != nil {
	log.Fatal(err)
}
LSize

Returns the size of key in the bucket in the bucket at given bucket and key.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
	        bucket := "bucketForList"
		key := []byte("myList")
		if size,err := tx.LSize(bucket, key); err != nil {
			return err
		} else {
			fmt.Println("myList size is ",size)
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

Set

SAdd

Adds the specified members to the set stored int the bucket at given bucket,key and items.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
	        bucket := "bucketForSet"
		key := []byte("mySet")
		return tx.SAdd(bucket, key, []byte("a"), []byte("b"), []byte("c"))
	}); err != nil {
	log.Fatal(err)
}
SAreMembers

Returns if the specified members are the member of the set int the bucket at given bucket,key and items.

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "bucketForSet"
		key := []byte("mySet")
		if ok, err := tx.SAreMembers(bucket, key, []byte("a"), []byte("b"), []byte("c")); err != nil {
			return err
		} else {
			fmt.Println("SAreMembers:", ok)
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
SCard

Returns the set cardinality (number of elements) of the set stored in the bucket at given bucket and key.

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "bucketForSet"
		key := []byte("mySet")
		if num, err := tx.SCard(bucket, key); err != nil {
			return err
		} else {
			fmt.Println("SCard:", num)
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
SDiffByOneBucket

Returns the members of the set resulting from the difference between the first set and all the successive sets in one bucket.

key1 := []byte("mySet1")
key2 := []byte("mySet2")
bucket := "bucketForSet"

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return tx.SAdd(bucket, key1, []byte("a"), []byte("b"), []byte("c"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return tx.SAdd(bucket, key2, []byte("c"), []byte("d"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		if items, err := tx.SDiffByOneBucket(bucket, key1, key2); err != nil {
			return err
		} else {
			fmt.Println("SDiffByOneBucket:", items)
			for _, item := range items {
				fmt.Println("item", string(item))
			}
			//item a
			//item b
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
SDiffByTwoBuckets

Returns the members of the set resulting from the difference between the first set and all the successive sets in two buckets.

bucket1 := "bucket1"
key1 := []byte("mySet1")

bucket2 := "bucket2"
key2 := []byte("mySet2")

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return tx.SAdd(bucket1, key1, []byte("a"), []byte("b"), []byte("c"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return tx.SAdd(bucket2, key2, []byte("c"), []byte("d"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		if items, err := tx.SDiffByTwoBuckets(bucket1, key1, bucket2, key2); err != nil {
			return err
		} else {
			fmt.Println("SDiffByTwoBuckets:", items)
			for _, item := range items {
				fmt.Println("item", string(item))
			}
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
SHasKey

Returns if the set in the bucket at given bucket and key.

bucket := "bucketForSet"

if err := db.View(
	func(tx *nutsdb.Tx) error {
		if ok, err := tx.SHasKey(bucket, []byte("mySet")); err != nil {
			return err
		} else {
			fmt.Println("SHasKey", ok)
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
SIsMember

Returns if member is a member of the set stored int the bucket at given bucket,key and item.

bucket := "bucketForSet"

if err := db.View(
	func(tx *nutsdb.Tx) error {
		if ok, err := tx.SIsMember(bucket, []byte("mySet"), []byte("a")); err != nil {
			return err
		} else {
			fmt.Println("SIsMember", ok)
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
SMembers

Returns all the members of the set value stored int the bucket at given bucket and key.

bucket := "bucketForSet"

if err := db.View(
	func(tx *nutsdb.Tx) error {
		if items, err := tx.SMembers(bucket, []byte("mySet")); err != nil {
			return err
		} else {
			fmt.Println("SMembers", items)
			for _, item := range items {
				fmt.Println("item", string(item))
			}
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
SMoveByOneBucket

Moves member from the set at source to the set at destination in one bucket.

bucket3 := "bucket3"

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return SAdd(bucket3, []byte("mySet1"), []byte("a"), []byte("b"), []byte("c"))
	}); err != nil {
	log.Fatal(err)
}
if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return tx.SAdd(bucket3, []byte("mySet2"), []byte("c"), []byte("d"), []byte("e"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		if ok, err := tx.SMoveByOneBucket(bucket3, []byte("mySet1"), []byte("mySet2"), []byte("a")); err != nil {
			return err
		} else {
			fmt.Println("SMoveByOneBucket", ok)
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		if items, err := tx.SMembers(bucket3, []byte("mySet1")); err != nil {
			return err
		} else {
			fmt.Println("after SMoveByOneBucket bucket3 mySet1 SMembers", items)
			for _, item := range items {
				fmt.Println("item", string(item))
			}
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		if items, err := tx.SMembers(bucket3, []byte("mySet2")); err != nil {
			return err
		} else {
			fmt.Println("after SMoveByOneBucket bucket3 mySet2 SMembers", items)
			for _, item := range items {
				fmt.Println("item", string(item))
			}
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
SMoveByTwoBuckets

Moves member from the set at source to the set at destination in two buckets.

bucket4 := "bucket4"
bucket5 := "bucket5"
if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return tx.SAdd(bucket4, []byte("mySet1"), []byte("a"), []byte("b"), []byte("c"))
	}); err != nil {
	log.Fatal(err)
}
if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return tx.SAdd(bucket5, []byte("mySet2"), []byte("c"), []byte("d"), []byte("e"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		if ok, err := tx.SMoveByTwoBuckets(bucket4, []byte("mySet1"), bucket5, []byte("mySet2"), []byte("a")); err != nil {
			return err
		} else {
			fmt.Println("SMoveByTwoBuckets", ok)
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		if items, err := tx.SMembers(bucket4, []byte("mySet1")); err != nil {
			return err
		} else {
			fmt.Println("after SMoveByTwoBuckets bucket4 mySet1 SMembers", items)
			for _, item := range items {
				fmt.Println("item", string(item))
			}
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		if items, err := tx.SMembers(bucket5, []byte("mySet2")); err != nil {
			return err
		} else {
			fmt.Println("after SMoveByTwoBuckets bucket5 mySet2 SMembers", items)
			for _, item := range items {
				fmt.Println("item", string(item))
			}
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
SPop

Removes and returns one or more random elements from the set value store in the bucket at given bucket and key.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		key := []byte("mySet")
		if item, err := tx.SPop(bucket, key); err != nil {
			return err
		} else {
			fmt.Println("SPop item from mySet:", string(item))
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
SRem

Removes the specified members from the set stored int the bucket at given bucket,key and items.

bucket6:="bucket6"
if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return tx.SAdd(bucket6, []byte("mySet"), []byte("a"), []byte("b"), []byte("c"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		if err := tx.SRem(bucket6, []byte("mySet"), []byte("a")); err != nil {
			return err
		} else {
			fmt.Println("SRem ok")
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		if items, err := tx.SMembers(bucket6, []byte("mySet")); err != nil {
			return err
		} else {
			fmt.Println("SMembers items:", items)
			for _, item := range items {
				fmt.Println("item:", string(item))
			}
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
SUnionByOneBucket

The members of the set resulting from the union of all the given sets in one bucket.

bucket7 := "bucket1"
key1 := []byte("mySet1")
key2 := []byte("mySet2")

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return tx.SAdd(bucket7, key1, []byte("a"), []byte("b"), []byte("c"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return tx.SAdd(bucket7, key2, []byte("c"), []byte("d"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		if items, err := tx.SUnionByOneBucket(bucket7, key1, key2); err != nil {
			return err
		} else {
			fmt.Println("SUnionByOneBucket:", items)
			for _, item := range items {
				fmt.Println("item", string(item))
			}
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
SUnionByTwoBuckets

The members of the set resulting from the union of all the given sets in two buckets.

bucket8 := "bucket1"
key1 := []byte("mySet1")

bucket9 := "bucket2"
key2 := []byte("mySet2")

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return tx.SAdd(bucket8, key1, []byte("a"), []byte("b"), []byte("c"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		return tx.SAdd(bucket9, key2, []byte("c"), []byte("d"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		if items, err := tx.SUnionByTwoBuckets(bucket8, key1, bucket9, key2); err != nil {
			return err
		} else {
			fmt.Println("SUnionByTwoBucket:", items)
			for _, item := range items {
				fmt.Println("item", string(item))
			}
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

Sorted Set

ZAdd

Adds the specified member with the specified score and the specified value to the sorted set stored at bucket.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet1"
		key := []byte("key1")
		return tx.ZAdd(bucket, key, 1, []byte("val1"))
	}); err != nil {
	log.Fatal(err)
}
ZCard

Returns the sorted set cardinality (number of elements) of the sorted set stored at bucket.

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet1"
		if num, err := tx.ZCard(bucket); err != nil {
			return err
		} else {
			fmt.Println("ZCard num", num)
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
ZCount

Returns the number of elements in the sorted set at bucket with a score between min and max and opts.

Opts includes the following parameters:

  • Limit int // limit the max nodes to return
  • ExcludeStart bool // exclude start value, so it search in interval (start, end] or (start, end)
  • ExcludeEnd bool // exclude end value, so it search in interval [start, end) or (start, end)
if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet1"
		if num, err := tx.ZCount(bucket, 0, 1, nil); err != nil {
			return err
		} else {
			fmt.Println("ZCount num", num)
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
ZGetByKey

Returns node in the bucket at given bucket and key.

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet1"
		key := []byte("key2")
		if node, err := tx.ZGetByKey(bucket, key); err != nil {
			return err
		} else {
			fmt.Println("ZGetByKey key2 val:", string(node.Value))
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
ZMembers

Returns all the members of the set value stored at bucket.

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet1"
		if nodes, err := tx.ZMembers(bucket); err != nil {
			return err
		} else {
			fmt.Println("ZMembers:", nodes)

			for _, node := range nodes {
				fmt.Println("member:", node.Key(), string(node.Value))
			}
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
ZPeekMax

Returns the member with the highest score in the sorted set stored at bucket.

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet1"
		if node, err := tx.ZPeekMax(bucket); err != nil {
			return err
		} else {
			fmt.Println("ZPeekMax:", string(node.Value))
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
ZPeekMin

Returns the member with lowest score in the sorted set stored at bucket.

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet1"
		if node, err := tx.ZPeekMin(bucket); err != nil {
			return err
		} else {
			fmt.Println("ZPeekMin:", string(node.Value))
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
ZPopMax

Removes and returns the member with the highest score in the sorted set stored at bucket.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet1"
		if node, err := tx.ZPopMax(bucket); err != nil {
			return err
		} else {
			fmt.Println("ZPopMax:", string(node.Value)) //val3
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
ZPopMin

Removes and returns the member with the lowest score in the sorted set stored at bucket.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet1"
		if node, err := tx.ZPopMin(bucket); err != nil {
			return err
		} else {
			fmt.Println("ZPopMin:", string(node.Value)) //val1
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
ZRangeByRank

Returns all the elements in the sorted set in one bucket at bucket and key with a rank between start and end (including elements with rank equal to start or end).

// ZAdd add items
if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet2"
		key1 := []byte("key1")
		return tx.ZAdd(bucket, key1, 1, []byte("val1"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet2"
		key2 := []byte("key2")
		return tx.ZAdd(bucket, key2, 2, []byte("val2"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet2"
		key3 := []byte("key3")
		return tx.ZAdd(bucket, key3, 3, []byte("val3"))
	}); err != nil {
	log.Fatal(err)
}

// ZRangeByRank
if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet2"
		if nodes, err := tx.ZRangeByRank(bucket, 1, 2); err != nil {
			return err
		} else {
			fmt.Println("ZRangeByRank nodes :", nodes)
			for _, node := range nodes {
				fmt.Println("item:", node.Key(), node.Score())
			}
			
			//item: key1 1
			//item: key2 2
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
ZRangeByScore

Returns all the elements in the sorted set at key with a score between min and max. And the parameter Opts is the same as ZCount's.

// ZAdd
if err := db.Update(
		func(tx *nutsdb.Tx) error {
			bucket := "myZSet3"
			key1 := []byte("key1")
			return tx.ZAdd(bucket, key1, 70, []byte("val1"))
		}); err != nil {
		log.Fatal(err)
	}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet3"
		key2 := []byte("key2")
		return tx.ZAdd(bucket, key2, 90, []byte("val2"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet3"
		key3 := []byte("key3")
		return tx.ZAdd(bucket, key3, 86, []byte("val3"))
	}); err != nil {
	log.Fatal(err)
}

// ZRangeByScore
if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet3"
		if nodes, err := tx.ZRangeByScore(bucket, 80, 100,nil); err != nil {
			return err
		} else {
			fmt.Println("ZRangeByScore nodes :", nodes)
			for _, node := range nodes {
				fmt.Println("item:", node.Key(), node.Score())
			}
			//item: key3 86
			//item: key2 90
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}	
ZRank

Returns the rank of member in the sorted set stored in the bucket at given bucket and key, with the scores ordered from low to high.

// ZAdd
if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet4"
		key1 := []byte("key1")
		return tx.ZAdd(bucket, key1, 70, []byte("val1"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet4"
		key2 := []byte("key2")
		return tx.ZAdd(bucket, key2, 90, []byte("val2"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet4"
		key3 := []byte("key3")
		return tx.ZAdd(bucket, key3, 86, []byte("val3"))
	}); err != nil {
	log.Fatal(err)
}

// ZRank
if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet4"
		key1 := []byte("key1")
		if rank, err := tx.ZRank(bucket, key1); err != nil {
			return err
		} else {
			fmt.Println("key1 ZRank :", rank) // key1 ZRank : 1
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

ZRevRank

Returns the rank of member in the sorted set stored in the bucket at given bucket and key,with the scores ordered from high to low.

// ZAdd
if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet8"
		key1 := []byte("key1")
		return tx.ZAdd(bucket, key1, 10, []byte("val1"))
	}); err != nil {
	log.Fatal(err)
}
if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet8"
		key2 := []byte("key2")
		return tx.ZAdd(bucket, key2, 20, []byte("val2"))
	}); err != nil {
	log.Fatal(err)
}
if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet8"
		key3 := []byte("key3")
		return tx.ZAdd(bucket, key3, 30, []byte("val3"))
	}); err != nil {
	log.Fatal(err)
}

// ZRevRank
if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet8"
		if rank, err := tx.ZRevRank(bucket, []byte("key3")); err != nil {
			return err
		} else {
			fmt.Println("ZRevRank key1 rank:", rank) //ZRevRank key3 rank: 1
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
ZRem

Removes the specified members from the sorted set stored in one bucket at given bucket and key.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet5"
		key1 := []byte("key1")
		return tx.ZAdd(bucket, key1, 10, []byte("val1"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet5"
		key2 := []byte("key2")
		return tx.ZAdd(bucket, key2, 20, []byte("val2"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet5"
		if nodes,err := tx.ZMembers(bucket); err != nil {
			return err
		} else {
			fmt.Println("before ZRem key1, ZMembers nodes",nodes)
			for _,node:=range nodes {
				fmt.Println("item:",node.Key(),node.Score())
			}
		}
		// before ZRem key1, ZMembers nodes map[key1:0xc00008cfa0 key2:0xc00008d090]
		// item: key1 10
		// item: key2 20
		return nil
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet5"
		if err := tx.ZRem(bucket, "key1"); err != nil {
			return err
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet5"
		if nodes,err := tx.ZMembers(bucket); err != nil {
			return err
		} else {
			fmt.Println("after ZRem key1, ZMembers nodes",nodes)
			for _,node:=range nodes {
				fmt.Println("item:",node.Key(),node.Score())
			}
			// after ZRem key1, ZMembers nodes map[key2:0xc00008d090]
			// item: key2 20
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
ZRemRangeByRank

Removes all elements in the sorted set stored in one bucket at given bucket with rank between start and end. The rank is 1-based integer. Rank 1 means the first node; Rank -1 means the last node.

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet6"
		key1 := []byte("key1")
		return tx.ZAdd(bucket, key1, 10, []byte("val1"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet6"
		key2 := []byte("key2")
		return tx.ZAdd(bucket, key2, 20, []byte("val2"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet6"
		key3 := []byte("key3")
		return tx.ZAdd(bucket, key3, 30, []byte("val2"))
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet6"
		if nodes,err := tx.ZMembers(bucket); err != nil {
			return err
		} else {
			fmt.Println("before ZRemRangeByRank, ZMembers nodes",nodes)
			for _,node:=range nodes {
				fmt.Println("item:",node.Key(),node.Score())
			}
			// before ZRemRangeByRank, ZMembers nodes map[key3:0xc00008d450 key1:0xc00008d270 key2:0xc00008d360]
			// item: key1 10
			// item: key2 20
			// item: key3 30
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

if err := db.Update(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet6"
		if err := tx.ZRemRangeByRank(bucket, 1,2); err != nil {
			return err
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet6"
		if nodes,err := tx.ZMembers(bucket); err != nil {
			return err
		} else {
			fmt.Println("after ZRemRangeByRank, ZMembers nodes",nodes)
			for _,node:=range nodes {
				fmt.Println("item:",node.Key(),node.Score())
			}
			// after ZRemRangeByRank, ZMembers nodes map[key3:0xc00008d450]
			// item: key3 30
			// key1 ZScore 10
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}
ZScore

Returns the score of member in the sorted set in the bucket at given bucket and key.

if err := db.View(
	func(tx *nutsdb.Tx) error {
		bucket := "myZSet7"
		if score,err := tx.ZScore(bucket, []byte("key1")); err != nil {
			return err
		} else {
			fmt.Println("ZScore key1 score:",score)
		}
		return nil
	}); err != nil {
	log.Fatal(err)
}

Comparison with other databases

BoltDB

BoltDB is similar to NutsDB, both use B+tree and support transaction. However, Bolt uses a B+tree internally and only a single file, and NutsDB is based on bitcask model with multiple log files. NutsDB supports TTL and many data structures, but BoltDB not supports them .

LevelDB, RocksDB

LevelDB and RocksDB are based on a log-structured merge-tree (LSM tree).An LSM tree optimizes random writes by using a write ahead log and multi-tiered, sorted files called SSTables. LevelDB does not have transactions. It supports batch writing of key/values pairs and it supports read snapshots but it will not give you the ability to do a compare-and-swap operation safely. NutsDB supports many data structures, but they not support them.

Badger

Badger is based in LSM tree with value log. It designed for SSDs. It also supports transaction and TTL. But in my benchmark its write performance is not as good as i thought. In addition, NutsDB supports data structures such as list、set、sorted set, but Badger not supports them.

Benchmarks

Tested kvstore

Selected kvstore which is embedded, persistence and support transactions.

  • BadgerDB (master branch with default options)
  • BoltDB (master branch with default options)
  • NutsDB (master branch with default options or custom options)

Benchmark System:

  • Go Version : go1.11.4 darwin/amd64
  • OS: Mac OS X 10.13.6
  • Architecture: x86_64
  • 16 GB 2133 MHz LPDDR3
  • CPU: 3.1 GHz Intel Core i7

Benchmark results:

badger 2019/03/11 18:06:05 INFO: All 0 tables opened in 0s
goos: darwin
goarch: amd64
pkg: github.com/xujiajun/kvstore-bench
BenchmarkBadgerDBPutValue64B-8    	   10000	    112382 ns/op	    2374 B/op	      74 allocs/op
BenchmarkBadgerDBPutValue128B-8   	   20000	     94110 ns/op	    2503 B/op	      74 allocs/op
BenchmarkBadgerDBPutValue256B-8   	   20000	     93480 ns/op	    2759 B/op	      74 allocs/op
BenchmarkBadgerDBPutValue512B-8   	   10000	    101407 ns/op	    3271 B/op	      74 allocs/op
BenchmarkBadgerDBGet-8            	 1000000	      1552 ns/op	     416 B/op	       9 allocs/op
BenchmarkBoltDBPutValue64B-8      	   10000	    203128 ns/op	   21231 B/op	      62 allocs/op
BenchmarkBoltDBPutValue128B-8     	    5000	    229568 ns/op	   13716 B/op	      64 allocs/op
BenchmarkBoltDBPutValue256B-8     	   10000	    196513 ns/op	   17974 B/op	      64 allocs/op
BenchmarkBoltDBPutValue512B-8     	   10000	    199805 ns/op	   17064 B/op	      64 allocs/op
BenchmarkBoltDBGet-8              	 1000000	      1122 ns/op	     592 B/op	      10 allocs/op
BenchmarkNutsDBPutValue64B-8      	   30000	     53614 ns/op	     626 B/op	      14 allocs/op
BenchmarkNutsDBPutValue128B-8     	   30000	     51998 ns/op	     664 B/op	      13 allocs/op
BenchmarkNutsDBPutValue256B-8     	   30000	     53958 ns/op	     920 B/op	      13 allocs/op
BenchmarkNutsDBPutValue512B-8     	   30000	     55787 ns/op	    1432 B/op	      13 allocs/op
BenchmarkNutsDBGet-8              	 2000000	       661 ns/op	      88 B/op	       3 allocs/op
BenchmarkNutsDBGetByHintKey-8     	   50000	     27255 ns/op	     840 B/op	      16 allocs/op
PASS
ok  	github.com/xujiajun/kvstore-bench	83.856s

Conclusions:

Put(write) Performance:

NutsDB is fastest. NutsDB is 2-5x faster than BoltDB, 0.5-2x faster than BadgerDB. And BadgerDB is 1-3x faster than BoltDB.

Get(read) Performance:

All are fast. And NutsDB is 1x faster than others. And NutsDB reads with HintKey option is much slower than its default option way.

the benchmark code can be found in the gokvstore-bench repo.

Caveats & Limitations

Index mode

From the version v0.3.0, NutsDB supports two modes about entry index: HintKeyValAndRAMIdxMode and HintKeyAndRAMIdxMode. From the version v0.5.0, NutsDB supports HintBPTSparseIdxMode mode.

The default mode use HintKeyValAndRAMIdxMode, entries are indexed base on RAM, so its read/write performance is fast. but can’t handle databases much larger than the available physical RAM. If you set the HintKeyAndRAMIdxMode mode, HintIndex will not cache the value of the entry. Its write performance is also fast. To retrieve a key by seeking to offset relative to the start of the data file, so its read performance more slowly that RAM way, but it can save memory. The mode HintBPTSparseIdxMode is based b+ tree sparse index, this mode saves memory very much (1 billion data only uses about 80MB of memory). And other data structures such as list, set, sorted set only supported with mode HintKeyValAndRAMIdxMode. It cannot switch back and forth between modes because the index structure is different.

NutsDB will truncate data file if the active file is larger than SegmentSize, so the size of an entry can not be set larger than SegmentSize , defalut SegmentSize is 8MB, you can set it(opt.SegmentSize) as option before DB opening. Once set, it cannot be changed.

Support OS

NutsDB currently works on Mac OS, Linux and Windows.

About merge operation

The HintBPTSparseIdxMode mode does not support the merge operation of the current version.

About transactions

Recommend use the latest version.

Contact

Contributing

See CONTRIBUTING for details on submitting patches and the contribution workflow.

Acknowledgements

This package is inspired by the following:

License

The NutsDB is open-sourced software licensed under the Apache 2.0 license.

Issues
  • No durability guarantee (benchmark is misleading)

    No durability guarantee (benchmark is misleading)

    I was wondering why nutsdb is so fast, so I did a quick review on your code, and there is no flush/sync call in the code, you should call it to ensure durability and update the benchmark. 💪

    Also, the README should mention the byte endianness issue as well, and the isolation level is not clear.

    enhancement good issue 
    opened by damnever 13
  • runtime error: invalid memory address or nil pointer dereference

    runtime error: invalid memory address or nil pointer dereference

    Describe the bug A clear and concise description of what the bug is.

    runtime error: invalid memory address or nil pointer dereference /usr/local/go/src/runtime/panic.go:220 (0x404c295) panicmem: panic(memoryError) /usr/local/go/src/runtime/signal_unix.go:818 (0x404c265) sigpanic: panicmem() /***/pkg/mod/github.com/xujiajun/[email protected]/tx.go:123 (0x48d6518) (Tx).getTxID: node, err := snowflake.NewNode(tx.db.opt.NodeNum) //pkg/mod/github.com/xujiajun/[email protected]/tx.go:111 (0x48d649e) newTx: txID, err = tx.getTxID() //pkg/mod/github.com/xujiajun/[email protected]/tx.go:85 (0x48d6364) (*DB).Begin: tx, err = newTx(db, writable) /*r/pkg/mod/github.com/xujiajun/[email protected]/db.go:936 (0x48d5184) (DB).managed: tx, err := db.Begin(writable) //pkg/mod/github.com/xujiajun/[email protected]/db.go:293 (0x4969490) (*DB).View: return db.managed(false, fn)

    To Reproduce Steps to reproduce the behavior(Be specific!):

    1. 由于程序退出导致再次启动后每次read一个key时报错,退出重启无法恢复,必须删除本地的nutsdb文件才可以正常运行。
    2. 程序平时可以正常运行,无法确定什么时间退出会导致这个异常,异常出现后除了删除文件外无法自动恢复
    3. 错误出现在

    Give sample code if you can.

    func read(k1 string) (string, error) {
        var out string
        if err := db.View(
            func(tx *nutsdb.Tx) error {
                key := []byte(k1)
                e, err := tx.Get(bucket, key)
                if err != nil {
                    return err
                }
                out = string(e.Value)
    
                return nil
            }); err != nil {
    
            return "", err
        } 
            
        return out, nil
    }
    

    Expected behavior A clear and concise description of what you expected to happen. key不存在不会导致异常panic,可以返回错误信息进行处理

    What actually happens A clear and concise description of what actually happens.

    Screenshots If applicable, add screenshots to help explain your problem. image

    please complete the following information :

    • OS: [e.g. Ubuntu 16.04] Mac OSX
    • NutsDB Version [e.g. 0.4.0] github.com/xujiajun/nutsdb v0.8.0

    Additional context Add any other context about the problem here.

    opened by oceanchang 9
  • fatal error: runtime: out of memory

    fatal error: runtime: out of memory

    Describe the bug Hello, I want to do a simple function of network disconnection and continuous transmission of timing data. I rpush the time sequence data after disconnection into a list. After the network connection replies, the data is lpop from the list and sent through mqtt. However, memory overflow often occurs when fetching data. What should I do?

    fatal error: runtime: out of memory

    OS: Linux am335x 3.2.0 #595 Wed Jan 31 00:13:29 PST 2018 armv7l GNU/Linux

    opened by Hanker-13 9
  • example with GetAll doe not work

    example with GetAll doe not work

    Describe the bug tx.GetAll undefined (type *nutsdb.Tx has no field or method GetAll)

    To Reproduce

    	err := NDB.View(
    		func(tx *nutsdb.Tx) error {
    			entries, err := tx.GetAll(bucket)
    			if err != nil {
    				return err
    			}
    
    			for _, entry := range entries {
    				fmt.Println(string(entry.Key), string(entry.Value))
    			}
    
    			return nil
    		})
    

    Expected behavior tx.GetAll should be available, accoring to: https://xujiajun.cn/nutsdb/#get-all

    What actually happens An error :-(

    please complete the following information :

    • OS: Mac OS 10.14.6
    • NutsDB Version : 0.4.0
    v0.5.0 
    opened by marcelloh 7
  • fix 'constant 2147483648 overflows int'

    fix 'constant 2147483648 overflows int'

    Describe the bug I got a constant 2147483648 overflows int message in go build process for the ARM 32-bit architecture.

    To Reproduce Steps to reproduce the behavior:

    $ git clone https://github.com/michilu/boilerplate
    $ cd boilerplate
    $ git checkout 23bee11
    $ GO111MODULE=on GOOS=linux GOARCH=arm go build
    

    Expected behavior

    $ GO111MODULE=on GOOS=linux GOARCH=arm go build
    $
    

    no errors.

    What actually happens

    $ GO111MODULE=on GOOS=linux GOARCH=arm go build
    # github.com/xujiajun/nutsdb/ds/zset
    ../../go/pkg/mod/github.com/xujiajun/[email protected]/ds/zset/sortedset.go:291:13: constant 2147483648 overflows int
    

    Screenshots N/A

    please complete the following information :

    • OS:
      • macOS 10.13.6
      • Ubuntu 16.04
    • NutsDB Version: v0.4.0 14f036b (2019-08-30T02:35:07Z)

    Additional context N/A

    opened by michilu 7
  • db.buildIndexes error: when build activeDataIndex readAt err:  crc error

    db.buildIndexes error: when build activeDataIndex readAt err: crc error

    Describe the bug

    When initialising the database, nutsdb generates an error:

    db.buildIndexes error: when build activeDataIndex readAt err: crc error

    To Reproduce

    My db setup code is below:

    type Store struct {
    	nuts *nutsdb.DB
    	log  *logging.Info
    }
    
    func NewStore() *Store {
    	D := Store{}
    
    	var err error
    
    	opt := nutsdb.DefaultOptions
    	opt.Dir = "data/"
    	D.nuts, err = nutsdb.Open(opt)
    	if err != nil {
    		panic(err)
    	}
    	D.Set("cd", []byte("_"), []byte("")) // create a bucket
    	return &D
    }
    

    The error occurred with a db that has been working fine for some months, but also occurs with a new one. It first manifested when using goconvey to run tests concurrently. I'm running go1.17.2 linux/amd64 on Ubuntu 20.04.

    Expected behavior

    I expected the database to not return an error.

    What actually happens

    The error db.buildIndexes error: when build activeDataIndex readAt err: crc error is returned.

    please complete the following information :

    • OS: Ubuntu 20.04
    • NutsDB Version 0.8.0 (but the error first manifested on 0.6.0)

    Additional context Only that I'm a fan of nutsdb -- it was the first golang db I tried, and it slotted straight into the project without difficulty. I appreciate your work :-)

    opened by icecolbeveridge 6
  • It is possible to run

    It is possible to run "in memory"?

    Hi! I've found nutsdb nearly perfect for my project. Nevertheless, I would like to know if it is possible to use it exclusively as in-memory database (without persistence)?

    Thanks in advance,

    ER.

    enhancement 
    opened by ecerichter 6
  • Is this production ready? Any users using it in production?

    Is this production ready? Any users using it in production?

    Is this production ready? Any users using it in production?

    great work by the way! i dont see how this is batched because it's not "Batch()" function. so how to batch them? this is from an example batch you have

                if err := db.Update(
                        func(tx *nutsdb.Tx) error {
                                for i := (j-1)*10000; i < j*1000; i++ {
                                        key := []byte("namename" + strconv2.IntToStr(i))
                                        val := []byte("valvalvavalvalvalvavalvalvalvavalvalvalvaval" + strconv2.IntToStr(i))
                                        if err := tx.Put(bucket, key, val, 0); err != nil {
                                                return err
                                        }
                                }
                                return nil
                        }); err != nil {
                                log.Fatal(err)
                        }
    
                } 
    
    opened by hiqsociety 6
  • Tx read isolation

    Tx read isolation

    Hi @xujiajun,

    By looking at the code is not clear that we're getting values of our own pending writes when reading within a transaction. I might be probably missing something.

    Thanks.

    Example: https://github.com/xujiajun/nutsdb/blob/5705bf9fa79506cbc5b32bf9cf5a0a4940e84716/tx_bptree.go#L25

    opened by brunotm 6
  • 无法判断key是否存在

    无法判断key是否存在

    标题有些冒昧,请见谅~~~ 最近有嵌入式数据库的需求,故而找到了nutsdb,了解下来觉得nutsdb的设计还是非常好的,实现常用的数据结构,可以大幅度提高开发效率,但是nutsdb的接口定义感觉存在不够完美

    使用SHasKey查询Key是否存在的时候,如果bucket不存在会返回报错,但nutsdb没有单独提供bucket操作的接口

    如何实现SetNX逻辑?

    opened by nightmeng 5
  • GetAll() function does not work with HintBPTSparseIdxMode

    GetAll() function does not work with HintBPTSparseIdxMode

    Describe the bug Iterating the key-value store doesn't work with HintBPTSparseIdxMode indexing option.

    To Reproduce Steps to reproduce the behavior(Be specific!): Run this

    err := c.db.View(
    		func(tx *nutsdb.Tx) error {
    			entries, err := tx.GetAll(bucketName)
    			if err != nil {
    				return err
    			}
    
    			for _, entry := range entries {
    				callback(entry.Key, entry.Value)
    			}
    			return nil
    		},
    	)
    

    Expected behavior Keys and values should be iterated.

    What actually happens It says bucket not found, while debugging, i see that the routine doesn't has a separate function for ActiveBPTTree, and instead it works with normal indexing.

    Screenshots

    This seems like the issue -

    if index, ok := tx.db.BPTreeIdx[bucket]; ok {
    		records, err := index.All()
    		if err != nil {
    			return nil, ErrBucketEmpty
    		}
    
    		entries, err = tx.getHintIdxDataItemsWrapper(records, ScanNoLimit, entries, RangeScan)
    		if err != nil {
    			return nil, ErrBucketEmpty
    		}
    	}
    

    please complete the following information :

    • OS: Ubuntu 16.04
    • NutsDB Version Latest
    bug v0.5.0 
    opened by Ice3man543 5
  • 内存模式下没有PrefixScan

    内存模式下没有PrefixScan

    Describe the bug 内存模式下没有PrefixScan的功能,似乎也少了其他的一些功能

    To Reproduce Steps to reproduce the behavior(Be specific!):

    opts := inmemory.DefaultOptions
    db, err := inmemory.Open(opts)
    
    db.PrefixScan()
    

    Expected behavior 内存模式下有完整功能 ps: 文档也没写内存模式有什么功能缺失

    What actually happens A clear and concise description of what actually happens.

    Screenshots If applicable, add screenshots to help explain your problem.

    please complete the following information :

    • NutsDB Version v0.9.0

    Additional context Add any other context about the problem here.

    enhancement 
    opened by ttttmr 1
  • [新 API 提案] ExpireSet:将对应 bucket, key 的集合设置为 x 秒后过期

    [新 API 提案] ExpireSet:将对应 bucket, key 的集合设置为 x 秒后过期

    API

    ExpireSet(bucket, key string, ttl uint32)

    将对应 bucket, key 的集合设置为 ttl 秒后过期,过期后集合的索引和记录将被删除。

    实现

    Set 原索引结构为:map[string]*set.Set,拓展此结构,使其能够保存 bucket 下某个 key 的 TTL。

    1. ExpireSet() 被调用时,遍历所有 (key, value) 对应的数据库记录,将记录的 ttl 值更新为 TTL。

    2. SAdd 添加元素时,若索引结构中 key 关联了 TTL,所添加元素对应的数据库记录的 ttl 值将被设置为 TTL。

    3. 读取或写入元素时,若检测到 TTL < 0,则将触发所有元素的删除操作。

    4. 数据库初始化加载索引时,已过期的记录不加载(需在适当的时机将这些记录标记为删除),可根据某一条记录恢复 TTL(同一个key 对应的全部记录一定具有相同的 TTL 值)。

    PS

    List 的实现逻辑完全相同。

    也许 ZSet 也可以以类似方法实现(暂未确认)。

    相关 issue

    #102 #172

    enhancement proposal 
    opened by jukanntenn 4
  • v0.9.0 release

    v0.9.0 release

    • [Bug Fix] close file before error check &remove redundant judgments (#137) @xujiajun
    • [Bug Fix] update golang.org/x/sys to support go1.18 build (#139)@ag9920
    • [Bug Fix] when use merge, error: The process cannot access the file because it is being used by another process (#166) @xujiajun
    • [Bug Fix] fix code example. (#143) @gphper
    • [Bug Fix] merge error after delete bucket (#153) @xujiajun
    • [Perf] add fd cache(#164) @elliotchenzichang
    • [Perf] optimize sadd function inserting duplicate data leads to datafile growth (#146) @gphper
    • [Refactor] rewrite managed to support panic rollback (#136)@ag9920
    • [Refactor] errors: optimize error management (#163) @xpzouying
    • [Test] Update testcase: use testify test tools (#138) @xpzouying
    • [Test] change list and set test with table driven test and testify (#145) @bigdaronlee163
    • [Test] refactor db_test for string use testify (#147) @Rand01ph
    • [Test] add [bucket_meat/entry] unit test (#148) @gphper
    • [Test] update bptree unittest (#149) @xpzouying
    • [Test] Update tx bptree testcase (#155) @xpzouying
    • [Test] complete zset tests with testify (#151) @bigdaronlee163
    • [Test] optimization tx_bucket_test and bucket_meta_test (#156) @gphper
    • [Test] test:complete tx_zset tests with testify (#162) @bigdaronlee163
    • [Chore] remove unused member (#157) @xpzouying
    • [Style] format code comments etc. (#140) @moyrne

    Note

    Starting from v0.9.0, defaultSegmentSize in DefaultOptions has been adjusted from 8MB to 256MB. The original value is the default value, which needs to be manually changed to 8MB, otherwise the original data will not be parsed. The reason for the size adjustment here is that there is a cache for file descriptors starting from v0.9.0 (detail see https://github.com/nutsdb/nutsdb/pull/164 ), so users need to look at the number of fds they use on the server, which can be set manually. If you have any questions, you can open an issue.

    release 
    opened by xujiajun 1
  • Hope to build nutsdb together(希望共建nutsdb&加群交流)

    Hope to build nutsdb together(希望共建nutsdb&加群交流)

    I found that I don't have enough energy and ability to do nutsdb. I hope to attract more developers, especially those who are professional in this field. The development of nutsdb is still very early. When you join, the threshold for joining is very low. As long as you can go, even It’s okay if you don’t, just learn it. The most important thing is that you have enthusiasm, willingness, and learning ability. This project has great potential. Let’s create a well-known project together! If you have an idea, chat with me privately !

    Main directions:

      1. Database engine direction
      1. Tool ecology such as cli, visualization tools, etc.
      1. Document construction, etc.

    contact me:[email protected]

    NutsDB proposals:https://github.com/nutsdb/proposal/issues


    本人做nutsdb精力和能力都不够,希望能吸引更多开发者,特别是这方面专业的,nutsdb发展还很早期,等你的加入,希望你会go,有热情,有意愿,有学习力,有开源精神,愿意贡献项目,这个项目非常有潜力,让我们一起打造一个知名项目!有想法联系我!更新下招募情况:目前开发者群有意向开发者22名。

    主要方向:

    • 1、数据库引擎方向
    • 2、工具生态如cli、可视化工具等
    • 3、文档建设等

    proposal草案:https://github.com/nutsdb/proposal/issues

    微信加群

    如果已经过期,加我个人微信(xujiajun1234567),备注:nutsdb,我拉你进群

    大群已经超200了,大家可以扫码下面的中转群。

    图片名称 good first issue 
    opened by xujiajun 0
Releases(v0.9.0)
  • v0.9.0(Jun 17, 2022)

    v0.9.0 (2022-06-17)

    • [Bug Fix] close file before error check &remove redundant judgments (#137) @xujiajun
    • [Bug Fix] update golang.org/x/sys to support go1.18 build (#139)@ag9920
    • [Bug Fix] when use merge, error: The process cannot access the file because it is being used by another process (#166) @xujiajun
    • [Bug Fix] fix code example. (#143) @gphper
    • [Bug Fix] merge error after delete bucket (#153) @xujiajun
    • [Perf] add fd cache(#164) @elliotchenzichang
    • [Perf] optimize sadd function inserting duplicate data leads to datafile growth (#146) @gphper
    • [Refactor] rewrite managed to support panic rollback (#136)@ag9920
    • [Refactor] errors: optimize error management (#163) @xpzouying
    • [Test] Update testcase: use testify test tools (#138) @xpzouying
    • [Test] change list and set test with table driven test and testify (#145) @bigdaronlee163
    • [Test] refactor db_test for string use testify (#147) @Rand01ph
    • [Test] add [bucket_meat/entry] unit test (#148) @gphper
    • [Test] update bptree unittest (#149) @xpzouying
    • [Test] Update tx bptree testcase (#155) @xpzouying
    • [Test] complete zset tests with testify (#151) @bigdaronlee163
    • [Test] optimization tx_bucket_test and bucket_meta_test (#156) @gphper
    • [Test] test:complete tx_zset tests with testify (#162) @bigdaronlee163
    • [Chore] remove unused member (#157) @xpzouying
    • [Style] format code comments etc. (#140) @moyrne
    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Apr 1, 2022)

    v0.8.0 (2022-04-01)

    • [Perf] optimize tx commit for batch write (#132)
    • [Bug Fix] fix: open file by variable (#118)
    • [Bug Fix] fix close file before error check(#122)
    • [Bug Fix] fix rwManager.Close order(#133)
    • [Bug Fix] fix last entry status error (#134)
    • [Bug Fix] fix: read ErrIndexOutOfBound err
    • [CI] add unit-test action (#120)
    • [Chore] add constant ErrKeyNotFound and ErrKeyNotExist (#125)
    • [Chore] chore: remove unnecessary conversion (#126)
    • [Chore] chore(ds/list): replace for-loop with append (#127)
    • [Chore] add push check for lint, typo (#131)
    • [Style] style: fix typo and ineffectual assignment (#130)
    Source code(tar.gz)
    Source code(zip)
  • v0.7.1(Mar 6, 2022)

  • v0.7.0(Mar 6, 2022)

    • [New Feature] Support im memory db (#109)
    • [New Feature] Add backup with tar+gz (#111)
    • [New Feature] Add IterateBuckets() and DeleteBucket()
    • [Refactor] Refactor error (#112)
    • [Bug Fix] Windows The process cannot access the file because it is being used by another process. (#110)
    • [Docs] Update README && CHANGELOG
    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(Mar 21, 2021)

    • [New Feature] Add PrefixSearchScan() with regexp search ability(#53)
    • [New Feature] Allow put with timestamp (#88 )
    • [Bug Fix] Fix ZMembers bug (#58 )
    • [Bug Fix] Repeated key merge fix (#83 )
    • [Bug Fix] The LRem implementation is not consistent with the description (#92 )
    • [Refactor] Improve buildBPTreeRootIdxes file reading (#67)
    • [Docs] Update README && CHANGELOG
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Nov 28, 2019)

    • [New feature] Support EntryIdxMode: HintBPTSparseIdxMode
    • [New feature] Support GetAll() function for all models
    • [Bug Fix] Fix error too many open files in system
    • [Bug Fix] Fix constant 2147483648 overflows int
    • [Bug Fix] Fix when the number of files waiting to be merged not at least 2
    • [Bug Fix] Fix data pollution when executing the merge method
    • [Change] Modify Records type && Entries type
    • [Change] Refactor for tx Commit function
    • [Change] Update Iterating over keys about README
    • [Change] Fix some grammatical mistakes about README
    • [Change] Rename variable for func ReadBPTreeRootIdxAt
    • [Change] Add issue templates
    • [Change] Update README && CHANGELOG
    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Mar 15, 2019)

    • [New feature] Support mmap loading file
    • [Bug Fix] Fix tx bug when a tx commits
    • [Change] Add rwmanager interface
    • [Change] Add new options: RWMode, SyncEnable and StartFileLoadingMode
    • [Change] Clean up some codes
    • [Change] Update README && CHANGELOG
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Mar 12, 2019)

    • [New feature] Support persistence
    • [Bug Fix] Fix when fn is nil
    • [Change] Discard mmap package
    • [Change] Discard EntryIdxMode options: HintAndRAMIdxMode and HintAndMemoryMapIdxMode
    • [Change] Add new EntryIdxMode options: HintKeyValAndRAMIdxMode and HintKeyAndRAMIdxMode
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Mar 5, 2019)

    • [New feature] Support list
    • [New feature] Support set
    • [New feature] Support sorted set
    • [Bug Fix] Fix error when batch put operations
    • [Change] Update README && CHANGELOG
    Source code(tar.gz)
    Source code(zip)
  • v0.1.0(Feb 28, 2019)

    • [New feature] Support Put/Get/Delete Operations
    • [New feature] Support TTL
    • [New feature] Support Range/Prefix Scanning
    • [New feature] Support Merge Operation
    • [New feature] Support BackUp Operation
    • [New feature] Support Bucket
    Source code(tar.gz)
    Source code(zip)
Owner
徐佳军
You will never know what you can do till you try.
徐佳军
a persistent real-time key-value store, with the same redis protocol with powerful features

a fast NoSQL DB, that uses the same RESP protocol and capable to store terabytes of data, also it integrates with your mobile/web apps to add real-time features, soon you can use it as a document store cause it should become a multi-model db. Redix is used in production, you can use it in your apps with no worries.

Mohammed Al Ashaal 1.1k Aug 7, 2022
FlashDB is an embeddable, in-memory key/value database in Go

FlashDB is an embeddable, in-memory key/value database in Go (with Redis like commands and super easy to read)

Farhan 257 Aug 6, 2022
CrankDB is an ultra fast and very lightweight Key Value based Document Store.

CrankDB is a ultra fast, extreme lightweight Key Value based Document Store.

Shrey Batra 30 Apr 12, 2022
Pogreb is an embedded key-value store for read-heavy workloads written in Go.

Embedded key-value store for read-heavy workloads written in Go

Artem Krylysov 914 Aug 2, 2022
yakv is a simple, in-memory, concurrency-safe key-value store for hobbyists.

yakv (yak-v. (originally intended to be "yet-another-key-value store")) is a simple, in-memory, concurrency-safe key-value store for hobbyists. yakv provides persistence by appending transactions to a transaction log and restoring data from the transaction log on startup.

Aadhav Vignesh 5 Feb 24, 2022
ShockV is a simple key-value store with RESTful API

ShockV is a simple key-value store based on badgerDB with RESTful API. It's best suited for experimental project which you need a lightweight data store.

delihiros 2 Sep 26, 2021
Simple in memory key-value store.

Simple in memory key-value store. Development This project is written in Go. Make sure you have Go installed (download). Version 1.17 or higher is req

Mustafa Navruz 0 Nov 6, 2021
A simple in-memory key-value store application

vtec vtec, is a simple in-memory key-value store application. vtec provides persistence by appending transactions to a json file and restoring data fr

Ahmet Tek 3 Jun 22, 2022
Build a simple decomposed Key-Value store by implementing two services which communicate over gRPC.

Build a simple decomposed Key-Value store by implementing two services which communicate over gRPC.

Robert Otting 0 Feb 13, 2022
kvStore is a simple key/value in-memory store

kvStore is a simple key/value in-memory store. It is designed for the API. kvStore keeps records at /tmp/kvStore/dbName.db. You can specify server port, dbName and, file save interval in your RunServer(Addr, dbName) call.

Buğra Mengi 2 Feb 24, 2022
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Burak Sezer 2.3k Aug 12, 2022
KV - a toy in-memory key value store built primarily in an effort to write more go and check out grpc

KV KV is a toy in-memory key value store built primarily in an effort to write more go and check out grpc. This is still a work in progress. // downlo

Ali Mir 0 Dec 30, 2021
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Peter Bourgon 1.2k Aug 8, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 40.8k Aug 15, 2022
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jake Grogan 725 Aug 13, 2022
Multithreaded key value pair store using thread safe locking mechanism allowing concurrent reads

Project Amnesia A Multi-threaded key-value pair store using thread safe locking mechanism allowing concurrent reads. Curious to Try it out?? Check out

Nikhil Nayak 6 Apr 7, 2022
A rest-api that works with golang as an in-memory key value store

In Store A rest-api that works with golang as an in-memory key value store Usage Fist of all, clone the repo with the command below. You must have gol

Eyüp Arslan 0 Oct 24, 2021
Distributed key-value store

Keva Distributed key-value store General Demo Start the server docker-compose up --build Insert data curl -XPOST http://localhost:5555/storage/test1

Yaroslav Gaponov 0 Nov 15, 2021
Biscuit is a multi-region HA key-value store for your AWS infrastructure secrets.

Biscuit Biscuit is a simple key-value store for your infrastructure secrets. Is Biscuit right for me? Biscuit is most useful to teams already using AW

Doug 559 Jul 27, 2022