Low-level key/value store in pure Go.

Overview

Build Status Go Report Card Documentation

Description

Package slowpoke is a simple key/value store written using Go's standard library only. Keys are stored in memory (with persistence), values stored on disk.

Description on russian: https://habr.com/post/354224/

slowpoke

Motivation

Replace Bolt with a simpler and more efficient engine.

Slowpoke (from version 2.0) based on pudge

How it works

Keys are stored in memory with persistence to disk. Values stored on disk only.

Slowpoke is parallel

Server

GRPC Server example: okdb

Complex examples

typegram

zen platform for authors and their subscribers with a minimalistic design and user-friendly interface.

golang-gin-realworld-example-app

This codebase was created to demonstrate a fully fledged fullstack application built with Golang/Gin/Slowpoke including CRUD operations, authentication, routing, pagination, and more.

Basic example

package main

import (
	"fmt"

	"github.com/recoilme/slowpoke"
)

func main() {
	// create database
	file := "test/example.db"
	// close all opened database
	defer slowpoke.CloseAll()
	// init key/val
	key := []byte("foo")
	val := []byte("bar")
	//store
	slowpoke.Set(file, key, val)
	// get
	res, _ := slowpoke.Get(file, key)
	//result
	fmt.Println(string(res))
}

Lazy example

func TestGob(t *testing.T) {
	file := "test/gob.db"
	DeleteFile(file)
	defer CloseAll()
	type Post struct {
		Id       int
		Content  string
		Category string
	}

	for i := 0; i < 20; i++ {
		post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i)}
		err := SetGob(file, post.Id, post)
		ch(err, t)
	}

	for i := 0; i < 20; i++ {
		var post = new(Post)
		err := GetGob(file, i, post)
		ch(err, t)
		fmt.Println("i:", i, "Post:", post)
	}
}

Advanced example

type Post struct {
	Id       int
	Content  string
	Category string
}

func main() {
	posts := "test/posts"
	tags := "test/tags"
	var pairs [][]byte
	for i := 0; i < 40; i++ {
		id := make([]byte, 4)
		binary.BigEndian.PutUint32(id, uint32(i))
		post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i), Category: "Category:" + strconv.Itoa(i/10)}
		b, _ := json.Marshal(post)
		pairs = append(pairs, id)
		pairs = append(pairs, b)
		tag := fmt.Sprintf("%s:%08d", strconv.Itoa(i/10), i)
		//store only tags keys
		slowpoke.Set(tags, []byte(tag), nil)
	}
	//store posts fast
	slowpoke.Sets(posts, pairs)

	//get last 2 post key with offset 2
	limit := uint32(2)
	offset := uint32(2)
	order := false //desc
	keys, _ := slowpoke.Keys(posts, nil, limit, offset, order)
	fmt.Println(keys) //[[0 0 0 37] [0 0 0 36]]

	//get key/ values
	res := slowpoke.Gets(posts, keys)
	for k, v := range res {
		if k%2 == 0 {
			fmt.Print(binary.BigEndian.Uint32(v))
		} else {
			var p Post
			json.Unmarshal(v, &p)
			fmt.Println(p)
		}
	}
	//37{37 Content:37 Category:3}
	//36{36 Content:36 Category:3}

	//free from memory
	slowpoke.Close(posts)
	slowpoke.Close(tags)

	//open Db and read tags by prefix 2:* in ascending order
	tagsKeys, _ := slowpoke.Keys(tags, []byte("2:*"), 0, 0, true)
	for _, v := range tagsKeys {
		fmt.Print(string(v) + ", ")
	}
	//2:00000020, 2:00000021, 2:00000022, 2:00000023, 2:00000024, 2:00000025, 2:00000026, 2:00000027, 2:00000028, 2:00000029,
}

Api

All methods are thread-safe.

  • Set/Sets/SetGob

Store val and key. If the file does not exist it will be created.

  • Get/Gets/GetGob

Return the value for the given key or nil and an error. Get will open the database if necessary.

  • Keys

Return keys in ascending/descending order.

With limit and offset.

If from is not nil, return keys lexicographically greater than the from value.

If from ends with asterix *, return keys with the prefix equal to from without the asterix.

Documentation

Status

Used in production (master branch)

Benchmark

All tests here

Some tests, MacBook Pro (Retina, 13-inch, Early 2015)

Test 1

Number of keys: 1000000 Minimum key size: 16, maximum key size: 64 Minimum value size: 128, maximum value size: 512 Concurrency: 2

pogreb goleveldb bolt badgerdb pudge slowpoke pudge(mem)
1M (Put+Get), seconds 187 38 126 34 23 23 2
1M Put, ops/sec 5336 34743 8054 33539 47298 46789 439581
1M Get, ops/sec 1782423 98406 499871 220597 499172 445783 1652069
FileSize,Mb 568 357 552 487 358 358 358
Comments
  • '*' in key

    '*' in key

    Ключом для сохранения является произвольный массив байт. Он может быть порожден как счетчиком (с преобразованием числа в байты), так и строкой. Обычно ключ будет нормален. Но что произойдет с логикой работы выборки по ключам (slowpoke.Keys(…)), если в самом ключе будет '*'?

    Насколько я понимаю, я могу ожидать, что получу следующую запись после указанного ключа, и это будет работать до тех пор, пока чисто случайно не окажется, что ключ заканчивается на '*'. И вместо выборки со следующего ключа, я получу выборку все, начинающихся на префикс.

    Я прав? Или я "дую на воду"?

    opened by mdigger 11
  • В slowpoke нет режима «nosync»

    В slowpoke нет режима «nosync»

    В двух словах: не нашел код, который реализует очень нужную мне фичу "нет режима nosync". Такое заявление было только на хабре и в коде "with sync at end", в readme не сказано, поэтому не знаю насколько вам интересно, может быть новый курс без sync и это приемлимо.

    • slowpoke.Set = return pudge.Set
    • pudge.Set = Open(filename, nil).Set
    • pudge.Open(filename, cfg*Config) cfg=nil -> DefaultConfig
    • DefaultConfig.SyncInterval = 0 sec, 0 - disable sync
    • pudge.db.Close() сделал бы нужные вызовы .Sync(), но его не видно
    opened by temoto 6
  • replace sort function

    replace sort function

    replace sort on smart append:

    func TestSortedInsert(t *testing.T) {
    	var keys = make([][]byte, 0)
    	ins := func(b []byte) {
    		keysLen := len(keys)
    		found := sort.Search(keysLen, func(i int) bool {
    			return bytes.Compare(keys[i], b) >= 0
    		})
    		if found == 0 {
    			//prepend
    			keys = append([][]byte{b}, keys...)
    
    		} else {
    			if found >= keysLen {
    				//not found - postpend ;)
    				keys = append(keys, b)
    			} else {
    				//found
    				keys = append(keys[:found-1], b)
    				keys = append(keys, keys[found:]...)
    			}
    		}
    	}
    	for i := 20; i >= 0; i-- {
    		k := []byte(fmt.Sprintf("%04d", i))
    		ins(k)
    	}
    	for i, j := range keys {
    		logg(i)
    		logg(string(j))
    	}
    
    }
    
    opened by recoilme 4
  • Counter in Keys

    Counter in Keys

    Еще одна неожиданность для меня с выборкой записей после определенного идентификатора:

    func TestJSONValue(t *testing.T) {
        f := "test/JSONValue.db"
        DeleteFile(f)
        var key = make([]byte, 8)
        binary.BigEndian.PutUint32(key, 0xfa)
        for i := 0; i < 8; i++ {
            // id := uint32(i)
            id, err := Counter(f, []byte("test"))
            if err != nil {
                t.Fatal(err)
            }
            data, _ := json.Marshal(&struct {
                Text string `json:"text"`
            }{
                Text: fmt.Sprintf("text %04d", id),
            })
            binary.BigEndian.PutUint32(key[4:], uint32(id))
            if err := Set(f, key, data); err != nil {
                t.Fatal(err)
            }
            fmt.Printf("%x = %s\n", key, data)
        }
        binary.BigEndian.PutUint32(key[4:], uint32(4))
        keys, err := Keys(f, key, 0, 0, true)
        if err != nil {
            t.Fatal(err)
        }
        fmt.Println("----------------")
        for i, val := range Gets(f, keys) {
            if i%2 == 0 {
                fmt.Printf("%x = ", val)
                continue
            }
            fmt.Printf("%s\n", val)
        }
    }
    

    Совершенно не ожидал здесь увидеть значение счетчика:

    000000fa00000001 = {"text":"text 0001"}
    000000fa00000002 = {"text":"text 0002"}
    000000fa00000003 = {"text":"text 0003"}
    000000fa00000004 = {"text":"text 0004"}
    000000fa00000005 = {"text":"text 0005"}
    000000fa00000006 = {"text":"text 0006"}
    000000fa00000007 = {"text":"text 0007"}
    000000fa00000008 = {"text":"text 0008"}
    ----------------
    000000fa00000005 = {"text":"text 0005"}
    000000fa00000006 = {"text":"text 0006"}
    000000fa00000007 = {"text":"text 0007"}
    000000fa00000008 = {"text":"text 0008"}
    74657374 = �������[bin hex]
    
    opened by mdigger 3
  • add method Counter

    add method Counter

    return unique int, for use like counter Counter(file,key) (uint32?,error?)

    for speedup set+sync on every val%1000 == 0 - val+1000 for example on 0 value - set 1000 all increment - in memory in case of crash os - value will be 1000 after crash on val%1000 ==0 set 2000 and so on

    may be mute error with 0 - for simplify in this case: Set(file,Counter(file,key),value)

    may be call Counter silently in Set with params Counter(file,file) in case of key is nil so Set(file,nil,val) - will be autoincrement

    opened by recoilme 1
  • Gob

    Gob

    New experimental methods: SetGob(file string, key interface{}, val interface{}) GetGob(file string, key interface{}, val interface{})

    Example:

    func TestGob(t *testing.T) {
    	file := "test/gob.db"
    	DeleteFile(file)
    	defer CloseAll()
    	type Post struct {
    		Id       int
    		Content  string
    		Category string
    	}
    
    	for i := 0; i < 20; i++ {
    		post := &Post{Id: i, Content: "Content:" + strconv.Itoa(i)}
    		err := SetGob(file, post.Id, post)
    		ch(err, t)
    	}
    
    	for i := 0; i < 20; i++ {
    		var post = new(Post)
    		err := GetGob(file, i, post)
    		ch(err, t)
    		fmt.Println("i:", i, "Post:", post)
    	}
    }
    
    opened by recoilme 0
Releases(v2.0.1)
Owner
Vadim Kulibaba
Vadim Kulibaba
Eagle - Eagle is a fast and strongly encrypted key-value store written in pure Golang.

EagleDB EagleDB is a fast and simple key-value store written in Golang. It has been designed for handling an exaggerated read/write workload, which su

null 9 Sep 28, 2022
pure golang key database support key have value. 非常高效实用的键值数据库。

orderfile32 pure golang key database support key have value The orderfile32 is standard alone fast key value database. It have two version. one is thi

null 3 Apr 30, 2022
🔑A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout (LSM+WAL) similar to Riak.

bitcask A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout

James Mills 10 Sep 26, 2022
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Peter Bourgon 1.2k Nov 26, 2022
An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.

go-cache go-cache is an in-memory key:value store/cache similar to memcached that is suitable for applications running on a single machine. Its major

Patrick Mylund Nielsen 6.7k Nov 25, 2022
Embedded key-value store for read-heavy workloads written in Go

Pogreb Pogreb is an embedded key-value store for read-heavy workloads written in Go. Key characteristics 100% Go. Optimized for fast random lookups an

Artem Krylysov 962 Nov 15, 2022
Fast and simple key/value store written using Go's standard library

Table of Contents Description Usage Cookbook Disadvantages Motivation Benchmarks Test 1 Test 4 Description Package pudge is a fast and simple key/valu

Vadim Kulibaba 331 Nov 17, 2022
Key-value store for temporary items :memo:

Tempdb TempDB is Redis-backed temporary key-value store for Go. Useful for storing temporary data such as login codes, authentication tokens, and temp

Rafael Jesus 17 Sep 26, 2022
A distributed key-value store. On Disk. Able to grow or shrink without service interruption.

Vasto A distributed high-performance key-value store. On Disk. Eventual consistent. HA. Able to grow or shrink without service interruption. Vasto sca

Chris Lu 242 Nov 10, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 41.9k Nov 23, 2022
a key-value store with multiple backends including leveldb, badgerdb, postgresql

Overview goukv is an abstraction layer for golang based key-value stores, it is easy to add any backend provider. Available Providers badgerdb: Badger

Mohammed Al Ashaal 52 Jun 7, 2022
A minimalistic in-memory key value store.

A minimalistic in-memory key value store. Overview You can think of Kiwi as thread safe global variables. This kind of library comes in helpful when y

SDSLabs 159 Dec 6, 2021
Membin is an in-memory database that can be stored on disk. Data model smiliar to key-value but values store as JSON byte array.

Membin Docs | Contributing | License What is Membin? The Membin database system is in-memory database smiliar to key-value databases, target to effici

Membin 3 Jun 3, 2021
A simple Git Notes Key Value store

Gino Keva - Git Notes Key Values Gino Keva works as a simple Key Value store built on top of Git Notes, using an event sourcing architecture. Events a

Philips Software 24 Aug 14, 2022
A distributed key value store in under 1000 lines. Used in production at comma.ai

minikeyvalue Fed up with the complexity of distributed filesystems? minikeyvalue is a ~1000 line distributed key value store, with support for replica

George Hotz 2.5k Nov 27, 2022
Distributed cache and in-memory key/value data store.

Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Burak Sezer 2.3k Nov 15, 2022
Simple key value database that use json files to store the database

KValDB Simple key value database that use json files to store the database, the key and the respective value. This simple database have two gRPC metho

Francisco Santos 0 Nov 13, 2021
A rest-api that works with golang as an in-memory key value store

Rest API Service in GOLANG A rest-api that works with golang as an in-memory key value store Usage Run command below in terminal in project directory.

sercan aydın 0 Dec 6, 2021
A SQLite-based hierarchical key-value store written in Go

camellia ?? A lightweight hierarchical key-value store camellia is a Go library that implements a simple, hierarchical, persistent key-value store, ba

Valerio De Benedetto 32 Nov 9, 2022