Fast and simple key/value store written using Go's standard library

Related tags

Database pudge
Overview

Documentation Go Report Card Build Status Mentioned in Awesome Go

Table of Contents

Description

Package pudge is a fast and simple key/value store written using Go's standard library.

It presents the following:

  • Supporting very efficient lookup, insertions and deletions
  • Performance is comparable to hash tables
  • Ability to get the data in sorted order, which enables additional operations like range scan
  • Select with limit/offset/from key, with ordering or by prefix
  • Safe for use in goroutines
  • Space efficient
  • Very short and simple codebase
  • Well tested, used in production

pudge

Usage

package main

import (
	"log"

	"github.com/recoilme/pudge"
)

func main() {
	// Close all database on exit
	defer pudge.CloseAll()

	// Set (directories will be created)
	pudge.Set("../test/test", "Hello", "World")

	// Get (lazy open db if needed)
	output := ""
	pudge.Get("../test/test", "Hello", &output)
	log.Println("Output:", output)

	ExampleSelect()
}


//ExampleSelect
func ExampleSelect() {
	cfg := &pudge.Config{
		SyncInterval: 1} // every second fsync
	db, err := pudge.Open("../test/db", cfg)
	if err != nil {
		log.Panic(err)
	}
	defer db.DeleteFile()
	type Point struct {
		X int
		Y int
	}
	for i := 100; i >= 0; i-- {
		p := &Point{X: i, Y: i}
		db.Set(i, p)
	}
	var point Point
	db.Get(8, &point)
	log.Println(point)
	// Output: {8 8}
	// Select 2 keys, from 7 in ascending order
	keys, _ := db.Keys(7, 2, 0, true)
	for _, key := range keys {
		var p Point
		db.Get(key, &p)
		log.Println(p)
	}
	// Output: {8 8}
	// Output: {9 9}
}

Cookbook

  • Store data of any type. Pudge uses Gob encoder/decoder internally. No limits on keys/values size.
pudge.Set("strings", "Hello", "World")
pudge.Set("numbers", 1, 42)

type User struct {
	Id int
	Name string
}
u := &User{Id: 1, Name: "name"}
pudge.Set("users", u.Id, u)
  • Pudge is stateless and safe for use in goroutines. You don't need to create/open files before use. Just write data to pudge, don't worry about state. web server example

  • Pudge is parallel. Readers don't block readers, but a writer - does, but by the stateless nature of pudge it's safe to use multiples files for storages.

Illustration from slowpoke (based on pudge)

  • Default store system: like memcache + file storage. Pudge uses in-memory hashmap for keys, and writes values to files (no value data stored in memory). But you may use inmemory mode for values, with custom config:
cfg = pudge.DefaultConfig()
cfg.StoreMode = 2
db, err := pudge.Open(dbPrefix+"/"+group, cfg)
...
db.Counter(key, val)

In that case, all data is stored in memory and will be stored on disk only on Close.

Example server for highload, with http api

  • You may use pudge as an engine for creating databases.

Example database

  • Don't forget to close all opened databases on shutdown/kill.
 	// Wait for interrupt signal to gracefully shutdown the server 
	quit := make(chan os.Signal)
	signal.Notify(quit, os.Interrupt, os.Kill)
	<-quit
	log.Println("Shutdown Server ...")
	if err := pudge.CloseAll(); err != nil {
		log.Println("Pudge Shutdown err:", err)
	}

example recovery function for gin framework

  • Pudge has a primitive select/query engine.
// Select 2 keys, from 7 in ascending order
   keys, _ := db.Keys(7, 2, 0, true)
// select keys from db where key>7 order by keys asc limit 2 offset 0
  • Pudge will work well on SSD or spined disks. Pudge doesn't eat memory or storage or your sandwich. No hidden compaction/rebalancing/resizing and so on tasks. No LSM Tree. No MMap. It's a very simple database with less than 500 LOC. It's good for simple social network or highload system

Disadvantages

  • No transaction system. All operations are isolated, but you don't may batching them with automatic rollback.
  • Keys function (select/query engine) may be slow. Speed of query may vary from 10ms to 1sec per million keys. Pudge don't use BTree/Skiplist or Adaptive radix tree for store keys in ordered way on every insert. Ordering operation is "lazy" and run only if needed.
  • If you need storage or database for hundreds of millions keys - take a look at Sniper or b52. They are optimized for highload (pudge - not).
  • No fsync on every insert. Most of database fsync data by the timer too
  • Deleted data don't remove from physically (but upsert will try to reuse space). You may shrink database only with backup right now
pudge.BackupAll("backup")
  • Keys automatically convert to binary and ordered with binary comparator. It's simple for use, but ordering will not work correctly for negative numbers for example
  • Author of project don't work at Google or Facebook and his name not Howard Chu or Brad Fitzpatrick. But I'm open for issue or contributions.

Motivation

Some databases very well for writing. Some of the databases very well for reading. But pudge is well balanced for both types of operations. It has small cute api, and don't have hidden graveyards. It's just hashmap where values written in files. And you may use one database for in-memory/persistent storage in a stateless stressfree way

Benchmarks

All tests here

Some tests, MacBook Pro (Retina, 13-inch, Early 2015)

Test 1

Number of keys: 1000000 Minimum key size: 16, maximum key size: 64 Minimum value size: 128, maximum value size: 512 Concurrency: 2

pogreb goleveldb bolt badgerdb pudge slowpoke pudge(mem)
1M (Put+Get), seconds 187 38 126 34 23 23 2
1M Put, ops/sec 5336 34743 8054 33539 47298 46789 439581
1M Get, ops/sec 1782423 98406 499871 220597 499172 445783 1652069
FileSize,Mb 568 357 552 487 358 358 358

Test 4

Number of keys: 10000000 Key size: 8 Value size: 16 Concurrency: 100

goleveldb badgerdb pudge
10M (Put+Get), seconds 165 120 243
10M Put, ops/sec 122933 135709 43843
10M Get, ops/sec 118722 214981 666067
FileSize,Mb 312 1370 381
Comments
  • Make DefaultConfig public

    Make DefaultConfig public

    Привет! Предлагаю сделать DefaultConfig доступным для изменения, чтобы можно было установить параметры по-умолчанию для всех открываемых файлов.

    Текущая проблема: необходимо вручную для каждого файла устанавливать параметры вручную После изменения: необходимо единожды изменить DefaultConfig

    opened by nxshock 4
  • Is Pudge still active?

    Is Pudge still active?

    Hello,

    I am looking for a very fast replacement for boltdb and similar that can be put into Cayley Graph DB as a new backend.

    I came across your Pudge, Sniper, and B52 repos but am trying to get an idea as to which is better for my needs.

    One question about Pudge is that I think I read that it does not reduce the file size when KV deletions occur unless there is a backup done. If so, will this be fixed in the future to allow for file size reductions during use?

    As I am working towards a rebirth of a graph database, there could be the need to handle dynamically adding/editing/deleting hundreds of millions of graph nodes and thus KV entries so I am looking for the fastest possible KV library that might help to make this an efficient system.

    Any thoughts would be greatly appreciated.

    opened by lonnietc 3
  • StoreMode - memory only

    StoreMode - memory only

    I would like to use this nice library as an in-memory cache only. There are only two modes and both are creating the files. It is unnecessary in memory only mode.

    StoreMode int // 0 - file first, 2 - memory first(with persist on close)

    I suggest to introduce third mode: memory only

    In this mode files shoudn't be opened at all, so path will be ignored and no need to call db.CloseAll

    opened by begetan 3
  • struct with field *int having zero do not return properly

    struct with field *int having zero do not return properly

    Hi! Thanks for nice lib. I found An issue. Code tells everything

    	type  test struct {
    		A *int
    	}
    
    	i := 0
    	p := test{A: &i}
    
    	pudge.Set("a", "a", p)
    
    	re := test{}
    	pudge.Get("a","a", &re)
    
    	log.Print(re.A) // prints <nil>, should print pointer (to zero)
    
    
    opened by Bysmyyr 2
  • log entry in api.go causes panik if db has no entries

    log entry in api.go causes panik if db has no entries

    Hello!

    in file api.go and method Keys (line 326)

    the log produces a panik if db is empty.

    log.Println(bytes.Equal(k[len(k)-1:], []byte("*")))

    please remove that log entry and import of log package

    regards Andreas

    opened by maj-o 1
  • Не сохраняется часть данных со StoreMode = 2

    Не сохраняется часть данных со StoreMode = 2

    Возьмём, к примеру, следующую программу, сохраняющая данные в несколько файлов:

    Код с StoreMode = 0
    package main
    
    import (
    	"fmt"
    	"log"
    
    	"github.com/recoilme/pudge"
    )
    
    func main() {
    	pudge.DefaultConfig.StoreMode = 0
    
    	for i := 0; i < 100000; i++ {
    		fileName := fmt.Sprintf("data/%d", i%5)
    		err := pudge.Set(fileName, i, i)
    		if err != nil {
    			log.Fatalln(err)
    		}
    	}
    
    	err := pudge.CloseAll()
    	if err != nil {
    		log.Fatalln(err)
    	}
    }
    

    Получаем на выходе 5 файлов с данными по 131 КБ и 5 файлов по 469 КБ с индексами.

    Теперь поменяем StoreMode = 2, и попробуем выполнить тот же код:

    Код с StoreMode = 2
    package main
    
    import (
    	"fmt"
    	"log"
    
    	"github.com/recoilme/pudge"
    )
    
    func main() {
    	pudge.DefaultConfig.StoreMode = 2
    
    	for i := 0; i < 100000; i++ {
    		fileName := fmt.Sprintf("data/%d", i%5)
    		err := pudge.Set(fileName, i, i)
    		if err != nil {
    			log.Fatalln(err)
    		}
    	}
    
    	err := pudge.CloseAll()
    	if err != nil {
    		log.Fatalln(err)
    	}
    }
    

    В результате получаю заполенным только один файл и данными и файл с индексом (который цифрой 0), а остальные 4 файла пустые - данные в них не сохраняются.

    opened by nxshock 1
  • deletion related perf and space improvements

    deletion related perf and space improvements

    This PR dramatically improves the performance of multiple deletions in a database with a large number of keys. The original previous code would resort the database before every deletion, the changes are simply to sort once and then delete multiple keys. An API call is also added to 'compact' both the key and value files to remove deleted keys and entries, and also entries that have 'outgrown' their original size in the value file (an existing value will be overwritten if the new value is the same size or smaller, or appended otherwise).

    opened by cosnicolaou 0
  • Panic for KeyToBinary for empty database

    Panic for KeyToBinary for empty database

    Test program:

    package main
    
    import (
    	"fmt"
    
    	"github.com/recoilme/pudge"
    )
    
    func main() {
    	db, err := pudge.Open("test.me", pudge.DefaultConfig)
    	if err != nil {
    		fmt.Println("error:", err)
    		return
    	}
    
    	bytes, err := pudge.KeyToBinary("test-prefix-")
    	if err != nil {
    		fmt.Println("error:", err)
    		return
    	}
    
    	keys, err := db.KeysByPrefix(bytes, 1, 0, false)
    	fmt.Printf("%#v %#v", keys, err)
    }
    

    Output:

    panic: runtime error: index out of range [-1]
    ...
    

    Crash occurs during checking right part of OR condition: https://github.com/recoilme/pudge/blob/master/api.go#L282

    bug 
    opened by ont 0
Releases(v1.0.2)
Owner
Vadim Kulibaba
Vadim Kulibaba
Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on commands and key-regexes written by GO

Welcome to NIPO Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on command

Morteza Bashsiz 16 Jun 13, 2022
Eagle - Eagle is a fast and strongly encrypted key-value store written in pure Golang.

EagleDB EagleDB is a fast and simple key-value store written in Golang. It has been designed for handling an exaggerated read/write workload, which su

null 9 Sep 28, 2022
🔑A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout (LSM+WAL) similar to Riak.

bitcask A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout

James Mills 10 Sep 26, 2022
Embedded key-value store for read-heavy workloads written in Go

Pogreb Pogreb is an embedded key-value store for read-heavy workloads written in Go. Key characteristics 100% Go. Optimized for fast random lookups an

Artem Krylysov 964 Dec 1, 2022
A SQLite-based hierarchical key-value store written in Go

camellia ?? A lightweight hierarchical key-value store camellia is a Go library that implements a simple, hierarchical, persistent key-value store, ba

Valerio De Benedetto 32 Nov 9, 2022
An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.

go-cache go-cache is an in-memory key:value store/cache similar to memcached that is suitable for applications running on a single machine. Its major

Patrick Mylund Nielsen 6.7k Nov 25, 2022
A simple Git Notes Key Value store

Gino Keva - Git Notes Key Values Gino Keva works as a simple Key Value store built on top of Git Notes, using an event sourcing architecture. Events a

Philips Software 24 Aug 14, 2022
Simple key value database that use json files to store the database

KValDB Simple key value database that use json files to store the database, the key and the respective value. This simple database have two gRPC metho

Francisco Santos 0 Nov 13, 2021
Distributed cache and in-memory key/value data store.

Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Burak Sezer 2.6k Nov 28, 2022
BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go

BadgerDB BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the underlying database for Dgraph, a fast,

Blizard 1 Dec 10, 2021
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Peter Bourgon 1.2k Nov 26, 2022
Low-level key/value store in pure Go.

Description Package slowpoke is a simple key/value store written using Go's standard library only. Keys are stored in memory (with persistence), value

Vadim Kulibaba 100 Oct 14, 2022
Key-value store for temporary items :memo:

Tempdb TempDB is Redis-backed temporary key-value store for Go. Useful for storing temporary data such as login codes, authentication tokens, and temp

Rafael Jesus 17 Sep 26, 2022
A distributed key-value store. On Disk. Able to grow or shrink without service interruption.

Vasto A distributed high-performance key-value store. On Disk. Eventual consistent. HA. Able to grow or shrink without service interruption. Vasto sca

Chris Lu 242 Nov 10, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 42k Dec 4, 2022
a key-value store with multiple backends including leveldb, badgerdb, postgresql

Overview goukv is an abstraction layer for golang based key-value stores, it is easy to add any backend provider. Available Providers badgerdb: Badger

Mohammed Al Ashaal 52 Jun 7, 2022
A minimalistic in-memory key value store.

A minimalistic in-memory key value store. Overview You can think of Kiwi as thread safe global variables. This kind of library comes in helpful when y

SDSLabs 159 Dec 6, 2021
Membin is an in-memory database that can be stored on disk. Data model smiliar to key-value but values store as JSON byte array.

Membin Docs | Contributing | License What is Membin? The Membin database system is in-memory database smiliar to key-value databases, target to effici

Membin 3 Jun 3, 2021
A distributed key value store in under 1000 lines. Used in production at comma.ai

minikeyvalue Fed up with the complexity of distributed filesystems? minikeyvalue is a ~1000 line distributed key value store, with support for replica

George Hotz 2.5k Nov 27, 2022