Key/Value database benchmark

Related tags

Benchmarks kvbench
Overview

USAGE:

kvbench OPTIONS

OVERVIEW

kvbench is a simple benchmarking tool to evaluate the read performance of a key/value store while writes are being applied in parallel.

DETAILS:

Running a benchmark consists of two steps:

(1) Generate a sample data file using the -o option. Use the -n, -b[01], -k[01], and -v[01] options to control the size of the sample. Use the -r option to change the pseudo-random data. Given identical inputs, an identical data file should be generated.

(2) Consume a sample data file and execute a benchmark using the -i option. Use the -d[01] options to control the inter-arrival rate of new row sets to be written to the collection. The -p option controls how often the benchmark will attempt to iterate over the keys.

You may specify both -o and -i options to generate the data and then immediately run the benchmark.

OUTPUT OPTIONS

  • -r n - pseudo-random seed
  • -n n - total number of blocks to generate
  • -b0 min - minimum number of records per block
  • -b1 max - maximum number of records per block
  • -k0 min - minimum length of key to generate
  • -k1 max - maximum length of key to generate
  • -v0 min - minimum length of value to generate
  • -v1 max - maximum length of value to generate -o dat - output path for data file

INPUT OPTIONS

  • -r n - pseudo-random seed
  • -d0 dur - minimum inter-arrival rate
  • -d1 dur - maximum inter-arrival rate (not guaranteed)
  • -p dur - poll db at this interval and print statistics
  • -i dat - input path for data file
  • -b bench - name of the benchmark to run (bolt, kv, kv-mu, leveldb, noop)
  • -f path - path to the database

EXAMPLE

$ ./kvbench -n 100 -b0 0 -b1 4000 -k0 34 -k1 34 -v0 70 -v1 78 -o sample.dat
2014/04/16 19:57:46 writing data to sample.dat

$ ./kvbench -i sample.dat -b leveldb -f test/leveldb.db -d0 50ms -d1 175ms -p 10s
2014/04/16 19:57:51 reading data from sample.dat
2014/04/16 19:58:02 leveldb: 178046 ops in 296 ms: 601 ops/ms
2014/04/16 19:58:12 leveldb: 202517 ops in 315 ms: 642 ops/ms
2014/04/16 19:58:12 100 row sets arrived at an average inter-arrival rate of 116.046572ms
2014/04/16 19:58:12 leveldb: 202517 ops in 308 ms: 657 ops/ms

$ ./kvbench -i sample.dat -b bolt -f test/bolt.db -d0 50ms -d1 175ms -p 10s
2014/04/16 19:58:31 reading data from sample.dat
2014/04/16 19:58:41 bolt: 151830 ops in 141 ms: 1076 ops/ms
2014/04/16 19:58:51 bolt: 202517 ops in 210 ms: 964 ops/ms
2014/04/16 19:58:51 100 row sets arrived at an average inter-arrival rate of 134.606206ms
2014/04/16 19:58:51 bolt: 202517 ops in 168 ms: 1205 ops/ms

$ ./kvbench -i sample.dat -b kv -f test/kv.db -d0 50ms -d1 175ms -p 10s
2014/04/16 19:58:53 reading data from sample.dat
^C

$ date
Wed Apr 16 19:59:54 PDT 2014

$ ./kvbench -i sample.dat -b kv-mu -f test/kv-mu.db -d0 50ms -d1 175ms -p 10s
2014/04/16 20:00:01 reading data from sample.dat
2014/04/16 20:00:12 kv-mu: 18575 ops in 378 ms: 49 ops/ms
2014/04/16 20:00:24 kv-mu: 34127 ops in 588 ms: 58 ops/ms
2014/04/16 20:00:36 kv-mu: 50581 ops in 554 ms: 91 ops/ms
2014/04/16 20:00:49 kv-mu: 65373 ops in 1267 ms: 51 ops/ms
2014/04/16 20:01:00 kv-mu: 78226 ops in 1108 ms: 70 ops/ms
2014/04/16 20:01:12 kv-mu: 90815 ops in 1395 ms: 65 ops/ms
2014/04/16 20:01:25 kv-mu: 102478 ops in 2174 ms: 47 ops/ms
2014/04/16 20:01:38 kv-mu: 112228 ops in 1869 ms: 60 ops/ms
2014/04/16 20:01:51 kv-mu: 122214 ops in 2357 ms: 51 ops/ms
2014/04/16 20:02:06 kv-mu: 133389 ops in 1981 ms: 67 ops/ms
2014/04/16 20:02:18 kv-mu: 142738 ops in 1950 ms: 73 ops/ms
2014/04/16 20:02:30 kv-mu: 150676 ops in 1979 ms: 76 ops/ms
2014/04/16 20:02:46 kv-mu: 161695 ops in 2518 ms: 64 ops/ms
2014/04/16 20:03:00 kv-mu: 170495 ops in 2986 ms: 57 ops/ms
2014/04/16 20:03:16 kv-mu: 180825 ops in 3610 ms: 50 ops/ms
2014/04/16 20:03:33 kv-mu: 191325 ops in 3493 ms: 54 ops/ms
2014/04/16 20:03:48 kv-mu: 198455 ops in 4151 ms: 47 ops/ms
2014/04/16 20:04:01 kv-mu: 202517 ops in 3252 ms: 62 ops/ms
2014/04/16 20:04:01 100 row sets arrived at an average inter-arrival rate of 2.300565857s
2014/04/16 20:04:05 kv-mu: 202517 ops in 3538 ms: 57 ops/ms

$ ./kvbench -i sample.dat -b kv -f test/kv-2.db -d0 2s -d1 5s -p 10s
2014/04/16 20:04:40 reading data from sample.dat
2014/04/16 20:04:50 kv: 2936 ops in 39 ms: 75 ops/ms
2014/04/16 20:05:00 kv: 7593 ops in 110 ms: 69 ops/ms
2014/04/16 20:05:11 kv: 14963 ops in 287 ms: 52 ops/ms
2014/04/16 20:05:21 kv: 20289 ops in 274 ms: 74 ops/ms
2014/04/16 20:05:33 kv: 27723 ops in 2334 ms: 11 ops/ms
2014/04/16 20:05:44 kv: 34127 ops in 469 ms: 72 ops/ms
2014/04/16 20:05:54 kv: 40052 ops in 524 ms: 76 ops/ms
2014/04/16 20:06:05 kv: 44990 ops in 672 ms: 66 ops/ms
2014/04/16 20:06:16 kv: 51604 ops in 733 ms: 70 ops/ms
2014/04/16 20:06:27 kv: 58077 ops in 976 ms: 59 ops/ms
2014/04/16 20:06:39 kv: 69760 ops in 2488 ms: 28 ops/ms
2014/04/16 20:06:50 kv: 77416 ops in 1133 ms: 68 ops/ms
2014/04/16 20:07:02 kv: 82087 ops in 1714 ms: 47 ops/ms
2014/04/16 20:07:13 kv: 88134 ops in 1253 ms: 70 ops/ms
2014/04/16 20:07:27 kv: 97417 ops in 3505 ms: 27 ops/ms
2014/04/16 20:07:44 kv: 107525 ops in 7438 ms: 14 ops/ms
2014/04/16 20:08:00 kv: 118256 ops in 6163 ms: 19 ops/ms
2014/04/16 20:08:16 kv: 130346 ops in 5284 ms: 24 ops/ms
2014/04/16 20:08:36 kv: 143446 ops in 10035 ms: 14 ops/ms
2014/04/16 20:08:48 kv: 150676 ops in 2252 ms: 66 ops/ms
2014/04/16 20:09:04 kv: 158712 ops in 6425 ms: 24 ops/ms
2014/04/16 20:09:41 kv: 184613 ops in 26382 ms: 6 ops/ms
2014/04/16 20:10:00 kv: 197430 ops in 9778 ms: 20 ops/ms
2014/04/16 20:10:15 kv: 202517 ops in 4464 ms: 45 ops/ms
2014/04/16 20:10:15 100 row sets arrived at an average inter-arrival rate of 3.28728241s
2014/04/16 20:10:18 kv: 202517 ops in 3482 ms: 58 ops/ms
Issues
  • Build error

    Build error

    Finally tried to start to do something using your project (thanks again for making it!):

    (10:01) [email protected]:~$ go get github.com/jimrobinson/kvbench
    # github.com/jimrobinson/kvbench
    src/github.com/jimrobinson/kvbench/collection_bolt.go:26: cannot use *bolt.Bucket as type error in return argument:
        *bolt.Bucket does not implement error (missing Error method)
    src/github.com/jimrobinson/kvbench/collection_bolt.go:26: too many arguments to return
    (11:06) [email protected]:~$ go get -u github.com/jimrobinson/kvbench
    # github.com/jimrobinson/kvbench
    src/github.com/jimrobinson/kvbench/collection_bolt.go:26: cannot use *bolt.Bucket as type error in return argument:
        *bolt.Bucket does not implement error (missing Error method)
    src/github.com/jimrobinson/kvbench/collection_bolt.go:26: too many arguments to return
    (11:09) [email protected]:~$
    

    Commit 81fbd6703b25cd26dff5d099cd7d9a49b04e2e3e

    opened by cznic 1
  • go fmt

    go fmt

    There are some non gofmt'ed files in the repo. Creating a separate issue for easy possible revert - if needed.

    $ gofmt -l .
    benchmark.go
    collection_test.go
    data.go
    main.go
    $ 
    
    enhancement 
    opened by cznic 0
Owner
James A. Robinson
James A. Robinson
Go HTTP request router and web framework benchmark

Go HTTP Router Benchmark This benchmark suite aims to compare the performance of HTTP request routers for Go by implementing the routing structure of

Julien Schmidt 1.6k May 17, 2022
:zap: Go web framework benchmark

go-web-framework-benchmark This benchmark suite aims to compare the performance of Go web frameworks. It is inspired by Go HTTP Router Benchmark but t

smallnest 1.7k May 13, 2022
A rapid http(s) benchmark tool written in Go

gonetx/httpit httpit is a rapid http(s) benchmark tool which on top of fasthttp. Also thanks to cobra and bubbletea. Installation Get binaries from re

null 134 May 11, 2022
Cache benchmark for Golang

Cache comparison benchmark for Go This benchmark compares cache algorithms using scrambled zipfian distribution (a few occur very often while many oth

Vladimir Mihailenco 39 May 5, 2022
Go http server benchmark

go-http-server-benchmark The more connections, nbio cost the less memory, and performance the better than other frameworks. We can serve for 1000k or

lesismal 1 Oct 27, 2021
Golang json encoders\decoders benchmark

Json encoder and decoder benchmark Test Encode for: Build-in encoder https://github.com/mailru/easyjson https://github.com/json-iterator/go Test Decod

Vladislav S 0 Dec 3, 2021
Go Script Lang Benchmark

Go Script Lang Benchmark git clone github.com/akkuman/go_script_lang_benchmark c

null 2 Feb 10, 2022
Redis-benchmark - Simple get, mget and pipelined get benchmark.

redis-benchmark Simple get, mget and pipelined get benchmark. Usage git clone https://github.com/Ali-A-A/redis-benchmark.git cd ./redis-benchmark dock

Ali Abbasi Alaei 0 Dec 31, 2021
Bxd redis benchmark - Redis benchmark tool for golang

使用 redis benchmark 工具, 测试 10 20 50 100 200 1k 5k 字节 value 大小,redis get set 性能。 r

bingxindan 0 Jan 22, 2022
Go-driver-benchmark - Driver benchmark with golang

We use ClickkHouse for time-series databases, and the driver's performance is ve

null 3 Jan 27, 2022
Benchmark - Benchmark of logr implementations

Benchmark of logr implementations Implementations a function (can bridge to non-

null 2 Mar 23, 2022
Golang-key-value-store - Key Value Store API Service with Go DDD Architecture

This document specifies the tools used in the Key-Value store and reorganizes how to use them. In this created service, In-Memory Key-Value Service was created and how to use the API is specified in the HTML file in the folder named "doc"

Kemal Emre Ballı 3 Mar 16, 2022
Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on commands and key-regexes written by GO

Welcome to NIPO Nipo is a powerful, fast, multi-thread, clustered and in-memory key-value database, with ability to configure token and acl on command

Morteza Bashsiz 15 Apr 23, 2021
pure golang key database support key have value. 非常高效实用的键值数据库。

orderfile32 pure golang key database support key have value The orderfile32 is standard alone fast key value database. It have two version. one is thi

null 3 Apr 30, 2022
Simple key value database that use json files to store the database

KValDB Simple key value database that use json files to store the database, the key and the respective value. This simple database have two gRPC metho

Francisco Santos 0 Nov 13, 2021
A key-value db api with multiple storage engines and key generation

Jet is a deadly-simple key-value api. The main goals of this project are : Making a simple KV tool for our other projects. Learn tests writing and git

null 12 Apr 5, 2022
An embedded key/value database for Go.

bbolt bbolt is a fork of Ben Johnson's Bolt key/value store. The purpose of this fork is to provide the Go community with an active maintenance and de

etcd-io 5.5k May 10, 2022
BuntDB is an embeddable, in-memory key/value database for Go with custom indexing and geospatial support

BuntDB is a low-level, in-memory, key/value store in pure Go. It persists to disk, is ACID compliant, and uses locking for multiple readers and a sing

Josh Baker 3.7k May 12, 2022
ACID key-value database.

Coffer Simply ACID* key-value database. At the medium or even low latency it tries to provide greater throughput without losing the ACID properties of

Eduard 30 May 1, 2022
LevelDB key/value database in Go.

This is an implementation of the LevelDB key/value database in the Go programming language. Installation go get github.com/syndtr/goleveldb/leveldb R

Suryandaru Triandana 5.1k May 9, 2022
An embedded key/value database for Go.

Bolt Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable datab

BoltDB 12.8k May 15, 2022
LevelDB key/value database in Go.

This is an implementation of the LevelDB key/value database in the Go programming language. Installation go get github.com/syndtr/goleveldb/leveldb R

Suryandaru Triandana 5.1k May 18, 2022
RocksDB/LevelDB inspired key-value database in Go

Pebble Nightly benchmarks Pebble is a LevelDB/RocksDB inspired key-value store focused on performance and internal usage by CockroachDB. Pebble inheri

CockroachDB 2.8k May 10, 2022
Graviton Database: ZFS for key-value stores.

Graviton Database: ZFS for key-value stores. Graviton Database is simple, fast, versioned, authenticated, embeddable key-value store database in pure

null 406 Apr 21, 2022
An embedded key/value database for Go.

bbolt bbolt is a fork of Ben Johnson's Bolt key/value store. The purpose of this fork is to provide the Go community with an active maintenance and de

etcd-io 5.5k May 15, 2022
Membin is an in-memory database that can be stored on disk. Data model smiliar to key-value but values store as JSON byte array.

Membin Docs | Contributing | License What is Membin? The Membin database system is in-memory database smiliar to key-value databases, target to effici

Membin 3 Jun 3, 2021
Key-value database stored in memory with option of persistence

Easy and intuitive command line tool allows you to spin up a database avaliable from web or locally in a few seconds. Server can be run over a custom TCP protocol or over HTTP.

Mario Petričko 7 Jan 8, 2022
Create key value sqlite3 database from tabular data, fast.

Turn tabular data into a lookup table using sqlite3. This is a working PROTOTYPE with limitations, e.g. no customizations, the table definition is fixed, etc.

Martin Czygan 5 Apr 2, 2022
An embedded, hardened key/value database for Go.

Bolt Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable datab

null 0 Nov 4, 2021