LevelDB key/value database in Go.

Overview

This is an implementation of the LevelDB key/value database in the Go programming language.

Build Status

Installation

go get github.com/syndtr/goleveldb/leveldb

Requirements

  • Need at least go1.5 or newer.

Usage

Create or open a database:

// The returned DB instance is safe for concurrent use. Which mean that all
// DB's methods may be called concurrently from multiple goroutine.
db, err := leveldb.OpenFile("path/to/db", nil)
...
defer db.Close()
...

Read or modify the database content:

// Remember that the contents of the returned slice should not be modified.
data, err := db.Get([]byte("key"), nil)
...
err = db.Put([]byte("key"), []byte("value"), nil)
...
err = db.Delete([]byte("key"), nil)
...

Iterate over database content:

iter := db.NewIterator(nil, nil)
for iter.Next() {
	// Remember that the contents of the returned slice should not be modified, and
	// only valid until the next call to Next.
	key := iter.Key()
	value := iter.Value()
	...
}
iter.Release()
err = iter.Error()
...

Seek-then-Iterate:

iter := db.NewIterator(nil, nil)
for ok := iter.Seek(key); ok; ok = iter.Next() {
	// Use key/value.
	...
}
iter.Release()
err = iter.Error()
...

Iterate over subset of database content:

iter := db.NewIterator(&util.Range{Start: []byte("foo"), Limit: []byte("xoo")}, nil)
for iter.Next() {
	// Use key/value.
	...
}
iter.Release()
err = iter.Error()
...

Iterate over subset of database content with a particular prefix:

iter := db.NewIterator(util.BytesPrefix([]byte("foo-")), nil)
for iter.Next() {
	// Use key/value.
	...
}
iter.Release()
err = iter.Error()
...

Batch writes:

batch := new(leveldb.Batch)
batch.Put([]byte("foo"), []byte("value"))
batch.Put([]byte("bar"), []byte("another value"))
batch.Delete([]byte("baz"))
err = db.Write(batch, nil)
...

Use bloom filter:

o := &opt.Options{
	Filter: filter.NewBloomFilter(10),
}
db, err := leveldb.OpenFile("path/to/db", o)
...
defer db.Close()
...

Documentation

You can read package documentation here.

Issues
  • Require explicit general database closing.

    Require explicit general database closing.

    To be idiomatic and play nicely as a library, control of the database should exist solely in the hands of the library user and not in that of the memory manager.

    Major Caveat: A number of internal types around iterators and versions retain their finalizers for now. I need to take some time to diagram their lifecycle to propose a solution for this. This domain is where, I believe, the nondeterminism I found was, but I want to start fixing the easy part first.

    @syndtr, if you have some recommendations for how to approach this, I would love to talk with you. If you look at other LevelDB implementations, they explicitly require the user to call Iterator.Close to free associated resources, including the an official but incomplete implementation from Google in Go at https://code.google.com/p/leveldb-go/source/browse/leveldb/db/db.go#72.

    opened by matttproud 23
  • Slice bounds out of range error during compaction

    Slice bounds out of range error during compaction

    Here's the stackdump:

    panic: runtime error: slice bounds out of range [recovered] panic: runtime error: slice bounds out of range

    goroutine 50 [running]: github.com/syndtr/goleveldb/leveldb.func·015() /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:392 +0x172 github.com/syndtr/goleveldb/leveldb.iKey.ukey(0xc2006180df, 0x0, 0xa7, 0x0, 0xa7, ...) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/key.go:73 +0x75 github.com/syndtr/goleveldb/leveldb.(_iComparer).Compare(0xc20016a660, 0xc2006180df, 0x0, 0xa7, 0xc201207720, ...) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/comparer.go:21 +0x85 github.com/syndtr/goleveldb/leveldb/iterator.(_MergedIterator).smallest(0xc200afca50) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:178 +0x158 github.com/syndtr/goleveldb/leveldb/iterator.(_MergedIterator).Next(0xc200afca50, 0xc20061802d) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:113 +0x185 github.com/syndtr/goleveldb/leveldb.func·013(0x0, 0x0) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:247 +0x1cc github.com/syndtr/goleveldb/leveldb.(_DB).transact(0xc20012d180, 0x6c7a50) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:136 +0xaa github.com/syndtr/goleveldb/leveldb.(_DB).doCompaction(0xc20012d180, 0xc200172540, 0x6c7f00) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:365 +0x7e0 github.com/syndtr/goleveldb/leveldb.(_DB).compaction(0xc20012d180) /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:463 +0x1cb created by github.com/syndtr/goleveldb/leveldb.open /Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db.go:70 +0x1ec

    bug 
    opened by x6j8x 21
  • Possible memory leak

    Possible memory leak

    I'm using goleveldb to keep an index from certain kinds metadata to entries in my logfiles. I have four leveldbs for each day which are rotated at midnight. After the process is running several days the memory consumption keeps growing and growing. I'm using a write buffer of 16 Mbytes.

    Here's the output from lsof:

    process 30008 user 68uW REG 253,2 0 15310869 ./20140527-indexdb.ldb/servicenames/LOCK process 30008 user 70w REG 253,2 165 15310870 ./20140527-indexdb.ldb/servicenames/LOG process 30008 user 74w REG 253,2 54 15310871 ./20140527-indexdb.ldb/servicenames/MANIFEST-000000 process 30008 user 82w REG 253,2 8740127 15310873 ./20140527-indexdb.ldb/servicenames/000001.log process 30008 user 129uW REG 253,2 0 15310875 ./20140527-indexdb.ldb/servicemetadata/LOCK process 30008 user 130w REG 253,2 165 15310876 ./20140527-indexdb.ldb/servicemetadata/LOG process 30008 user 131w REG 253,2 54 15310877 ./20140527-indexdb.ldb/servicemetadata/MANIFEST-000000 process 30008 user 134w REG 253,2 361647 15310879 ./20140527-indexdb.ldb/servicemetadata/000001.log process 30008 user 146uW REG 253,2 0 15310851 ./20140527-indexdb.ldb/base/LOCK process 30008 user 153w REG 253,2 24820721 15310912 ./20140527-indexdb.ldb/transactions/000002.log process 30008 user 155w REG 253,2 165 15310852 ./20140527-indexdb.ldb/base/LOG process 30008 user 159w REG 253,2 54 15310853 ./20140527-indexdb.ldb/base/MANIFEST-000000 process 30008 user 161w REG 253,2 59070 15310855 ./20140527-indexdb.ldb/base/000001.log process 30008 user 164uW REG 253,2 0 15310857 ./20140527-indexdb.ldb/transactions/LOCK process 30008 user 171w REG 253,2 365 15310858 ./20140527-indexdb.ldb/transactions/LOG process 30008 user 175w REG 253,2 128 15310859 ./20140527-indexdb.ldb/transactions/MANIFEST-000000 process 30008 user 192uW REG 253,2 0 15310863 ./20140527-indexdb.ldb/message/LOCK process 30008 user 193w REG 253,2 166 15310864 ./20140527-indexdb.ldb/message/LOG process 30008 user 194w REG 253,2 54 15310865 ./20140527-indexdb.ldb/message/MANIFEST-000000 process 30008 user 196w REG 253,2 26185266 15310867 ./20140527-indexdb.ldb/message/000001.log

    Here's the output from pprof:

    screenshot-1

    bug 
    opened by x6j8x 20
  • Effort to reduce heap allocations

    Effort to reduce heap allocations

    Hi!

    I use goleveldb in syncthing (https://github.com/syncthing/syncthing) and have been doing a fair bit of profiling and hacking lately to reduce syncthing's memory footprint. I'm now at the point where goleveldb represents ~70-80 MB of a 100 MB footprint in my benchmarks. To a certain extent this is fair, with caches etc. But a large part of it is temporary heap allocations, in fact pretty much all of them made by snappy.Decode() from readRawBlock(). I made a naive attempt to reuse buffers with a sync.Pool in https://github.com/calmh/goleveldb/tree/reduce-heap, and it makes a dent but it's not huge. Probably the reason it's not very effective is that the pool isn't filled very often, and I'm not sure about the code paths that result in the data buffers becoming unused to hook into.

    This isn't a bug report as such, more a humble request to give it some work at some point, or pass me some hints and I'm happy to root around in it. :)

    enhancement 
    opened by calmh 18
  • Compaction time can be quite big

    Compaction time can be quite big

    Hey! I am investigation the performance decline in our software https://github.com/tendermint/tendermint/issues/1835. We're using goleveldb as a primary DB for storing blocks plus some other data.

    Looks like saving a batch of data (~1MB) can result in ~ 15 sec. due to compaction process or something:

    https://github.com/tendermint/tendermint/issues/1835#issuecomment-402054099 (the time seems to be increasing with every spike)

    The write speed is approximately 1MB per second.

    Is this expected and we need to fine tune compaction options or this looks like a bug to you?

    Thank you!

    opened by melekes 15
  • Possible corruption on the block data

    Possible corruption on the block data

    This could be caused by bug in the table writer or block buffer.

    Corruption on the block buffer could be caused by same buffer acquired more than once from buffer pool by two different block instance (or probably added to buffer pool more than once?).

    https://gist.github.com/syndtr/a97bef48b12df8181d6b

    Recovering the DB doesn't indicate any corruption but the corrupted table may already been deleted.

    Could be related to syncthing/syncthing#987.

    hardwareissue 
    opened by syndtr 13
  • db corruption on 'out of disk space'

    db corruption on 'out of disk space'

    Using btcd, which currently uses goleveldb on commit 92de35cffa094e330ee854d44ccbcb954e344d08, if disk space runs out (write /tmp/mainnet/blocks_leveldb/000054.log: no space left on device), btcd can be restarted without problem.

    After updating goleveldb to commit 62cc4d4e8b1ba35c2d1597e54892c022c376995f, goleveldb corrupts the database on the same error. When btcd starts back up, it says leveldb: bad record in batch

    bug 
    opened by dajohi 13
  • Significant memory usage when trying to insert data

    Significant memory usage when trying to insert data

    I am trying to profile the memory usage of my go program, and found that the majority of memory seems to be consumed within leveldb functions (github.com/syndtr/goleveldb/leveldb/memdb.(*DB).Put).

    Below is the top profile output:

    (pprof) top
    6815712.39kB of 6881450.49kB total (99.04%)
    Dropped 372 nodes (cum <= 34407.25kB)
    Showing top 10 nodes out of 28 (cum >= 242528.01kB)
          flat  flat%   sum%        cum   cum%
     6471904kB 94.05% 94.05%  6471904kB 94.05%  github.com/syndtr/goleveldb/leveldb/memdb.(*DB).Put
    242528.01kB  3.52% 97.57% 242528.01kB  3.52%  bytes.makeSlice
       55040kB   0.8% 98.37%    55040kB   0.8%  lib.FP.GenerateHashMap
    46004.76kB  0.67% 99.04% 54040.69kB  0.79%  main.main
      149.16kB 0.0022% 99.04% 6483852.34kB 94.22%  github.com/syndtr/goleveldb/leveldb.(*DB).Write
       68.67kB 0.001% 99.04% 319400.85kB  4.64%  main.Worker
       13.77kB 0.0002% 99.04% 6507694.47kB 94.57%  main.Worker1
        4.03kB 5.9e-05% 99.04% 6495736.37kB 94.39%  github.com/syndtr/goleveldb/leveldb.(*DB).Put
             0     0% 99.04%   196616kB  2.86%  bytes.(*Buffer).WriteString
             0     0% 99.04% 242528.01kB  3.52%  bytes.(*Buffer).grow
    
    bug 
    opened by xia0pin9 12
  • disable auto compaction

    disable auto compaction

    Is there a way to disable the auto db compaction goleveldb does? My db keeps being corrupted, and if it does panic, it is always in db_compaction.go

    I want to test filling my DB without goleveldb running compaction at all to see if it still corrupts.

    opened by dajohi 12
  • Compression change behind the scenes

    Compression change behind the scenes

    btcd (github.com/conformal/btcd) uses goleveldb to store data.

    The database is opened with:

            myCache := cache.NewEmptyCache()
            opts := &opt.Options{Flag: flag,
                    BlockCache:      myCache,
                    MaxOpenFiles:    256,
                    CompressionType: opt.NoCompression,
            }
    

    After a while, the database throws an error about a missing data.

    I added two fmt.Printfs to reader.go in readRawBlock()'s select case to print the type of Compression being used. I also added spew.Dump(bh) We have a tool to dump data.

    (table.blockHandle) {
     offset: (uint64) 1937402,
     length: (uint64) 4827
    }
    (uint8) 0
    No Compression
    

    On a different block:

    (table.blockHandle) {
     offset: (uint64) 0,
     length: (uint64) 22302
    }
    (uint8) 1
    Snappy Compression
    1381438195854721313 [Warn] unable to locate block sha 00000000000033640c6b34c40b8b95ba4fae759ec7970e546660a3abd9f9314c err snappy: unsupported COPY_4 tag
    1381438195854866024 [Info] Invalid block 00000000000033640c6b34c40b8b95ba4fae759ec7970e546660a3abd9f9314c
    

    And another:
    (table.blockHandle) {
     offset: (uint64) 1237318,
     length: (uint64) 3572
    }
    (uint8) 1
    Snappy Compression
    

    Yet another:

    (table.blockHandle) {
     offset: (uint64) 1119730,
     length: (uint64) 4331
    }
    (uint8) 0
    No Compression
    

    Any idea what is causing the compression to be changed? Multiple people are seeing this on different machines. I am the only one getting the COPY_4 tag issue.

    opened by dajohi 11
  • go test fail on Windows/386

    go test fail on Windows/386

    Hello,

    I saw you fixed #14 so I wanted to test your package and here what I got:

    go version
    go version devel +d29da2ced72b Mon May 13 10:28:12 2013 +0400 windows/386
    

    Windows 7 x32

    Error:

    D:\Golibs\os\src\github.com\syndtr\goleveldb\leveldb>go test
    fatal error: runtime: cannot map pages in arena address space
    
    goroutine 328 [running]:
    [fp=0x16c787c] runtime.throw(0x665341)
            c:/Go/src/pkg/runtime/panic.c:473 +0x65
    [fp=0x16c789c] runtime.SysMap(0x3ffa0000, 0x100000)
            c:/Go/src/pkg/runtime/mem_windows.c:70 +0x78
    [fp=0x16c78bc] runtime.MHeap_SysAlloc(0x1730000, 0x100000)
            c:/Go/src/pkg/runtime/zmalloc_windows_386.c:360 +0x103
    [fp=0x16c78e4] MHeap_Grow(0x1730000, 0x10)
            c:/Go/src/pkg/runtime/mheap.c:224 +0x4f
    [fp=0x16c7904] MHeap_AllocLocked(0x1730000, 0x4, 0x2b)
            c:/Go/src/pkg/runtime/mheap.c:108 +0x286
    [fp=0x16c7918] runtime.MHeap_Alloc(0x1730000, 0x4, 0x2b, 0x0, 0x1, ...)
            c:/Go/src/pkg/runtime/mheap.c:77 +0x5c
    [fp=0x16c7950] MCentral_Grow(0x1b35d4c)
            c:/Go/src/pkg/runtime/mcentral.c:203 +0x81
    [fp=0x16c7974] runtime.MCentral_AllocList(0x1b35d4c, 0xc, 0x16c7988)
            c:/Go/src/pkg/runtime/mcentral.c:47 +0x4c
    [fp=0x16c7994] runtime.MCache_Alloc(0x15d0000, 0x2b, 0x1400, 0x1)
            c:/Go/src/pkg/runtime/mcache.c:24 +0x72
    [fp=0x16c79c4] runtime.mallocgc(0x1400, 0x1, 0x1, 0x1)
            c:/Go/src/pkg/runtime/zmalloc_windows_386.c:47 +0xdf
    [fp=0x16c79e4] makeslice1(0x5021e0, 0x13e0, 0x13e0, 0x16c7a18)
            c:/Go/src/pkg/runtime/slice.c:61 +0x7f
    [fp=0x16c7a04] runtime.makeslice(0x5021e0, 0x13e0, 0x0, 0x13e0, 0x0, ...)
            c:/Go/src/pkg/runtime/slice.c:34 +0x9c
    [fp=0x16c7a58] github.com/syndtr/goleveldb/leveldb/table.(*bInfo).readAll(0x3f9feb80, 0x201818c0, 0x3f9b4a98, 0x20181800, 0x0, ...)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/table/block.go:66 +0x74
    [fp=0x16c7a90] github.com/syndtr/goleveldb/leveldb/table.(*Reader).getBlock(0x3f9bd0f0, 0x3f9feb80, 0x20181440, 0x3f9b4988, 0x0, ...)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:171 +0x8e
    [fp=0x16c7ac8] github.com/syndtr/goleveldb/leveldb/table.(*Reader).getDataIter(0x3f9bd0f0, 0x3f9feb80, 0x20181440, 0x3f9b4988, 0x0, ...)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:202 +0x174
    [fp=0x16c7afc] github.com/syndtr/goleveldb/leveldb/table.(*indexIter).Get(0x3f9b19c0, 0x0, 0x0, 0x0, 0x0, ...)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:234 +0xd0
    [fp=0x16c7b14] github.com/syndtr/goleveldb/leveldb/iterator.(*IndexedIterator).setData(0x3f9b19e0, 0x1)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:138
    +0x2d
    [fp=0x16c7b20] github.com/syndtr/goleveldb/leveldb/iterator.(*IndexedIterator).Next(0x3f9b19e0, 0x3ff9d800)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:85 +0x7f
    [fp=0x16c7b2c] github.com/syndtr/goleveldb/leveldb/iterator.(*IndexedIterator).Next(0x3f9b17e0, 0x4ba7c0)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:84 +0x45
    [fp=0x16c7b70] github.com/syndtr/goleveldb/leveldb/iterator.(*MergedIterator).Next(0x3ea83ea0, 0x3fdb03a0)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:112 +0x124
    [fp=0x16c7c1c] github.com/syndtr/goleveldb/leveldb.func┬Ě015(0x0, 0x0)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:247 +0x172
    [fp=0x16c7c6c] github.com/syndtr/goleveldb/leveldb.(*DB).transact(0x200bd500, 0x16c7cac)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:136 +0x74
    [fp=0x16c7f2c] github.com/syndtr/goleveldb/leveldb.(*DB).doCompaction(0x200bd500, 0x3e997480, 0x16c7f00)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:365 +0x75d
    [fp=0x16c7fd4] github.com/syndtr/goleveldb/leveldb.(*DB).compaction(0x200bd500)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:463 +0x17b
    [fp=0x16c7fd8] runtime.goexit()
            c:/Go/src/pkg/runtime/proc.c:1223
    created by github.com/syndtr/goleveldb/leveldb.openDB
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db.go:72 +0x1e6
    
    goroutine 1 [chan receive]:
    testing.RunTests(0x58aa7c, 0x666dc0, 0x3c, 0x3c, 0x1, ...)
            c:/Go/src/pkg/testing/testing.go:434 +0x69e
    testing.Main(0x58aa7c, 0x666dc0, 0x3c, 0x3c, 0x665a60, ...)
            c:/Go/src/pkg/testing/testing.go:365 +0x68
    main.main()
            C:/Users/Dobek/AppData/Local/Temp/go-build124087311/github.com/syndtr/goleveldb/leveldb/_test/_testmain.go:199 +0x80
    
    goroutine 327 [chan send]:
    github.com/syndtr/goleveldb/leveldb.(*DB).flush(0x200bd500, 0x0, 0x0, 0x0)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_write.go:62 +0x176
    github.com/syndtr/goleveldb/leveldb.(*DB).Write(0x200bd500, 0x3ea85f20, 0x204cd010, 0x0, 0x0, ...)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_write.go:109 +0x181
    github.com/syndtr/goleveldb/leveldb.(*DB).Put(0x200bd500, 0x3ea6ecb0, 0xb, 0xb, 0x3eaa6400, ...)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_write.go:162 +0x84
    github.com/syndtr/goleveldb/leveldb.(*dbHarness).put(0x204f5480, 0x3ea6eca0, 0xb, 0x21ba8800, 0x3e8, ...)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_test.go:122 +0xb5
    github.com/syndtr/goleveldb/leveldb.TestDb_SparseMerge(0x2012b540)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_test.go:928 +0x165
    testing.tRunner(0x2012b540, 0x666f4c)
            c:/Go/src/pkg/testing/testing.go:353 +0x86
    created by testing.RunTests
            c:/Go/src/pkg/testing/testing.go:433 +0x683
    
    goroutine 329 [chan receive]:
    github.com/syndtr/goleveldb/leveldb.(*DB).writeJournal(0x200bd500)
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_write.go:25 +0x3b
    created by github.com/syndtr/goleveldb/leveldb.openDB
            D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db.go:73 +0x1fb
    
    goroutine 7 [runnable]:
    exit status 2
    FAIL    github.com/syndtr/goleveldb/leveldb     15.997s
    

    Best regards, Dobrosław Żybort

    opened by matrixik 11
  • leveldb: fix table file leaks when manifest is rotated

    leveldb: fix table file leaks when manifest is rotated

    When switched to the master branch, the size of my project's result db is much larger than before. The size is shrunk when restarted and many logs [email protected] removing table-xxx printed. I found that it's caused by the recently merge PR #380.

    opened by qianbin 0
  • util: improve BufferPool, eliminate unexpected allocations

    util: improve BufferPool, eliminate unexpected allocations

    Using sync.Pool to manage bare slices is not the proper way, since it'll bring unexpected allocations. This PR solves this problem with small tricks and makes BufferPool a bit faster.

    benchmarks:

    $ benchstat before.txt after.txt 
    name          old time/op    new time/op    delta
    BufferPool-8    55.9ns ± 0%    34.8ns ± 0%   -37.71%  (p=0.008 n=5+5)
    
    name          old alloc/op   new alloc/op   delta
    BufferPool-8     24.0B ± 0%      0.0B       -100.00%  (p=0.008 n=5+5)
    
    name          old allocs/op  new allocs/op  delta
    BufferPool-8      1.00 ± 0%      0.00       -100.00%  (p=0.008 n=5+5)
    
    opened by qianbin 0
  • Browser indexeddb in golang / wasm

    Browser indexeddb in golang / wasm

    There is also an indexeddb implementation in golang here. It compiles to wasm and uses the browsers built in indexeddb.

    https://github.com/hack-pad/go-indexeddb

    it would be cool to setup tests to run against both implementations.

    opened by gedw99 0
  • memdb: merge nodeData and kvData to speedup access

    memdb: merge nodeData and kvData to speedup access

    This PR makes memdb save skiplist nodes and kvs together in one byte-slice. The performance of memdb is significantly improved with this change(it's believed that the CPU cache hit-rate goes up).

    Benchmarks:

    $ benchstat before.txt after.txt 
    name         old time/op  new time/op  delta
    Put-8        1.08µs ± 1%  1.00µs ± 1%   -7.98%  (p=0.008 n=5+5)
    PutRandom-8  1.85µs ± 1%  1.51µs ± 0%  -18.42%  (p=0.008 n=5+5)
    Get-8        1.24µs ± 1%  1.13µs ± 1%   -9.54%  (p=0.008 n=5+5)
    GetRandom-8  2.09µs ± 1%  1.69µs ± 1%  -19.17%  (p=0.008 n=5+5)
    
    opened by qianbin 1
Releases(v1.0.0)
Owner
Suryandaru Triandana
Suryandaru Triandana
Golang-key-value-store - Key Value Store API Service with Go DDD Architecture

This document specifies the tools used in the Key-Value store and reorganizes how to use them. In this created service, In-Memory Key-Value Service was created and how to use the API is specified in the HTML file in the folder named "doc"

Kemal Emre Ballı 3 Mar 16, 2022
A key-value db api with multiple storage engines and key generation

Jet is a deadly-simple key-value api. The main goals of this project are : Making a simple KV tool for our other projects. Learn tests writing and git

null 12 Apr 5, 2022
levigo is a Go wrapper for LevelDB

levigo levigo is a Go wrapper for LevelDB. The API has been godoc'ed and is available on the web. Questions answered at [email protected]

Jeff Hodges 404 May 5, 2022
An embedded key/value database for Go.

Bolt Bolt is a pure Go key/value store inspired by Howard Chu's LMDB project. The goal of the project is to provide a simple, fast, and reliable datab

BoltDB 12.9k Jun 25, 2022
Key-value database stored in memory with option of persistence

Easy and intuitive command line tool allows you to spin up a database avaliable from web or locally in a few seconds. Server can be run over a custom TCP protocol or over HTTP.

Mario Petričko 7 Jan 8, 2022
ZedisDB - a key-value memory database written in Go

ZedisDB - a key-value memory database written in Go

Znkisoft 3 Feb 2, 2022
Simple Distributed key-value database (in-memory/disk) written with Golang.

Kallbaz DB Simple Distributed key-value store (in-memory/disk) written with Golang. Installation go get github.com/msam1r/kallbaz-db Usage API // Get

Mohamed Samir 5 Jan 18, 2022
FlashDB is an embeddable, in-memory key/value database in Go

FlashDB is an embeddable, in-memory key/value database in Go (with Redis like commands and super easy to read)

Farhan 248 Jun 23, 2022
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Peter Bourgon 1.2k Jun 26, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 40.3k Jun 24, 2022
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Burak Sezer 2.2k Jul 1, 2022
Simple, ordered, key-value persistence library for the Go Language

gkvlite gkvlite is a simple, ordered, ACID, key-value persistence library for Go. Overview gkvlite is a library that provides a simple key-value persi

Steve Yen 254 Jun 23, 2022
Distributed, fault-tolerant key-value storage written in go.

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.

Igor German 359 Mar 4, 2022
a persistent real-time key-value store, with the same redis protocol with powerful features

a fast NoSQL DB, that uses the same RESP protocol and capable to store terabytes of data, also it integrates with your mobile/web apps to add real-time features, soon you can use it as a document store cause it should become a multi-model db. Redix is used in production, you can use it in your apps with no worries.

Mohammed Al Ashaal 1.1k Jun 13, 2022
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jake Grogan 721 Jun 9, 2022
Pogreb is an embedded key-value store for read-heavy workloads written in Go.

Embedded key-value store for read-heavy workloads written in Go

Artem Krylysov 900 Jun 26, 2022
HA LDAP based key/value solution for projects configuration storing with multi master replication support

Recon is the simple solution for storing configs of you application. There are no specified instruments, no specified data protocols. For the full power of Recon you only need curl.

Mikhail Panfilov 12 Jun 15, 2022
Fault tolerant, sharded key value storage written in GoLang

Ravel is a sharded, fault-tolerant key-value store built using BadgerDB and hashicorp/raft. You can shard your data across multiple clusters with mult

Aditya Meharia 75 May 11, 2022
CrankDB is an ultra fast and very lightweight Key Value based Document Store.

CrankDB is a ultra fast, extreme lightweight Key Value based Document Store.

Shrey Batra 30 Apr 12, 2022