Constant Database native golang implementation

Related tags

Database tools cdb
Overview

CDB golang implementation

Go Report Card GoDoc

cdb is a fast, reliable, simple package for creating and reading constant databases see docs for more details

Advantages

  • Iterator support
  • Thread safe for reading
  • Lazily key, value reading using io.SectionReader
  • Buffered disc write

Example

f, _ := os.Create("test.cdb")
defer f.Close()

handle := cdb.New()

data := []struct {
    key, value string
}{
    {"key1", "value1"},
    {"key2", "value2"},
    {"key3", "value3"},
    {"key4", "value4"},
    {"key5", "value5"},
    {"key6", "value6"},
    {"key7", "value7"},
}

writer, err := handle.GetWriter(f)
if err != nil {
    t.Error(err)
}

for _, c := range data {
    writer.Put([]byte(c.key), []byte(c.value))
}

writer.Close()
reader, _ := handle.GetReader(f)

for _, c := range data {
    value, err := reader.Get([]byte(c.key))
}

Performance tricks

  • File mmap shows better performance. Example.
Comments
  • Iterator performance improvements

    Iterator performance improvements

    Hi, @alldroll !

    I removed few allocations in this PR. And added KeyBytes, ValueBytes methods to public api of Record. These methods don't requires extra allocation for io.NewSectionReader and they are simple to use.

    Here is benchmark result:

    BenchmarkGetReader-8              581259              2252 ns/op            4352 B/op          2 allocs/op
    BenchmarkReaderGet-8              693465              1508 ns/op              64 B/op          6 allocs/op
    BenchmarkWriterPut-8              312033              3523 ns/op             283 B/op         16 allocs/op
    BenchmarkIteratorAt-8             777184              1369 ns/op             185 B/op          9 allocs/op
    BenchmarkIteratorNext-8          2015468               593 ns/op               8 B/op          0 allocs/op
    BenchmarkIteratorRecordKey-8     1315321              1072 ns/op             129 B/op          5 allocs/op
    BenchmarkIteratorKey-8           2032114               553 ns/op               1 B/op          1 allocs/op
    

    Old benchmark result:

    BenchmarkGetReader-8              334984              3207 ns/op            4352 B/op          2 allocs/op
    BenchmarkReaderGet-8              440910              2632 ns/op              64 B/op          6 allocs/op
    BenchmarkWriterPut-8              158534              8599 ns/op             279 B/op         16 allocs/op
    BenchmarkIteratorAt-8             404857              2549 ns/op             185 B/op          9 allocs/op
    BenchmarkIteratorNext-8          1831125               771 ns/op               8 B/op          1 allocs/op
    
    ## Release notes:
    
    ### Added
    * `Key()` and `Value()` methods in `Iterator` interface. It is usually easier to use and faster then `iterator.Record().Key()`. Because it doesn't requiers allocation for record copy. But `iterator.Record().Key()` is lazy. It means, a data is read on require.
    ```
    BenchmarkIteratorRecordKey-8     1315321              1072 ns/op             129 B/op          5 allocs/op
    BenchmarkIteratorKey-8           2032114               553 ns/op               1 B/op          1 allocs/op
    ```
    
    ### Changed:
    * `cdbReader.Iterator()` method returns `ErrEmptyCDB` for empty cdb files
    * `cdbReader.Get(key)` method returns `ErrEntryNotFound` if `key` was not found in cdb file
    
    enhancement 
    opened by mrsndmn 7
  • Update tests

    Update tests

    I've refactored tests with testify. Seems it looks nice now.

    Unfortunately, I cant do anything with benchmarks. There is no support of benchmarks in testify/suite.

    opened by mrsndmn 3
  • cdb mmaping

    cdb mmaping

    Hi!

    CDB file mmapping could improve performance. I wouldn't recomend to use it with fat CDBs.

    I benched it here and got 7 times performance improvement on my laptop.

    BenchmarkReaderGet-8          	  759301	      1397 ns/op
    BenchmarkReaderGetMMaped-8    	 6424041	       194 ns/op
    

    May be we should add this point to README.md?

    mmapping is better than using of tmpfs because mmapped file reading does not requre any syscall

    good first issue 
    opened by mrsndmn 1
  • use bytes.Buffer copy instead of make slice #2

    use bytes.Buffer copy instead of make slice #2

    #2 I fix to be using io.Copy from reader stream to byte.Buffer instead of making slice object. And I would like to indicate memory usage performance. Let me how to measure the performance which you tested.

    opened by u5surf 1
  • make .go

    make .go

    You have moved github.com/alldroll/go-datastructures/cdb

    https://github.com/alldroll/go-datastructures/commit/0f1c7859cbe52838a8a2faf7b69b82d51d1915bf

    opened by robotamer 1
  • dump.go optimize memory usage

    dump.go optimize memory usage

    Record Key, Value methods return io.Reader with corresponding size. Every loop step allocates memory to read the key and value. Reusing already allocated memory should help with this issue.

    enhancement help wanted 
    opened by alldroll 0
Releases(v1.0.3)
Owner
Alexander Petrov
Alexander Petrov
[mirror] the database client and tools for the Go vulnerability database

The Go Vulnerability Database golang.org/x/vulndb This repository is a prototype of the Go Vulnerability Database. Read the Draft Design. Neither the

Go 201 Nov 25, 2022
Database - Example project of database realization using drivers and models

database Golang based database realization Description Example project of databa

Denis 1 Feb 10, 2022
MatrixOne is a planet scale, cloud-edge native big data engine crafted for heterogeneous workloads.

What is MatrixOne? MatrixOne is a planet scale, cloud-edge native big data engine crafted for heterogeneous workloads. It provides an end-to-end data

Matrix Origin 1.4k Dec 7, 2022
Minimal memory usage, cloud native logstash alternative

Mr-Plow Tiny and minimal tool to export data from relational db (postgres or mysql) to elasticsearch. The tool does not implement all the logstash fea

RingLoop 29 Aug 18, 2022
Database migrations. CLI and Golang library.

migrate Database migrations written in Go. Use as CLI or import as library. Migrate reads migrations from sources and applies them in correct order to

null 10.4k Dec 7, 2022
database to golang struct

中文文档 mysql database to golang struct conversion tools base on gorm(v1/v2),You can automatically generate golang sturct from mysql database. big Camel-

xxj 2.1k Dec 2, 2022
Golang restAPI crud project with mySql database.

Golang RestAPI using gorilla/mux Golang restAPI crud project with mySql database. Test Api with Thunder Client vs code beautiful Extension. and use Be

Md Abu. Raihan 6 Mar 26, 2022
Database Access Layer for Golang - Testable, Extendable and Crafted Into a Clean and Elegant API

REL Modern Database Access Layer for Golang. REL is golang orm-ish database layer for layered architecture. It's testable and comes with its own test

REL 605 Dec 6, 2022
A simple Golang-based application that queries a PostgreSQL database

Qwik-E-Mart Demo App A simple Golang-based application that queries a PostgreSQL database named qwikemart to read and return customer data stored in t

Joe Garcia 0 Nov 6, 2021
Implementasi database oracle kedalam golang

Go with Oracle database Implementasi database oracle kedalam golang How to using swagger Install generator swagger menggunakan perintah : go get -u gi

Muhammad Rais Adlani 0 Nov 20, 2021
Golang database driver for SQLite

go-sqlite Golang database driver for SQLite. Does not use cgo. This driver is based on pure-Go SQLite implementation (https://gitlab.com/cznic/sqlite)

glebarez 153 Dec 5, 2022
BigQuery database/sql golang driver

BigQuery SQL Driver This library is compatible with Go 1.17+ Please refer to CHA

Viant, Inc 6 Aug 31, 2022
🏋️ dbbench is a simple database benchmarking tool which supports several databases and own scripts

dbbench Table of Contents Description Example Installation Supported Databases Usage Custom Scripts Troubeshooting Development Acknowledgements Descri

Simon Jürgensmeyer 77 Nov 21, 2022
Database wrapper that manage read write connections

rwdb Database wrapper that manage read write connections Install go get github.com/andizzle/rwdb Create connections package main import "github.com/

Andy Zhang 16 Sep 26, 2022
Vitess is a database clustering system for horizontal scaling of MySQL.

Vitess Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vite

Vitess 15.2k Dec 2, 2022
sqlx is a library which provides a set of extensions on go's standard database/sql library

sqlx is a library which provides a set of extensions on go's standard database/sql library. The sqlx versions of sql.DB, sql.TX, sql.Stmt, et al. all leave the underlying interfaces untouched, so that their interfaces are a superset on the standard ones. This makes it relatively painless to integrate existing codebases using database/sql with sqlx.

Jason Moiron 12.9k Dec 6, 2022
Convert data exports from various services to a single SQLite database

Bionic Bionic is a tool to convert data exports from web apps to a single SQLite database. Bionic currently supports data exports from Google, Apple H

Bionic 173 Dec 4, 2022
Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

?? Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

PingCAP 267 Nov 9, 2022
Zero boilerplate database operations for Go

(Now compatible with MySQL and PostgreSQL!) Everyone knows that performing simple DATABASE queries in Go takes numerous lines of code that is often re

null 355 Nov 14, 2022