Distributed disk storage database based on Raft and Redis protocol.

Related tags

Key-Value Store redis golang distributed-systems database disk nosql key-value storage high-performance cache raft leveldb persistence databases kvstore distributed-database lsm
Overview

IceFireDB

IceFireDB

test build FOSSA Status

Distributed disk storage system based on Raft and RESP protocol.

  1. High performance
  2. Distributed consistency
  3. Reliable LSM disk storage
  4. Cold and hot mixed data storage structure
  5. Multiple groups of raft clusters, horizontal expansion and contraction of data groups（Upgrading soon）
  6. More advanced cache implementation, faster LSM persistent storage

Imagine Architecture

IceFireDB_Architecture

Command support

1. String operating

  • APPEND
  • BITCOUNT
  • BITOP
  • BITPOS
  • DECR
  • DECRBY
  • DEL
  • EXISTS
  • GET
  • GETBIT
  • SETBIT
  • GETRANGE
  • GETSET
  • INCR
  • INCRBY
  • MGET
  • MSET
  • SET
  • SETEX
  • SETEXAT
  • SETRANGE
  • EXPIRE
  • EXPIREAT
  • TTL

2. Hash operating

  • HSET
  • HGET
  • HDEL
  • HEXISTS
  • HGETALL
  • HINCRBY
  • HKEYS
  • HLEN
  • HMGET
  • HMSET
  • HSETEX
  • HSTRLEN
  • HVALS
  • HCLEAR
  • HMCLEAR
  • HEXPIRE
  • HEXPIREAT
  • HKEYEXIST
  • HTTL

3. List operating

  • RPUSH
  • LPOP
  • LINDEX
  • LPUSH
  • RPOP
  • LRANGE
  • LSET
  • LLEN
  • RPOPLPUSH
  • LCLEAR
  • LMCLEAR
  • LEXPIRE
  • LEXPIREAT
  • LKEYEXISTS
  • LTRIM
  • LTTL

4. Sorted Sets

  • ZADD
  • ZCARD
  • ZCOUNT
  • ZREM
  • ZCLEAR
  • ZRANK
  • ZRANGE
  • ZREVRANGE
  • ZSCORE
  • ZINCRBY
  • ZREVRANK
  • ZRANGEBYSCORE
  • ZREVRANGEBYSCORE
  • ZREMRANGEBYSCORE
  • ZREMRANGEBYRANK

5. Sets

  • SADD
  • SCARD
  • SDIFF
  • SDIFFSTORE
  • SINTER
  • SINTERSTORE
  • SISMEMBER
  • SMEMBERS
  • SREM
  • SUNION
  • SUNIONSTORE
  • SCLEAR
  • SMCLEAR
  • SEXPIRE
  • SEXPIREAT
  • STTL
  • SPERSIST
  • SKEYEXISTS

6. System cmd

  • INFO
  • FLUSHALL
  • HSCAN
  • SSCAN
  • ZSCAN
  • XSCAN
  • XHSCAN
  • XSSCAN
  • XZSCAN

Performance

corerman@ubuntu:~/DATA/ICODE/GoLang/IceFireDB$ redis-benchmark  -h 127.0.0.1 -p 11001 -n 10000000 -t set,get -c 512 -P 512 -q

SET: 253232.12 requests per second
GET: 2130875.50 requests per second

License

FOSSA Status

Thanks

I stood on the shoulders of giants and did only simple things. Thank you for your attention.

Disclaimers

When you use this software, you have agreed and stated that the author, maintainer and contributor of this software are not responsible for any risks, costs or problems you encounter. If you find a software defect or BUG, ​​please submit a patch to help improve it!

Comments
  • chore(deps): bump github.com/dgraph-io/badger/v3 from 3.2103.1 to 3.2103.2

    chore(deps): bump github.com/dgraph-io/badger/v3 from 3.2103.1 to 3.2103.2

    Bumps github.com/dgraph-io/badger/v3 from 3.2103.1 to 3.2103.2.

    Release notes

    Sourced from github.com/dgraph-io/badger/v3's releases.

    BadgerDB v3.2103.2

    This patch release contains:

    Fixed

    • fix(compact): close vlog after the compaction at L0 has been completed (#1752)
    • fix(builder): put the upper limit on reallocation (#1748)
    • deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)
    • fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)
    • fix(pubsub): avoid deadlock in publisher and subscriber (#1749) (#1751)

    Full Changelog: https://github.com/dgraph-io/badger/compare/v3.2103.1...v3.2103.2

    Changelog

    Sourced from github.com/dgraph-io/badger/v3's changelog.

    [3.2103.2] - 2021-10-07

    Fixed

    • fix(compact): close vlog after the compaction at L0 has completed (#1752)
    • fix(builder): put the upper limit on reallocation (#1748)
    • deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)
    • fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)
    • fix(pubsub): avoid deadlock in publisher and subscriber (#1749) (#1751)
    Commits
    • f388e91 Update changelog for v3.2103.2
    • 09a6157 fix(compact): close vlog after the compaction at L0 has completed (#1752)
    • 4298c58 fix(builder): put the upper limit on reallocation (#1748)
    • 644971a deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)
    • 0c13869 Fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)
    • e4e2ad7 fix: avoid deadlock in publisher and subscriber (#1749) (#1751)
    • See full diff in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot will merge this PR once CI passes on it, as requested by @gitsrc.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 8
  • chore(deps): bump github.com/go-redis/redis/v8 from 8.11.3 to 8.11.4

    chore(deps): bump github.com/go-redis/redis/v8 from 8.11.3 to 8.11.4

    Bumps github.com/go-redis/redis/v8 from 8.11.3 to 8.11.4.

    Release notes

    Sourced from github.com/go-redis/redis/v8's releases.

    v8.11.4

    Please refer to CHANGELOG.md for details

    Changelog

    Sourced from github.com/go-redis/redis/v8's changelog.

    8.11.4 (2021-10-04)

    Features

    • add acl auth support for sentinels (f66582f)
    • add Cmd.{String,Int,Float,Bool}Slice helpers and an example (5d3d293)
    • add SetVal method for each command (168981d)

    v8.11

    • Remove OpenTelemetry metrics.
    • Supports more redis commands and options.

    v8.10

    • Removed extra OpenTelemetry spans from go-redis core. Now go-redis instrumentation only adds a single span with a Redis command (instead of 4 spans). There are multiple reasons behind this decision:

      • Traces become smaller and less noisy.
      • It may be costly to process those 3 extra spans for each query.
      • go-redis no longer depends on OpenTelemetry.

      Eventually we hope to replace the information that we no longer collect with OpenTelemetry Metrics.

    v8.9

    • Changed PubSub.Channel to only rely on Ping result. You can now use WithChannelSize, WithChannelHealthCheckInterval, and WithChannelSendTimeout to override default settings.

    v8.8

    • To make updating easier, extra modules now have the same version as go-redis does. That means that you need to update your imports:
    github.com/go-redis/redis/extra/redisotel -> github.com/go-redis/redis/extra/redisotel/v8
github.com/go-redis/redis/extra/rediscensus -> github.com/go-redis/redis/extra/rediscensus/v8

    v8.5

    • knadh contributed long-awaited ability to scan Redis Hash into a struct:
    </tr></table>

    ... (truncated)

    Commits
    • fa51524 Merge pull request #1917 from go-redis/release/v8.11.4
    • 0457d0a chore: release v8.11.4 (release.sh)
    • 6dcfc56 chore: use conv commit message
    • 86258a1 chore: lazy reload when moved or ask
    • 98bb99d Fix Redis Cluster issue during roll outs of new nodes with same addr (#1914)
    • 5072031 chore: add Lua scripting example
    • 739c188 chore: tweak GH actions to run all jobs
    • cc09f96 Merge pull request #1909 from go-redis/feature/slice-helpers
    • 5d3d293 feat: add Cmd.{String,Int,Float,Bool}Slice helpers and an example
    • 329d87f Merge pull request #1906 from Stavrospanakakis/master
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot will merge this PR once it's up-to-date and CI passes on it, as requested by @gitsrc.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 4
  • Feat add crdt driver

    Feat add crdt driver

    New decentralized crdt kV database driver.

    Usage:

    IceFireDB -a :11001 \
    --storage-backend crdt \
    --servicename icefiredb \
    --nettopic icefiredb-net \
    --datatopic icefiredb-data
    opened by pretty66 3
  • Documentation to start up and query cluster?

    Documentation to start up and query cluster?

    Hello,

    I was able to compile IceFireDB but have not found any documentation that shows the steps to start up a multi-node cluster and then make some queries?

    I am interested in highly distributed P2P usage and scaling limits of IceFireDB Thanks

    opened by lonnietc 3
  • Missing Copyright Notice and License to Uhaha

    Missing Copyright Notice and License to Uhaha

    Hi,

    The source code in the rafthub directory of this project is directly copied from https://github.com/tidwall/uhaha/blob/master/uhaha.go, which is 3K+ lines of original source code written and copyrighted by me.

    The original copyright notice and LICENSE were omitted.

    The Uhaha LICENSE states:

    The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

    opened by tidwall 3
  • chore(deps): bump github.com/tidwall/redcon from 1.4.1 to 1.4.2

    chore(deps): bump github.com/tidwall/redcon from 1.4.1 to 1.4.2

    Bumps github.com/tidwall/redcon from 1.4.1 to 1.4.2.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 2
  • feat(RedisCmdRewrite): Implementation of redis instruction rewrite

    feat(RedisCmdRewrite): Implementation of redis instruction rewrite

    Hello, because the raft log rollback exception occurs when raft is combined with the underlying storage, I helped you build an instruction rewrite plug-in layer, and "ArgsHandler" implements the redis instruction rewrite interface. Processing logic introduction:

    • Receive the redis command and current time of type [][]string.
    • Traverse the args, and rewrite the instructions as setex, expire, lexpire, and hexpire through the timestamp.
    opened by Jchicode 2
  • feat(hybriddb): add the cache layer

    feat(hybriddb): add the cache layer

    Hello, I helped you build a hot data caching layer, "Hybriddb" implements ledisdb storage interface.

    Introduction to processing logic:

    • When reading kv data, first check the memory cache, if it is lost, query the disk data and store it in the memory

    • When writing kv data, directly clear the memory cache and wait for the next read data cache to the memory

    opened by pretty66 2
  • feat: add tests for cmds

    feat: add tests for cmds

    Hi,

    Great works, I add tests for commands.

    Still working in progress.

    Done!

    • [x] strings commands
    • [x] hash commands
    • [x] list commands
    • [x] sorted commands
    opened by mayocream 2
  • Neat idea, but need more documentation

    Neat idea, but need more documentation

    I want to understand what is the architecture. What is the process of configuring, adding, removing nodes?

    How was raft integrated with redis, IE. where was raft end and redis begin?

    documentation 
    opened by Pyrolistical 2
  • chore(deps): bump github.com/libp2p/go-libp2p from 0.14.4 to 0.18.0 in /IceFireDB-PubSub

    chore(deps): bump github.com/libp2p/go-libp2p from 0.14.4 to 0.18.0 in /IceFireDB-PubSub

    Bumps github.com/libp2p/go-libp2p from 0.14.4 to 0.18.0.

    Release notes

    Sourced from github.com/libp2p/go-libp2p's releases.

    v0.18.0

    We're happy to announce the v0.18.0 release of go-libp2p.

    🛠 Breaking Changes

    In this release, we're continuing our repo consolidation. We've moved the following repositories into go-libp2p:

    • go-libp2p-autonat
    • go-libp2p-discovery
    • go-libp2p-connmgr
    • go-addr-utils
    • go-libp2p-netutil
    • go-sockaddr

    If you've been using functions from these repositories, please make sure to point those to go-libp2p.

    🔦 Highlights

    The main feature of v0.18.0 is the Resource Manager. The Resource Manager allows the configuration of (among others):

    • the number of incoming / outgoing connections
    • the number of incoming / outgoing streams
    • the number of streams per protocol, and per service
    • memory usage

    For details on how this works, and how to change limits, refer to the README of the Resource Manager.

    In this release, we've finished the transition from ipfs/go-log@v1 to ipfs/go-log@v2. One less dependency!

    Changelog

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page.
    dependencies go 
    opened by dependabot[bot] 1
Owner
IceFireDB
Distributed disk storage database based on Raft and Redis protocol.
IceFireDB
GitHub https://www.icefiredb.com
Simple Distributed key-value database (in-memory/disk) written with Golang.

Kallbaz DB Simple Distributed key-value store (in-memory/disk) written with Golang. Installation go get github.com/msam1r/kallbaz-db Usage API // Get

Mohamed Samir 5 Jan 18, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

null 7 Jan 4, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

ytt 0 Jul 26, 2022
Tinykv - The TinyKV course builds a key-value storage system with the Raft consensus algorithm

The TinyKV Course The TinyKV course builds a key-value storage system with the R

null 1 Dec 7, 2022
a persistent real-time key-value store, with the same redis protocol with powerful features

a fast NoSQL DB, that uses the same RESP protocol and capable to store terabytes of data, also it integrates with your mobile/web apps to add real-time features, soon you can use it as a document store cause it should become a multi-model db. Redix is used in production, you can use it in your apps with no worries.

Mohammed Al Ashaal 1.1k Dec 25, 2022
Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.

Consul Website: https://www.consul.io Tutorials: HashiCorp Learn Forum: Discuss Consul is a distributed, highly available, and data center aware solut

HashiCorp 25.8k Jan 2, 2023
Distributed, fault-tolerant key-value storage written in go.

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.

Igor German 359 Jan 3, 2023
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Peter Bourgon 1.2k Jan 1, 2023
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 42.2k Dec 28, 2022
rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask.

rosedb is a fast, stable and embedded key-value (k-v) storage engine based on bitcask. Its on-disk files are organized as WAL(Write Ahead Log) in LSM trees, optimizing for write throughput.

roseduan 3.4k Dec 28, 2022
Implementation of distributed key-value system based on TiKV

Distributed_key-value_system A naive implementation of distributed key-value system based on TiKV Features Features of this work are listed below: Dis

null 2 Mar 7, 2022
A key-value db api with multiple storage engines and key generation

Jet is a deadly-simple key-value api. The main goals of this project are : Making a simple KV tool for our other projects. Learn tests writing and git

null 12 Apr 5, 2022
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Burak Sezer 2.7k Jan 4, 2023
moss - a simple, fast, ordered, persistable, key-val storage library for golang

moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library.

null 895 Dec 18, 2022
Fault tolerant, sharded key value storage written in GoLang

Ravel is a sharded, fault-tolerant key-value store built using BadgerDB and hashicorp/raft. You can shard your data across multiple clusters with mult

Aditya Meharia 77 Nov 1, 2022
A key-value storage transaction interpretator.

kv-txn-interpreter A key-value storage transaction interpreter, which provides an etcd-like transaction interface to help you build a transaction over

JmPotato 2 Feb 22, 2022
Keyval - A simple key-value storage library written in Go

keyval keyval is a simple key-value storage library written in Go and its back c

sina safari 2 Sep 16, 2022
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jake Grogan 734 Jan 6, 2023
Distributed key-value store

Keva Distributed key-value store General Demo Start the server docker-compose up --build Insert data curl -XPOST http://localhost:5555/storage/test1

Yaroslav Gaponov 0 Nov 15, 2021