IceFireDB - Distributed disk storage system based on Raft and RESP protocol.

Related tags

Database redis golang distributed-systems database disk nosql key-value storage high-performance cache raft leveldb persistence databases kvstore distributed-database lsm
Overview

IceFireDB

IceFireDB

FOSSA Status

Distributed disk storage system based on Raft and RESP protocol.

  1. High performance
  2. Distributed consistency
  3. Reliable LSM disk storage
  4. Cold and hot mixed data storage structure（Upgrading soon）
  5. Multiple groups of raft clusters, horizontal expansion and contraction of data groups
  6. More advanced cache implementation, faster LSM persistent storage

Imagine Architecture

IceFireDB_Architecture

Command support

1. String operating

  • APPEND
  • BITCOUNT
  • BITOP
  • BITPOS
  • DECR
  • DECRBY
  • DEL
  • EXISTS
  • GET
  • GETBIT
  • SETBIT
  • GETRANGE
  • GETSET
  • INCR
  • INCRBY
  • MGET
  • MSET
  • SET
  • SETEX
  • SETEXAT
  • SETRANGE
  • EXPIRE
  • EXPIREAT
  • TTL

2. Hash operating

  • HSET
  • HGET
  • HDEL
  • HEXISTS
  • HGETALL
  • HINCRBY
  • HKEYS
  • HLEN
  • HMGET
  • HMSET
  • HSETEX
  • HSTRLEN
  • HVALS
  • HCLEAR
  • HMCLEAR
  • HEXPIRE
  • HEXPIREAT
  • HKEYEXIST
  • HTTL

3. List operating

  • RPUSH
  • LPOP
  • LINDEX
  • LPUSH
  • RPOP
  • LRANGE
  • LSET
  • LLEN
  • RPOPLPUSH
  • LCLEAR
  • LMCLEAR
  • LEXPIRE
  • LEXPIREAT
  • LKEYEXISTS
  • LTRIM
  • LTTL

Performance

corerman@ubuntu:~/DATA/ICODE/GoLang/IceFireDB$ redis-benchmark  -h 127.0.0.1 -p 11001 -n 10000000 -t set,get -c 512 -P 512 -q

SET: 253232.12 requests per second
GET: 2130875.50 requests per second

License

FOSSA Status

Thanks

I stood on the shoulders of giants and did only simple things. Thank you for your attention.

Comments
  • chore(deps): bump github.com/dgraph-io/badger/v3 from 3.2103.1 to 3.2103.2

    chore(deps): bump github.com/dgraph-io/badger/v3 from 3.2103.1 to 3.2103.2

    Bumps github.com/dgraph-io/badger/v3 from 3.2103.1 to 3.2103.2.

    Release notes

    Sourced from github.com/dgraph-io/badger/v3's releases.

    BadgerDB v3.2103.2

    This patch release contains:

    Fixed

    • fix(compact): close vlog after the compaction at L0 has been completed (#1752)
    • fix(builder): put the upper limit on reallocation (#1748)
    • deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)
    • fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)
    • fix(pubsub): avoid deadlock in publisher and subscriber (#1749) (#1751)

    Full Changelog: https://github.com/dgraph-io/badger/compare/v3.2103.1...v3.2103.2

    Changelog

    Sourced from github.com/dgraph-io/badger/v3's changelog.

    [3.2103.2] - 2021-10-07

    Fixed

    • fix(compact): close vlog after the compaction at L0 has completed (#1752)
    • fix(builder): put the upper limit on reallocation (#1748)
    • deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)
    • fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)
    • fix(pubsub): avoid deadlock in publisher and subscriber (#1749) (#1751)
    Commits
    • f388e91 Update changelog for v3.2103.2
    • 09a6157 fix(compact): close vlog after the compaction at L0 has completed (#1752)
    • 4298c58 fix(builder): put the upper limit on reallocation (#1748)
    • 644971a deps: Bump github.com/google/flatbuffers to v1.12.1 (#1746)
    • 0c13869 Fix(levels): Avoid a deadlock when acquiring read locks in levels (#1744)
    • e4e2ad7 fix: avoid deadlock in publisher and subscriber (#1749) (#1751)
    • See full diff in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot will merge this PR once CI passes on it, as requested by @gitsrc.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 8
  • chore(deps): bump github.com/go-redis/redis/v8 from 8.11.3 to 8.11.4

    chore(deps): bump github.com/go-redis/redis/v8 from 8.11.3 to 8.11.4

    Bumps github.com/go-redis/redis/v8 from 8.11.3 to 8.11.4.

    Release notes

    Sourced from github.com/go-redis/redis/v8's releases.

    v8.11.4

    Please refer to CHANGELOG.md for details

    Changelog

    Sourced from github.com/go-redis/redis/v8's changelog.

    8.11.4 (2021-10-04)

    Features

    • add acl auth support for sentinels (f66582f)
    • add Cmd.{String,Int,Float,Bool}Slice helpers and an example (5d3d293)
    • add SetVal method for each command (168981d)

    v8.11

    • Remove OpenTelemetry metrics.
    • Supports more redis commands and options.

    v8.10

    • Removed extra OpenTelemetry spans from go-redis core. Now go-redis instrumentation only adds a single span with a Redis command (instead of 4 spans). There are multiple reasons behind this decision:

      • Traces become smaller and less noisy.
      • It may be costly to process those 3 extra spans for each query.
      • go-redis no longer depends on OpenTelemetry.

      Eventually we hope to replace the information that we no longer collect with OpenTelemetry Metrics.

    v8.9

    • Changed PubSub.Channel to only rely on Ping result. You can now use WithChannelSize, WithChannelHealthCheckInterval, and WithChannelSendTimeout to override default settings.

    v8.8

    • To make updating easier, extra modules now have the same version as go-redis does. That means that you need to update your imports:
    github.com/go-redis/redis/extra/redisotel -> github.com/go-redis/redis/extra/redisotel/v8
github.com/go-redis/redis/extra/rediscensus -> github.com/go-redis/redis/extra/rediscensus/v8

    v8.5

    • knadh contributed long-awaited ability to scan Redis Hash into a struct:
    </tr></table>

    ... (truncated)

    Commits
    • fa51524 Merge pull request #1917 from go-redis/release/v8.11.4
    • 0457d0a chore: release v8.11.4 (release.sh)
    • 6dcfc56 chore: use conv commit message
    • 86258a1 chore: lazy reload when moved or ask
    • 98bb99d Fix Redis Cluster issue during roll outs of new nodes with same addr (#1914)
    • 5072031 chore: add Lua scripting example
    • 739c188 chore: tweak GH actions to run all jobs
    • cc09f96 Merge pull request #1909 from go-redis/feature/slice-helpers
    • 5d3d293 feat: add Cmd.{String,Int,Float,Bool}Slice helpers and an example
    • 329d87f Merge pull request #1906 from Stavrospanakakis/master
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot will merge this PR once it's up-to-date and CI passes on it, as requested by @gitsrc.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 4
  • Feat add crdt driver

    Feat add crdt driver

    New decentralized crdt kV database driver.

    Usage:

    IceFireDB -a :11001 \
    --storage-backend crdt \
    --servicename icefiredb \
    --nettopic icefiredb-net \
    --datatopic icefiredb-data
    opened by pretty66 3
  • Documentation to start up and query cluster?

    Documentation to start up and query cluster?

    Hello,

    I was able to compile IceFireDB but have not found any documentation that shows the steps to start up a multi-node cluster and then make some queries?

    I am interested in highly distributed P2P usage and scaling limits of IceFireDB Thanks

    opened by lonnietc 3
  • Missing Copyright Notice and License to Uhaha

    Missing Copyright Notice and License to Uhaha

    Hi,

    The source code in the rafthub directory of this project is directly copied from https://github.com/tidwall/uhaha/blob/master/uhaha.go, which is 3K+ lines of original source code written and copyrighted by me.

    The original copyright notice and LICENSE were omitted.

    The Uhaha LICENSE states:

    The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

    opened by tidwall 3
  • chore(deps): bump github.com/tidwall/redcon from 1.4.1 to 1.4.2

    chore(deps): bump github.com/tidwall/redcon from 1.4.1 to 1.4.2

    Bumps github.com/tidwall/redcon from 1.4.1 to 1.4.2.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies go 
    opened by dependabot[bot] 2
  • feat(RedisCmdRewrite): Implementation of redis instruction rewrite

    feat(RedisCmdRewrite): Implementation of redis instruction rewrite

    Hello, because the raft log rollback exception occurs when raft is combined with the underlying storage, I helped you build an instruction rewrite plug-in layer, and "ArgsHandler" implements the redis instruction rewrite interface. Processing logic introduction:

    • Receive the redis command and current time of type [][]string.
    • Traverse the args, and rewrite the instructions as setex, expire, lexpire, and hexpire through the timestamp.
    opened by Jchicode 2
  • feat(hybriddb): add the cache layer

    feat(hybriddb): add the cache layer

    Hello, I helped you build a hot data caching layer, "Hybriddb" implements ledisdb storage interface.

    Introduction to processing logic:

    • When reading kv data, first check the memory cache, if it is lost, query the disk data and store it in the memory

    • When writing kv data, directly clear the memory cache and wait for the next read data cache to the memory

    opened by pretty66 2
  • feat: add tests for cmds

    feat: add tests for cmds

    Hi,

    Great works, I add tests for commands.

    Still working in progress.

    Done!

    • [x] strings commands
    • [x] hash commands
    • [x] list commands
    • [x] sorted commands
    opened by mayocream 2
  • Neat idea, but need more documentation

    Neat idea, but need more documentation

    I want to understand what is the architecture. What is the process of configuring, adding, removing nodes?

    How was raft integrated with redis, IE. where was raft end and redis begin?

    documentation 
    opened by Pyrolistical 2
  • chore(deps): bump github.com/libp2p/go-libp2p from 0.14.4 to 0.18.0 in /IceFireDB-PubSub

    chore(deps): bump github.com/libp2p/go-libp2p from 0.14.4 to 0.18.0 in /IceFireDB-PubSub

    Bumps github.com/libp2p/go-libp2p from 0.14.4 to 0.18.0.

    Release notes

    Sourced from github.com/libp2p/go-libp2p's releases.

    v0.18.0

    We're happy to announce the v0.18.0 release of go-libp2p.

    🛠 Breaking Changes

    In this release, we're continuing our repo consolidation. We've moved the following repositories into go-libp2p:

    • go-libp2p-autonat
    • go-libp2p-discovery
    • go-libp2p-connmgr
    • go-addr-utils
    • go-libp2p-netutil
    • go-sockaddr

    If you've been using functions from these repositories, please make sure to point those to go-libp2p.

    🔦 Highlights

    The main feature of v0.18.0 is the Resource Manager. The Resource Manager allows the configuration of (among others):

    • the number of incoming / outgoing connections
    • the number of incoming / outgoing streams
    • the number of streams per protocol, and per service
    • memory usage

    For details on how this works, and how to change limits, refer to the README of the Resource Manager.

    In this release, we've finished the transition from ipfs/go-log@v1 to ipfs/go-log@v2. One less dependency!

    Changelog

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the Security Alerts page.
    dependencies go 
    opened by dependabot[bot] 1
Owner
GITSRC
GITSRC
GitHub https://www.icefiredb.com
A distributed key-value store. On Disk. Able to grow or shrink without service interruption.

Vasto A distributed high-performance key-value store. On Disk. Eventual consistent. HA. Able to grow or shrink without service interruption. Vasto sca

Chris Lu 246 Jan 6, 2023
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 42.3k Jan 9, 2023
Hard Disk Database based on a former database

Hard Disk Database based on a former database

null 0 Nov 1, 2021
The MySQL Cluster Autopilot Management with GTID and Raft

Xenon Overview Xenon is a MySQL HA and Replication Management tool using Raft protocol. Xenon has many cool features, such as: Fast Failover with no l

RadonDB 640 Jan 3, 2023
🔑A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout (LSM+WAL) similar to Riak.

bitcask A high performance Key/Value store written in Go with a predictable read/write performance and high throughput. Uses a Bitcask on-disk layout

James Mills 10 Sep 26, 2022
TiDB is an open source distributed HTAP database compatible with the MySQL protocol

Slack Channel Twitter: @PingCAP Reddit Mailing list: lists.tidb.io For support, please contact PingCAP What is TiDB? TiDB ("Ti" stands for Titanium) i

PingCAP 33.1k Jan 9, 2023
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Peter Bourgon 1.2k Jan 7, 2023
Membin is an in-memory database that can be stored on disk. Data model smiliar to key-value but values store as JSON byte array.

Membin Docs | Contributing | License What is Membin? The Membin database system is in-memory database smiliar to key-value databases, target to effici

Membin 3 Jun 3, 2021
Fsyncperf - A very simple program to tell how fast/slow is fsync on your disk

fsyncperf This is a very simple program to tell you who fast/slow is fsync on yo

lni 11 May 9, 2022
Disgo - A distributed lock based on redis developed with golang

English | 中文 DisGo Introduce DisGo is a distributed lock based on Redis, develop

Jinyu Chen 15 Dec 2, 2022
MyCache - A distributed cache based on GeeCache

借鉴GeeCache实现了MyCache（https://geektutu.com/post/geecache.html） 主要功能： 1.实现了fifo和lr

null 2 Feb 18, 2022
Multitiered file storage API built on Filecoin and IPFS

Powergate Powergate is a multitiered file storage API built on Filecoin and IPFS, and an index builder for Filecoin data. It's designed to be modular

textile.io 368 Dec 20, 2022
A GPU-powered real-time analytics storage and query engine.

AresDB AresDB is a GPU-powered real-time analytics storage and query engine. It features low query latency, high data freshness and highly efficient i

Uber Open Source 2.9k Jan 7, 2023
Export output from pg_stat_activity and pg_stat_statements from Postgres into a time-series database that supports the Influx Line Protocol (ILP).

pgstat2ilp pgstat2ilp is a command-line program for exporting output from pg_stat_activity and pg_stat_statements (if the extension is installed/enabl

Zikani Nyirenda Mwase 4 Dec 15, 2021
Secure storage for personal records built to comply with GDPR

Databunker Databunker is a Personally Identifiable Information (PII) Data Storage Service built to Comply with GDPR and CCPA Privacy Requirements. Pro

null 1k Jan 8, 2023
moss - a simple, fast, ordered, persistable, key-val storage library for golang

moss moss provides a simple, fast, persistable, ordered key-val collection implementation as a 100% golang library. moss stands for "memory-oriented s

null 895 Dec 18, 2022
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice. A simple in-memory database and table implementation are provided, and you can query any data source you want by implementing a few interfaces.

DoltHub 947 Dec 27, 2022
Key-Value Storage written in Go.

kvs kvs is an in-memory key-value storage written in Go. It has 2 different usage. It can be used as a package by importing it to your code or as a se

Gökhan Özeloğlu 4 Jun 15, 2022
TalariaDB is a distributed, highly available, and low latency time-series database for Presto

TalariaDB is a distributed, highly available, and low latency time-series database that stores real-time data. It's built on top of Badger DB.

Grab 104 Nov 16, 2022