a persistent real-time key-value store, with the same redis protocol with powerful features

Overview

Build Status License Docker Contributors

a fast NoSQL DB, that uses the same RESP protocol and capable to store terabytes of data, also it integrates with your mobile/web apps to add real-time features, soon you can use it as a document store cause it should become a multi-model db. Redix is used in production, you can use it in your apps with no worries.

Features

  • Core data structure: KV, List, Hashmap with advanced implementations.
  • Advanced Publish/Subscribe using webhook and websocket!
  • Pluggable Storage Engine (badgerdb, boltdb, leveldb, null)
  • Very compatible with any redis client including redis-cli
  • Standalone with no external dependencies
  • Helpers commands for Time, Encode , RANDINT, RANDSTR
  • Implements RATELIMIT helpers natively.

Why

I started this software to learn more about data modeling, data structures and how to map any data to pure key value, I don't need to build a redis clone, but I need to build something with my own concepts in my own style. I decided to use RESP (redis protocol) so you can use Redix with any redis client out there.

Install

  • Using Homebrew:
    • Add Homebrew Tap brew tap alash3al/redix https://github.com/alash3al/redix
    • Install Redix brew install alash3al/redix/redix
  • From Binaries: go there and choose your platform based binary, then download and execute from the command line with -h flag to see the help text.
  • Using Docker: docker run -P -v /path/to/redix-data:/root/redix-data alash3al/redix
  • From Source: go get github.com/alash3al/redix.

Configurations

It is so easy to configure Redix, there is no configuration files, it is all about running ./redix after you download it from the releases, if you downloaded i.e 'redix_linux_amd64' and unziped it.

$ ./redix_linux_amd64 -h

  -engine string
        the storage engine to be used, available (default "leveldb")
  -http-addr string
        the address of the http server (default ":7090")
  -resp-addr string
        the address of resp server (default ":6380")
  -storage string
        the storage directory (default "./redix-data")
  -verbose
        verbose or not
  -workers int
        the default workers number (default ...)

Examples

# i.e: $mykey1 = "this is my value"
$ redis-cli -p 6380 set mykey1 "this is my value"

# i.e: $mykey1 = "this is my value" and expire it after 10 seconds
$ redis-cli -p 6380 set mykey1 "this is my value" 10000

# i.e: echo $mykey1
$ redis-cli -p 6380 get mykey1

# i.e: $mymap1[x] = y
$ redis-cli -p 6380 hset mymap1 x y

# i.e: $mymap1[x] = y and expires it after 10 seconds
$ redis-cli -p 6380 hset mymap1 x y 10000

# i.e: sha512 of "test"
$ redis-cli -p 6380 encode sha512 test

# you want to notify an endpoint i.e: "http://localhost:800/new-data" that there is new data available, in other words, you want to subscribe a webhook to channel updates.
$ redis-cli -p 6380 webhookset testchan http://localhost:800/new-data

# add data to a list
# i.e: [].push(....)
$ redis-cli -p 6380 lpush mylist1 "I'm Mohammed" "I like to Go using Go" "I love coding"

# search in the list
$ redis-cli -p 6380 lsrch mylist1 "mo(.*)"

DB Engines

  • Redix supports two engines called badger and bolt
  • badger is the default, it is inspired by Facebook RocksDB, it meant to be fast on-disk engine, read more
  • bolt is our alternate engine, it is inspired by LMDB, read more

Supported Commands

Redix doesn't implement all redis commands, but instead it supports the core concepts that will help you to build any type of data models on top of it, there are more commands and features in all next releases.

# Basic

  • PING
  • QUIT
  • SELECT

# Strings

  • SET []
  • MSET [ ...]
  • GET []
  • MGET [ ...]
  • DEL [ ...]
  • EXISTS
  • INCR []
  • TTL returns -1 if key will never expire, -2 if it doesn't exists (expired), otherwise will returns the seconds remain before the key will expire.
  • KEYS []

# HASHES

I enhanced the HASH MAP implementation and added some features like TTL per nested key, also you can check whether the hash map itself exists or not using HEXISTS or a nested key exists using HEXISTS .

  • HSET []
  • HMSET [ ...]
  • HGET
  • HDEL [ ...] (deletes the map itself or keys in the map)
  • HGETALL
  • HMSET [ ...]
  • HEXISTS [].
  • HINCR []
  • HTTL , the same as TTL but for HASHMAP
  • HKEYS
  • HLEN

# LIST

I applied a new concept, you can push or push-unique values into the list, based on that I don't need to implement two different data structures, as well as, you can quickly iterate over a list in a high performance way, every push will return the internal offset of the value, also, the iterator lrange will tell you the next offset you can start from.

  • LPUSH [ ...] (push the item into the list "it doesn't check for uniqueness, it will append anyway (duplicate)")
  • LPUSHU [ ...] (push the item into the list only if it isn't exists)
  • LRANGE [ ]
  • LREM [ ...] (deletes the list itself or values in the list)
  • LCOUNT (get the list members count)
  • LCARD (alias of LCOUNT)
  • LSUM (sum the members of the list "in case they were numbers")
  • LAVG (get the avg of the members of the list "in case they were numbers")
  • LMIN (get the minimum of the members of the list "in case they were numbers")
  • LMAX (get the maximum of the members of the list "in case they were numbers")
  • LSRCH (text-search using (string search or regex) in the list)
  • LSRCHCOUNT (size of text-search result using (string search or regex) in the list)

# SET

  • SADD [ ...] (alias of LUPUSH)
  • SMEMBERS [ ] (alias of LRANGE)
  • SSCAN [ ] (alias of LRANGE)
  • SCARD (aliad of LCOUNT)
  • SREM [ ...] (alias of LREM)

# Pub/Sub

Redix has very simple pub/sub functionality, you can subscribe to internal logs on the * channel or any custom defined channel, and publish to any custom channel.

  • SUBSCRIBE [ ], if there is no channel specified, it will be set to *
  • PUBLISH
  • WEBHOOKSET , register a http endpoint so it can be notified through JSON POST request with the channel updates, this command will return a reference ID so you can manage it later.
  • WEBHOOKDEL , stops listening on a channel using the above reference ID.
  • WEBSOCKETOPEN , opens a websocket endpoint and returns its id, so you can receive updates through ws://server.address:port/stream/ws/{generated_id_here}
  • WEBSOCKETCLOSE , closes the specified websocket endpoint using the above generated id.

# Ratelimit

  • RATELIMITSET , create a new $bucket that accepts num of $limit of actions per the specified num of $seconds, it will returns 1 for success.
  • RATELIMITTAKE , do an action in the specified bucket and take an item from it, it will return -1 if the bucket not exists or it has unlimited actions $limit < 1, 0 if there are no more actions to be done right now, reminder of actions on success.
  • RATELIMITGET , returns array [$limit, $seconds, $remaining_time, $counter] information for the specified bucket

# Utils

some useful utils that you can use within your app to remove some hassle from it.

  • ENCODE , encode the specified using the specified (md5, sha1, sha256, sha512, hex)
  • UUIDV4, generates a uuid-v4 string, i.e 0b98aa17-eb06-42b8-b39f-fd7ba6aba7cd.
  • UNIQID, generates a unique string.
  • RANDSTR [, default size is 10], generates a random string using the specified length.
  • RANDINT , generates a random string between the specified and .
  • TIME, returns the current time in utc, seconds and nanoseconds
  • DBSIZE, returns the database size in bytes.
  • GC, runs the Garbage Collector.
  • ECHO [ ...]
  • INFO
  • FLUSHDB, clear the database
  • FLUSHALL, clear all databases

TODO

  • Basic Commands
  • Strings Commands
  • Hashmap Commands
  • List Commands
  • PubSub Commands
  • Utils Commands
  • Adding BoltDB engine
  • Adding LevelDB engine
  • Adding Null engine
  • Adding RAM engine
Comments
  • No permissions to pull

    No permissions to pull

    docker pull ghcr.io/alash3al/redix                                                                                               ✘  15:19:47
    Using default tag: latest
    Error response from daemon: unauthorized
    

    maybe make it publicly readable?

    opened by ficapy 12
  • availability of v1 branch?

    availability of v1 branch?

    Hi I am actually still keen on using version 1 of this repo which had some very useful features, but it seems that branch is no longer available? Is it possible to keep that version available somewhere else? I noted that docker pull alash3al/redix does pull the old version and so docker pull ghcr.io/alash3al/redix is needed for v5

    opened by c-nv-s 8
  • benchmark

    benchmark

    redis-benchmark -n 1000000 -t set,get -P 16 -q

    Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz 2 Core 4G RAM SSDB Disk

    redis-5.0.3 SET: 548245.62 requests per second GET: 671140.94 requests per second

    redix V1.7 SET: 6992.27 requests per second GET: 188323.92 requests per second

    redix V1.8 (use leveldb -engine=level) SET: 166334.00 requests per second GET: 280190.53 requests per second

    ssdb 1.9.7 SET: 34694.52 requests per second GET: 35812.77 requests per second

    opened by RegulusStar 7
  • Too many open files

    Too many open files

    I use "predis/predis": "^1.1" with PHP and redix as an IP rate limiter. Today my server sent ERROR 500, but redix was still alive. So i looked at the web server error log. Here is what I got :

    www[5]: PHP Fatal error:
    Uncaught Predis\Response\ServerException: db error: 
    open redix-data/level/0/CURRENT: 
    too many open files in /flex/storage/xxx/php/vendor/predis/predis/src/Client.php:370
    #012
    Stack trace:
    #012#0
    /flex/storage/xxx/php/vendor/predis/predis/src/Client.php(335): 
    Predis\Client->onErrorResponse(Object(Predis\Command\KeyTimeToLive), 
    Object(Predis\Response\Error))
    #012#1
    /flex/storage/xxx/php/vendor/predis/predis/src/Client.php(314):
    Predis\Client->executeCommand(Object(Predis\Command\KeyTimeToLive))
    #012#2
    /flex/storage/xxx/php/myapp/class.ratelimit.redix.php(72): 
    Predis\Client->__call('ttl', Array)
    #012#3
    /flex/storage/xxx/php/myapp/script.php(61):
    RateLimit->getSleepTime($ipaddr)
    #012#4
    {main}
    #012
    thrown in /flex/storage/xxx/php/vendor/predis/predis/src/Client.php on line 370
    

    To solve the issue, I stopped redix, then I did rm -rf redix-data and started it again. I'm not sure about what was wrong here. If it helps.

    opened by Tazeg 6
  • the release 1.8 version label show error in cli

    the release 1.8 version label show error in cli

    the release 1.8 version label show error in cli redix server version: 1.7

    MacBookPro 2018 16G Mem 512G ssd redis-benchmark -n 1000000 -t set,get -P 16 -p 6380 -q SET: 323519.88 requests per second GET: 1078748.62 requests per second

    very good

    Originally posted by @withu2018 in https://github.com/alash3al/redix/issue_comments#issuecomment-451419162

    opened by withu2018 5
  • `go build` raise errors

    `go build` raise errors

    When I run go build in the project, it raises errors:

    # github.com/alash3al/redix/kvstore/bolt
    kvstore/bolt/bolt.go:13:2: imported and not used: "go.etcd.io/bbolt" as bolt
    kvstore/bolt/bolt.go:18:17: undefined: bbolt
    
    # github.com/alash3al/redix
    ./commands_utils.go:23:10: assignment mismatch: 2 variables but 1 values
    ./init.go:27:3: undefined: "github.com/dgraph-io/badger".SetLogger
    

    So looks like we have 3 issues here:

    1. go.etcd.io/bbolt imported as bolt(defined in the package), but never used. Considering kvstore/bolt/bolt.go package is also named bolt, we rename it to 'bbolt'
    2. satori/go.uuid v1.2.0 uuid.NewV4() returns only one value
    3. dgraph-io/badger v1.5.4 does not contains the method SetLogger, remove it.
    opened by lylex 4
  • the first data stored in list is not correct data.

    the first data stored in list is not correct data.

    the first data in list by lpush is not correct! It like this belllow. image

    this is my correct data . image

    is this the right way i used lpush. I really don't konw.

    opened by kekek 3
  • command line params change

    command line params change

    1. can't find the version cmd 2.-engine default leveldb, at version 1.8 ,the string is level or badger image

    3.口 image

    1. in php7.1.27 [get] cmd error. the "+" before returning the data image
    opened by withu2018 3
  • what's the params of lrange

    what's the params of lrange

    when i run command bellow in termnal : lrange f10.data.4:600030 0 0 it return "(error) invalid offset specified"

    is there any example or document for this command.

    opened by kekek 2
  • Possible to add INFO command?

    Possible to add INFO command?

    Hi, lot of thanks about your work! I try to use redix with NodeJS app and using popular client module, node_redis. But his architecture needed to use INFO command when obtain client connection. And we cant used it with Redix:

    Redix error! { ReplyError: Ready check failed: unknown commands [info] at parseError (/opt/tendermint/app/node_modules/redis-parser/lib/parser.js:193:12) at parseType (/opt/tendermint/app/node_modules/redis-parser/lib/parser.js:303:14) command: 'INFO' }

    Can you add this command with base support of redis-compartability?

    opened by aleksraiden 2
  • Possible config file

    Possible config file

    Hi Mohammed Al Ashaal,

    I wanted to thank you for writing this awesome layer over Badger KV store! Is it possible to have an optional config file? I would like to create .deb and .rpm packages for this great tool so people can easily install and upgrade it.

    Thanks again for building this!

    opened by VvanGemert 2
Releases(v5.2.0)
Owner
Mohammed Al Ashaal
a software engineer 🤓
Mohammed Al Ashaal
NutsDB a simple, fast, embeddable and persistent key/value store written in pure Go.

A simple, fast, embeddable, persistent key/value store written in pure Go. It supports fully serializable transactions and many data structures such as list, set, sorted set.

徐佳军 2.6k Nov 28, 2022
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Peter Bourgon 1.2k Nov 30, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 41.9k Nov 27, 2022
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Burak Sezer 2.6k Nov 30, 2022
GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

Jake Grogan 729 Nov 13, 2022
Pogreb is an embedded key-value store for read-heavy workloads written in Go.

Embedded key-value store for read-heavy workloads written in Go

Artem Krylysov 963 Nov 24, 2022
CrankDB is an ultra fast and very lightweight Key Value based Document Store.

CrankDB is a ultra fast, extreme lightweight Key Value based Document Store.

Shrey Batra 30 Apr 12, 2022
yakv is a simple, in-memory, concurrency-safe key-value store for hobbyists.

yakv (yak-v. (originally intended to be "yet-another-key-value store")) is a simple, in-memory, concurrency-safe key-value store for hobbyists. yakv provides persistence by appending transactions to a transaction log and restoring data from the transaction log on startup.

Aadhav Vignesh 5 Feb 24, 2022
Multithreaded key value pair store using thread safe locking mechanism allowing concurrent reads

Project Amnesia A Multi-threaded key-value pair store using thread safe locking mechanism allowing concurrent reads. Curious to Try it out?? Check out

Nikhil Nayak 7 Oct 29, 2022
ShockV is a simple key-value store with RESTful API

ShockV is a simple key-value store based on badgerDB with RESTful API. It's best suited for experimental project which you need a lightweight data store.

delihiros 2 Sep 26, 2021
A rest-api that works with golang as an in-memory key value store

In Store A rest-api that works with golang as an in-memory key value store Usage Fist of all, clone the repo with the command below. You must have gol

Eyüp Arslan 0 Oct 24, 2021
Distributed key-value store

Keva Distributed key-value store General Demo Start the server docker-compose up --build Insert data curl -XPOST http://localhost:5555/storage/test1

Yaroslav Gaponov 0 Nov 15, 2021
Simple in memory key-value store.

Simple in memory key-value store. Development This project is written in Go. Make sure you have Go installed (download). Version 1.17 or higher is req

Mustafa Navruz 0 Nov 6, 2021
A simple in-memory key-value store application

vtec vtec, is a simple in-memory key-value store application. vtec provides persistence by appending transactions to a json file and restoring data fr

Ahmet Tek 3 Jun 22, 2022
Biscuit is a multi-region HA key-value store for your AWS infrastructure secrets.

Biscuit Biscuit is a simple key-value store for your infrastructure secrets. Is Biscuit right for me? Biscuit is most useful to teams already using AW

Doug 557 Nov 10, 2022
An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.

go-cache go-cache is an in-memory key:value store/cache similar to memcached that is suitable for applications running on a single machine. Its major

Patrick Mylund Nielsen 6.7k Nov 30, 2022
KV - a toy in-memory key value store built primarily in an effort to write more go and check out grpc

KV KV is a toy in-memory key value store built primarily in an effort to write more go and check out grpc. This is still a work in progress. // downlo

Ali Mir 0 Dec 30, 2021
PrimeKV is a Secure, REST API driven Key/Value store.

PrimeKV PrimeKV is a Secure, REST API driven Key/Value store. Features 100% In-memory. Encrypted by default, All stored values are bi-directionally en

Josh Burns 0 Jan 10, 2022
Dokvs - This project aims to provide document-store capabilities, built over key-value persisted backends

This project aims to provide document-store capabilities, built over key-value persisted backends. It is built in Go using the new generics features to expose a friendly, type-safe API.

George 2 Apr 25, 2022