GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

Overview

GhostDB logo

Build Twitter

Discord

📚 Overview

GhostDB is a distributed, in-memory, general purpose key-value data store that delivers microsecond performance at any scale.

GhostDB is designed to speed up dynamic database or API driven websites by storing data in RAM in order to reduce the number of times an external data source such as a database or API must be read. GhostDB provides a very large hash table that is distributed across multiple machines and stores large numbers of key-value pairs within the hash table.

🚗 Roadmap

GhostDB was a university project - it is not fully featured but we're getting there!

This is a high-level roadmap of what we want GhostDB to become by the end of 2020. If you have any feature requests please create one from the template and label it as feature request!

  • First hand support for list, set, stack and queue data structures
  • Atomic command queues
  • Subscribable streams
  • Monitoring & administration dashboard
  • Enhanced security features
  • Transition to TCP sockets as transport protocol
  • CLI
  • Support for a wide range of programming languages

🔧 Installation

To install GhostDB please consult the installation guide for a quick walkthrough on setting up the system.

🔨 Cluster Configuration

To configure a GhostDB cluster please follow the instructions in the configuration guide

✏️ Authors

Jake Grogan

Connor Mulready

Show your support

Give a if this project helped you!

Comments
  • Use YAML as default config format

    Use YAML as default config format

    We and our users currently write our configs in JSON. We want to move to the YAML format for writing configs.

    To help you get started, the following files will need updating (some other files may require changes):

    • config_reader.go
    • config_reader_test.go
    • INSTALLATION.md
    good first issue 
    opened by jakekgrog 2
  • Please add a license to this repo

    Please add a license to this repo

    First, thank you for sharing this project with us!

    Could you please add an explicit LICENSE file to the repo so that it's clear under what terms the content is provided, and under what terms user contributions are licensed?

    Per GitHub docs on licensing:

    [...] without a license, the default copyright laws apply, meaning that you retain all rights to your source code and no one may reproduce, distribute, or create derivative works from your work. If you're creating an open source project, we strongly encourage you to include an open source license.

    Thanks!

    opened by mbrukman 2
  • GhostDB 2.0

    GhostDB 2.0

    GhostDB 2.0

    The structure and implementations of parts of GhostDB has changed in order to make it more maintainable, scalable and reliable.

    This is not the final version of GhostDB 2.0. Ignore commented files, these are parts I need to either remove or clean up.

    opened by jakekgrog 1
  • Add handler to get leader node

    Add handler to get leader node

    As this system is a leader-follower system, we need some way of determining which node in the cluster is the leader. This PR addresses this by adding a handler on the server to return the leader node IP address.

    opened by jakekgrog 0
  • Added API to get the number of clients connected to Node

    Added API to get the number of clients connected to Node

    #24 Added \getActiveConnections to get the number of active client connections to a node. Created a map with client's IP address as key and value as active number of requests. When the number of requests corresponding to the client equals 0 , the key is removed. The length of the map corresponds to the number of active client connections.

    opened by Adm28 0
  • Add support for Queue data type

    Add support for Queue data type

    Changes

    • Added basic Queue implementation
    • Added Enqueue() and Dequeue() to Cache interface
    • Added Enqueue() and Dequeue() to LRU

    Tests

    • Added tests for queue methods on store
    • go test ./... passing

    Issue

    https://github.com/jakekgrog/GhostDB/issues/21

    opened by jakekgrog 0
  • Tidb CDC example integration

    Tidb CDC example integration

    GhostDB would be a great match for databases that support CDC

    You could use GhostDB to hold your data in a cdb network like cloudflare as a materialised view that is fed from tidb.

    Here is an example that feeds data from tidb to Kafka to get an idea : https://github.com/pingcap/ticdc/tree/master/kafka_consumer.

    GhostDB would then act as the read only dB and TiDb cluster as the write dB.

    This would be amazingly efficient I think.

    opened by joe-getcouragenow 4
  • Data type metadata should be included in cache node objects

    Data type metadata should be included in cache node objects

    Now that we support more than 1 data type, we should include the values data type in cache nodes. This give us a way of easily identifying what type of data a key points to e.g. string, queue (and more types in the future)

    Currently a cache node object consists of a Key, a Value and a TTL. We should also add a field specifying the type of the Value

    enhancement good first issue 
    opened by jakekgrog 3
  • Keyspace analysis

    Keyspace analysis

    We would like a way for our users to be able to perform analysis of the keyspace of a node. This would require keeping track of keys that are frequently used (keys with high number of hits; we need to define what constitutes a high number of hits, perhaps just the top x keys can be returned), the longest keys, shortest keys etc.

    We would also like to be able to analyse the largest items in the cache. For example, a user might want to know the largest queue on the node, the longest string, largest set etc.

    Again, this is up for discussion and I'm interested in hearing your thoughts!

    discussion 
    opened by jakekgrog 0
  • Request history

    Request history

    We would like to be able to create a log of recent request history for the node (the last hour for example). These logs would contain information about what commands were executed, who the request came from (what client). Whether to request failed or succeeded etc. It is open for discussion here so please contribute if you've any good ideas!

    discussion 
    opened by jakekgrog 4
Owner
Jake Grogan
Software Development Engineer @aws | AWS DynamoDB
Jake Grogan
Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service.

Olric Distributed cache and in-memory key/value data store. It can be used both as an embedded Go library and as a language-independent service. With

Burak Sezer 2.3k Sep 22, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The master branch may be in an unstable or even broken state during development. Please use releases instead of the master branch in order

etcd-io 41.3k Sep 26, 2022
yakv is a simple, in-memory, concurrency-safe key-value store for hobbyists.

yakv (yak-v. (originally intended to be "yet-another-key-value store")) is a simple, in-memory, concurrency-safe key-value store for hobbyists. yakv provides persistence by appending transactions to a transaction log and restoring data from the transaction log on startup.

Aadhav Vignesh 5 Feb 24, 2022
A rest-api that works with golang as an in-memory key value store

In Store A rest-api that works with golang as an in-memory key value store Usage Fist of all, clone the repo with the command below. You must have gol

Eyüp Arslan 0 Oct 24, 2021
Simple in memory key-value store.

Simple in memory key-value store. Development This project is written in Go. Make sure you have Go installed (download). Version 1.17 or higher is req

Mustafa Navruz 0 Nov 6, 2021
A simple in-memory key-value store application

vtec vtec, is a simple in-memory key-value store application. vtec provides persistence by appending transactions to a json file and restoring data fr

Ahmet Tek 3 Jun 22, 2022
An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.

go-cache go-cache is an in-memory key:value store/cache similar to memcached that is suitable for applications running on a single machine. Its major

Patrick Mylund Nielsen 6.5k Sep 21, 2022
KV - a toy in-memory key value store built primarily in an effort to write more go and check out grpc

KV KV is a toy in-memory key value store built primarily in an effort to write more go and check out grpc. This is still a work in progress. // downlo

Ali Mir 0 Dec 30, 2021
kvStore is a simple key/value in-memory store

kvStore is a simple key/value in-memory store. It is designed for the API. kvStore keeps records at /tmp/kvStore/dbName.db. You can specify server port, dbName and, file save interval in your RunServer(Addr, dbName) call.

Buğra Mengi 2 Feb 24, 2022
Simple Distributed key-value database (in-memory/disk) written with Golang.

Kallbaz DB Simple Distributed key-value store (in-memory/disk) written with Golang. Installation go get github.com/msam1r/kallbaz-db Usage API // Get

Mohamed Samir 5 Jan 18, 2022
Turn any key/value index into a high-performance two-dimensional spatial index

modular-spatial-index For the demo that this animated gif was generated from

SequentialRead.com 5 Mar 6, 2022
Distributed key-value store

Keva Distributed key-value store General Demo Start the server docker-compose up --build Insert data curl -XPOST http://localhost:5555/storage/test1

Yaroslav Gaponov 0 Nov 15, 2021
High-performance, columnar, in-memory store with bitmap indexing in Go

This package contains a high-performance, columnar, in-memory storage engine that supports fast querying, update and iteration with zero-allocations and bitmap indexing.

Roman Atachiants 982 Sep 15, 2022
Key-value database stored in memory with option of persistence

Easy and intuitive command line tool allows you to spin up a database avaliable from web or locally in a few seconds. Server can be run over a custom TCP protocol or over HTTP.

Mario Petričko 6 Aug 1, 2022
ZedisDB - a key-value memory database written in Go

ZedisDB - a key-value memory database written in Go

Znkisoft 4 Sep 4, 2022
FlashDB is an embeddable, in-memory key/value database in Go

FlashDB is an embeddable, in-memory key/value database in Go (with Redis like commands and super easy to read)

Farhan 266 Sep 29, 2022
A disk-backed key-value store.

What is diskv? Diskv (disk-vee) is a simple, persistent key-value store written in the Go language. It starts with an incredibly simple API for storin

Peter Bourgon 1.2k Sep 26, 2022
a persistent real-time key-value store, with the same redis protocol with powerful features

a fast NoSQL DB, that uses the same RESP protocol and capable to store terabytes of data, also it integrates with your mobile/web apps to add real-time features, soon you can use it as a document store cause it should become a multi-model db. Redix is used in production, you can use it in your apps with no worries.

Mohammed Al Ashaal 1.1k Sep 24, 2022
Pogreb is an embedded key-value store for read-heavy workloads written in Go.

Embedded key-value store for read-heavy workloads written in Go

Artem Krylysov 936 Sep 18, 2022