In memory Key/Value store in go using gRPC.

Related tags

gRPC-cache
Overview

In memory cache, using gRPC

Build Status Coverage Status Go Report Card Documentation

Contents

About

Go in memory cache using gRPC to generate API. Functionalities include

  • Adding/Replacing key/value
  • Getting value using a key
  • Getting all keys
  • Deleting a particular key
  • Deleting all keys
  • Concurrency safe and on client side calls can be made using goroutines

Running server

You can run server locally the following ways

  • Using go run
  • Docker
  • Kubernetes
  • Helm
  • Terraform

Local

You can run the server locally but make sure you have the following requirements

Now you can start the server using below commands

Start your server `./server-cache` or `make server`
`./server-cache -addr=":5001"` to run server on port `5001`

Docker

Make sure you have following installed

After the basic requirements are met, you can easily run server from below command.

  • Run this command make docker // This will build the image
  • Run the server using make dockerServer

After running the server, start your client ./client-cache or make client in a different terminal

./client-cache -addr=":5001" is server is running running on port 5001

Kubernetes

You need to have following things installed on your OS to run server using kubernetes

Important:- Make sure your ingress-controller in running. To enable in minikube

minikube addons enable ingress

You can run server on your kubernetes cluster. All the resources are created on grpc-cache namespace. Before running the command below make sure your cluster is up and running.

  • Run this command make run-k8s-server

This will create all the required resources needed to run your grpc server. You will be asked for passowrd once, to enter ingress address to host mapping automatically to /etc/hosts

Example

# To list all services exposed by a server, use the "list" verb.
grpcurl --insecure grpc-cache.example.com:443 list

# The "describe" verb will print the type of any symbol that the server knows about or that is found in a given protoset file. It also prints a description of that symbol, in the form of snippets of proto source. It won't necessarily be the original source that defined the element, but it will be equivalent.
grpcurl --insecure grpc-cache.example.com:443 list cacheService.CacheService

# To add a key
grpcurl --insecure -d '{"key": "knrt10", "value": "pro", "expiration": "3m"}' grpc-cache.example.com:443 cacheService.CacheService/Add

# To get key
grpcurl --insecure -d '{"key": "knrt10"}' grpc-cache.example.com:443 cacheService.CacheService/Get

# To get all keys
grpcurl --insecure grpc-cache.example.com:443 cacheService.CacheService/GetAllItems

Similarly you can use all the methods as shown in API below

Helm

You need to have following things installed on your OS to run server using helm

Important:- Make sure your ingress-controller in running. To enable in minikube

minikube addons enable ingress

Just run make run-helm-server and it will deploy your application to your kubernetes cluster. This will create all the required resources needed to run your grpc server. You will be asked for passowrd once, to enter ingress address to host mapping automatically to /etc/hosts

Now you can easily test it using grpcurl. For API usage you can refer to example

Terraform

You need to have following things installed on your OS to run server using terraform

Important:- Make sure your ingress-controller in running. To enable in minikube

minikube addons enable ingress

Running with Terraform is the most easiest way. Just run make run-terraform-server and it will deploy your application to your kubernetes cluster.

Now you can easily test it using grpcurl. For API usage you can refer to example

API

Proto syntax proto3 is used. You can find the proto file here

Add

This is used to add key/value to the cache

func (c Cache) Add(ctx context.Context, item *api.Item) (*api.Item, error)

Get

This is used to get key value pair for a particular key

func (c Cache) Get(ctx context.Context, args *api.GetKey) (*api.Item, error)

GetByPrefix

Used to get all key value pairs by prefix

func (c Cache) GetByPrefix(ctx context.Context, args *api.GetKey) (*api.AllItems, error)

GetAllItems

Used to get all key value pairs

func (c Cache) GetAllItems(ctx context.Context, in *empty.Empty) (*api.AllItems, error)

DeleteKey

Used to delete item by a particular key from the cache

func (c Cache) DeleteKey(ctx context.Context, args *api.GetKey) (*api.Success, error)

DeleteAll

Used to clear the whole cache

func (c Cache) DeleteAll(ctx context.Context, in *empty.Empty) (*api.Success, error)

Testing

After running make build just run make test to run the tests. It has coverage of 92.7%

go test api/server/* -v -cover -race
=== RUN   TestAdd
--- PASS: TestAdd (0.03s)
=== RUN   TestGet
--- PASS: TestGet (0.01s)
=== RUN   TestGetByPrefix
--- PASS: TestGetByPrefix (0.01s)
=== RUN   TestGetAllItems
--- PASS: TestGetAllItems (0.01s)
=== RUN   TestDeleteKey
--- PASS: TestDeleteKey (0.00s)
=== RUN   TestDeleteAll
--- PASS: TestDeleteAll (0.00s)
=== RUN   TestGetDeletedKey
--- PASS: TestGetDeletedKey (0.01s)
=== RUN   TestDeleteKeyByExpiration
--- PASS: TestDeleteKeyByExpiration (2.01s)
PASS
coverage: 92.7% of statements
ok  	command-line-arguments	3.709s	coverage: 92.7% of statements

Development

You can develop and debug the application locally, but you need to have below softwares installed locally

Once you've cloned this repo, run these commands in this directory:

# Only needed the first time:
$ make build

# Then run to start server
$ ./server-cache --help

Usage of ./server-cache:
  -addr string
      Address on which you want to run server (default ":5001")
  -cln int
      Cleanup interval duration of expired cache is 5 min (default 5)
  -exp int
      Default expiration duration of cache is 10 min (default 10)

# To use client
$ ./client-cache --help

Usage of ./client-cache:
  -addr string
      Address on which you want to run server (default ":5001")

Example

Please refer to examples directory for more information

Issues
  • Documentation bug

    Documentation bug

    https://github.com/knrt10/gRPC-cache/blob/4136cb10cfd825cda858f2837d5cca6dec2bb017/api/server/cache_service.go#L35 says that you ignore the values if they are less than one but both of the places where you check you check if they are less than 0 AFAICT.

    opened by WesternStar 1
Owner
Kautilya Tripathi
CKA | Distributed Systems | Systems Programming | OSS ❤️
Kautilya Tripathi
A suite of gRPC debugging tools. Like Fiddler/Charles but for gRPC.

grpc-tools A suite of tools for gRPC debugging and development. Like Fiddler/Charles but for gRPC! The main tool is grpc-dump which transparently inte

Bradley Kemp 938 Jul 23, 2021
Use Consul to do service discovery, use gRPC +kafka to do message produce and consume. Use redis to store result.

目录 gRPC/consul/kafka简介 gRPC+kafka的Demo gRPC+kafka整体示意图 限流器 基于redis计数器生成唯一ID kafka生产消费 kafka生产消费示意图 本文kafka生产消费过程 基于pprof的性能分析Demo 使用pprof统计CPU/HEAP数据的

null 38 Jan 31, 2021
The Go language implementation of gRPC. HTTP/2 based RPC

gRPC-Go The Go implementation of gRPC: A high performance, open source, general RPC framework that puts mobile and HTTP/2 first. For more information

grpc 14.1k Jul 23, 2021
grpc-http1: A gRPC via HTTP/1 Enabling Library for Go

grpc-http1: A gRPC via HTTP/1 Enabling Library for Go This library enables using all the functionality of a gRPC server even if it is exposed behind a

StackRox 53 Jun 3, 2021
A Protocol Buffers compiler that generates optimized marshaling & unmarshaling Go code for ProtoBuf APIv2

vtprotobuf, the Vitess Protocol Buffers compiler This repository provides the protoc-gen-go-vtproto plug-in for protoc, which is used by Vitess to gen

PlanetScale 262 Jul 23, 2021
gRPC over WebRTC

gRPC over WebRTC Just a proof of concept, please be kind. How to Start all the things Client, create-react-app + grpc-web signaling + webrtc extension

Jean-Sebastien Mouret 227 Jul 10, 2021
In memory Key/Value store in go using gRPC.

In memory cache, using gRPC Contents About Running Server Local Docker Kubernetes Example Helm Terraform API Add Get GetByPrefix GetAllItems DeleteKey

Kautilya Tripathi 136 Jul 23, 2021
protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript clients that connect the web frontend and golang backend fronted by grpc-gateway.

protoc-gen-grpc-gateway-ts protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript

gRPC Ecosystem 25 Jul 21, 2021
gRPC/REST proxy for Kafka

Kafka-Pixy (gRPC/REST Proxy for Kafka) Kafka-Pixy is a dual API (gRPC and REST) proxy for Kafka with automatic consumer group control. It is designed

Mailgun Team 646 Jul 16, 2021
Fast time-series data storage server accessible over gRPC

tstorage-server Persistent fast time-series data storage server accessible over gRPC. tstorage-server is lightweight local on-disk storage engine serv

Bartlomiej Mika 3 Jul 17, 2021
A pluggable backend API that enforces the Event Sourcing Pattern for persisting & broadcasting application state changes

A pluggable "Application State Gateway" that enforces the Event Sourcing Pattern for securely persisting & broadcasting application state changes

null 24 Apr 15, 2021
Create a gRPC Server from Database

xo-grpc Create a gRPC Server from the generated code by the xo project. Requirements Go 1.16 or superior protoc xo, protoc-gen-go and protoc-gen-go-gr

Walter Wanderley 17 Jun 23, 2021
Create a gRPC server from code generated by sqlc

sqlc-grpc Create a gRPC Server from the generated code by the awesome sqlc project. Dependencies Go 1.16 or superior protoc sqlc, protoc-gen-go and pr

Walter Wanderley 35 Jul 8, 2021
gproxy is a tiny service/library for creating lets-encrypt/acme secured gRPC and http reverse proxies

gproxy is a reverse proxy service AND library for creating flexible, expression-based, lets-encrypt/acme secured gRPC/http reverse proxies GProxy as a

null 13 Feb 23, 2021