Fast time-series data storage server accessible over gRPC

Overview

tstorage-server

Persistent fast time-series data storage server accessible over gRPC.

tstorage-server is lightweight local on-disk storage engine server for time-series data accessible over gRPC. Run this server once and share fast time-series data CRUD operations in either local or remote applications. The purpose of this server is to allow interprocess communication overtop the tstorage package.

Installation

Get our latest code.

go install github.com/bartmika/[email protected]

Usage

To start the server, run the following command in your terminal:

$GOBIN/tstorage-server serve

That's it! If everything works, you should see a message saying gRPC server is running..

Sub-Commands Reference

serve

Details:

Run the gRPC server to allow other services to access the storage application

Usage:
  tstorage-server serve [flags]

Flags:
  -d, --dataPath string                The location to save the database files to. (default "./tsdb")
  -h, --help                           help for serve
  -b, --partitionDurationInHours int   The timestamp range inside partitions. (default 1)
  -p, --port int                       The port to run this server on (default 50051)
  -t, --timestampPrecision string      The precision of timestamps to be used by all operations. Options:  (default "s")
  -w, --writeTimeoutInSeconds int      The timeout to wait when workers are busy (in seconds). (default 30)

Example:

$GOBIN/tstorage-server serve -p=50051 -d="./tsdb" -t="s" -b=1 -w=30

insert_row

Details:

Connect to the gRPC server and sends a single time-series datum.

Usage:
  tstorage-server insert_row [flags]

Flags:
  -h, --help            help for insert_row
  -m, --metric string   The metric to attach to the TSD.
  -p, --port int        The port of our server. (default 50051)
  -t, --timestamp int   The timestamp to attach to the TSD.
  -v, --value float     The value to attach to the TSD.

Example:

$GOBIN/tstorage-server insert_row -p=50051 -m="solar_biodigester_temperature_in_degrees" -v=50 -t=1600000000

Developer Notes:

  • There also exists a insert_rows subcommand but it works exactly as insert_row command with the exception that the internal code is using streaming. This is done so programmers can look at the code and see how to use streaming of time-series data.

select

Details:

Connect to the gRPC server and return list of results based on a selection filter.

Usage:
  tstorage-server select [flags]

Flags:
  -e, --end int         The end timestamp to finish our range
  -h, --help            help for select
  -m, --metric string   The metric to filter by
  -p, --port int        The port of our server. (default 50051)
  -s, --start int       The start timestamp to begin our range

Example:

$GOBIN/tstorage-server select --port=50051 --metric="bio_reactor_pressure_in_kpa" --start=1600000000 --end=1725946120

How to Access using gRPC

What is the gRPC service definition?

Please see the tstorage.proto file for more details. Code snippet from that file:

service TStorage {
    rpc InsertRow (TimeSeriesDatum) returns (google.protobuf.Empty) {}
    rpc InsertRows (stream TimeSeriesDatum) returns (google.protobuf.Empty) {}
    rpc Select (Filter) returns (stream DataPoint) {}
}

message DataPoint {
    double value = 3;
    google.protobuf.Timestamp timestamp = 4;
}

message Label {
    string name = 1;
    string value = 2;
}

message TimeSeriesDatum {
    string metric = 1;
    repeated Label labels = 2;
    double value = 3;
    google.protobuf.Timestamp timestamp = 4;
}

message Filter {
    string metric = 1;
    repeated Label labels = 2;
    google.protobuf.Timestamp start = 3;
    google.protobuf.Timestamp end = 4;
}

message SelectResponse {
    repeated DataPoint points = 1;
}

Contributing

Development

If you'd like to setup the project for development. Here are the installation steps:

  1. Go to your development folder.

    cd ~/go/src/github.com/bartmika
  2. Clone the repository.

    git clone https://github.com/bartmika/tstorage-server.git
    cd tstorage-server
  3. Install the package dependencies

    go mod tidy
  4. In your terminal, make sure we export our path (if you haven’t done this before) by writing the following:

    export PATH="$PATH:$(go env GOPATH)/bin"
  5. Run the following to generate our new gRPC interface. Please note in your development, if you make any changes to the gRPC service definition then you'll need to rerun the following:

    protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative proto/tstorage.proto
  6. You are now ready to start the server and begin contributing!

    go run main.go serve

Quality Assurance

Found a bug? Need Help? Please create an issue.

License

BSD 3-Clause License © Bartlomiej Mika

Acknowledgement

This gRPC server is built overtop tstorage which was architected and written by Ryo Nakao.

Owner
Bartlomiej Mika
Technical Consultant
Bartlomiej Mika
A suite of gRPC debugging tools. Like Fiddler/Charles but for gRPC.

grpc-tools A suite of tools for gRPC debugging and development. Like Fiddler/Charles but for gRPC! The main tool is grpc-dump which transparently inte

Bradley Kemp 938 Jul 23, 2021
Use Consul to do service discovery, use gRPC +kafka to do message produce and consume. Use redis to store result.

目录 gRPC/consul/kafka简介 gRPC+kafka的Demo gRPC+kafka整体示意图 限流器 基于redis计数器生成唯一ID kafka生产消费 kafka生产消费示意图 本文kafka生产消费过程 基于pprof的性能分析Demo 使用pprof统计CPU/HEAP数据的

null 38 Jan 31, 2021
The Go language implementation of gRPC. HTTP/2 based RPC

gRPC-Go The Go implementation of gRPC: A high performance, open source, general RPC framework that puts mobile and HTTP/2 first. For more information

grpc 14.1k Jul 23, 2021
grpc-http1: A gRPC via HTTP/1 Enabling Library for Go

grpc-http1: A gRPC via HTTP/1 Enabling Library for Go This library enables using all the functionality of a gRPC server even if it is exposed behind a

StackRox 53 Jun 3, 2021
Fast time-series data storage server accessible over gRPC

tstorage-server Persistent fast time-series data storage server accessible over gRPC. tstorage-server is lightweight local on-disk storage engine serv

Bartlomiej Mika 3 Jul 17, 2021
Application written in Go which polls Time-series data at specific intervals and saves to persistent storage

TPoller Server Overview The purpose of this application is to poll time-series data per time interval from any (telemetry) application running a gRPC

Bartlomiej Mika 3 Jul 17, 2021
gRPC over WebRTC

gRPC over WebRTC Just a proof of concept, please be kind. How to Start all the things Client, create-react-app + grpc-web signaling + webrtc extension

Jean-Sebastien Mouret 227 Jul 10, 2021
protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript clients that connect the web frontend and golang backend fronted by grpc-gateway.

protoc-gen-grpc-gateway-ts protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript

gRPC Ecosystem 25 Jul 21, 2021
In memory Key/Value store in go using gRPC.

In memory cache, using gRPC Contents About Running Server Local Docker Kubernetes Example Helm Terraform API Add Get GetByPrefix GetAllItems DeleteKey

Kautilya Tripathi 136 Jul 23, 2021
A Protocol Buffers compiler that generates optimized marshaling & unmarshaling Go code for ProtoBuf APIv2

vtprotobuf, the Vitess Protocol Buffers compiler This repository provides the protoc-gen-go-vtproto plug-in for protoc, which is used by Vitess to gen

PlanetScale 262 Jul 23, 2021
gRPC/REST proxy for Kafka

Kafka-Pixy (gRPC/REST Proxy for Kafka) Kafka-Pixy is a dual API (gRPC and REST) proxy for Kafka with automatic consumer group control. It is designed

Mailgun Team 646 Jul 16, 2021
A pluggable backend API that enforces the Event Sourcing Pattern for persisting & broadcasting application state changes

A pluggable "Application State Gateway" that enforces the Event Sourcing Pattern for securely persisting & broadcasting application state changes

null 24 Apr 15, 2021
Create a gRPC Server from Database

xo-grpc Create a gRPC Server from the generated code by the xo project. Requirements Go 1.16 or superior protoc xo, protoc-gen-go and protoc-gen-go-gr

Walter Wanderley 17 Jun 23, 2021
Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator

Trickster is an HTTP reverse proxy/cache for http applications and a dashboard query accelerator for time series databases. Learn more below, and chec

null 1.5k Jul 16, 2021