Application written in Go which polls Time-series data at specific intervals and saves to persistent storage

Overview

TPoller Server

Overview

The purpose of this application is to poll time-series data per time interval from any (telemetry) application running a gRPC server which implements the the following gRPC service definition:

service Telemetry {
    rpc GetTimeSeriesData (google.protobuf.Empty) returns (stream TelemetryDatum) {}
}

message TelemetryLabel {
    string name = 1;
    string value = 2;
}

message TelemetryDatum {
    string metric = 1;
    repeated TelemetryLabel labels = 2;
    double value = 3;
    google.protobuf.Timestamp timestamp = 4;
}

Your server which implemented the gRPC service definition is called a data reader. tpoller-server will then send your polled data over gRPC to a fast time-series data storage server called tstorage-server.

Installation

Get our latest code.

go install github.com/bartmika/[email protected]

Usage

Run our application.

$GOBIN/tpoller-server serve --telemetry_addr="127.0.0.1:50051" --storage_addr="127.0.0.1:50052"

If the server successfully starts you should see a message in your termnal:

2021/07/15 00:51:00 Storage connected
2021/07/15 00:51:00 Telemeter connected
2021/07/15 00:51:00 Synching with local time...
2021/07/15 00:52:00 Synchronized with local time.
2021/07/15 00:52:00 TPoller is now running.

More sub-command details:

Run the tpoller service in the foreground which will periodically call the "data reader" to retrieve time-series data and save it to "tstorage" server.

Usage:
  tpoller-server serve [flags]

Flags:
  -h, --help                    help for serve
  -o, --storage_addr string     The time-series data storage gRPC server address. (default "localhost:50051")
  -i, --telemetry_addr string   The telemetry gRPC server address with the 'data reader'. (default "localhost:50052")

Used by:

This server is confirmed to successfully poll from the following application(s):

If you'd like to add your app, please create a pull request with a link to your app.

Contributing

Development

If you'd like to setup the project for development. Here are the installation steps:

  1. Go to your development folder.

    cd ~/go/src/github.com/bartmika
  2. Clone the repository.

    git clone https://github.com/bartmika/tpoller-server.git
    cd tpoller-server
  3. Install the package dependencies

    go mod tidy
  4. In your terminal, make sure we export our path (if you haven’t done this before) by writing the following:

    export PATH="$PATH:$(go env GOPATH)/bin"
  5. Run the following to generate our new gRPC interface. Please note in your development, if you make any changes to the gRPC service definition then you'll need to rerun the following:

    protoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative proto/telemetry.proto
  6. You are now ready to start the server and begin contributing!

    go run main.go serve --telemetry_addr="127.0.0.1:50051" --storage_addr="127.0.0.1:50052"

Quality Assurance

Found a bug? Need Help? Please create an issue.

License

This application is licensed under the BSD 3-Clause License. See LICENSE for more information.

Owner
Bartlomiej Mika
Technical Consultant
Bartlomiej Mika
Open Source HTTP Reverse Proxy Cache and Time Series Dashboard Accelerator

Trickster is an HTTP reverse proxy/cache for http applications and a dashboard query accelerator for time series databases. Learn more below, and chec

null 1.5k Jul 16, 2021
Fast time-series data storage server accessible over gRPC

tstorage-server Persistent fast time-series data storage server accessible over gRPC. tstorage-server is lightweight local on-disk storage engine serv

Bartlomiej Mika 3 Jul 17, 2021
V3IO Frames ("Frames") is a Golang based remote data frames access (over gRPC or HTTP stream)

V3IO Frames ("Frames") is a multi-model open-source data-access library that provides a unified high-performance DataFrame API for working with different types of data sources (backends). The library was developed by Iguazio to simplify working with data in the Iguazio Data Science Platform ("the platform"), but it can be extended to support additional backend types.

null 22 Jul 12, 2021
A pluggable backend API that enforces the Event Sourcing Pattern for persisting & broadcasting application state changes

A pluggable "Application State Gateway" that enforces the Event Sourcing Pattern for securely persisting & broadcasting application state changes

null 24 Apr 15, 2021
A Filecoin Network sidecar for miners to bid in storage deal auctions.

bidbot Bidbot is a Filecoin Network sidecar for miners to bid in storage deal auctions. Join us on our public Slack channel for news, discussions, and

textile.io 9 Jul 23, 2021
HTTP rate limiting module for Caddy 2

This module implements both internal and distributed HTTP rate limiting. Requests can be rejected after a specified rate limit is hit.

Matt Holt 48 Jul 7, 2021
Package for downloading things from a string URL using a variety of protocols.

go-getter is a library for Go (golang) for downloading files or directories from various sources using a URL as the primary form of input.

HashiCorp 1.2k Jul 16, 2021
🌌 A libp2p DHT crawler that gathers information about running nodes in the network.

A libp2p DHT crawler that gathers information about running nodes in the network. The crawler runs every 30 minutes by connecting to the standard DHT bootstrap nodes and then recursively following all entries in the k-buckets until all peers have been visited.

Dennis Trautwein 46 Jul 20, 2021
A SimpleHTTPServer written in Go, enhanced with features and with a nice design

goshs is a replacement for Python's SimpleHTTPServer. It allows uploading and downloading via HTTP/S with either self-signed certificate or user provi

Patrick Hener 74 May 22, 2021
Centralized Configuration System written in Golang - Spring cloud compatible

Centralized Configuration System What is Vecosy Vecosy is a configuration service exposed through REST/GRPC. Is Spring Cloud Conf compatible and also

null 18 Jun 19, 2021
Deece is an open, collaborative, and decentralised search mechanism for IPFS

Deece Deece is an open, collaborative, and decentralised search mechanism for IPFS. Any node running the client is able to crawl content on IPFS and a

null 3 Jul 18, 2021
Go binding for gammu

gogammu is binding for SMS related functions of libGammu (documentation). gogammu/smsd is simple, MySQL based, SMS daemon, written entirely in Go (it

Michał Derkacz 14 Dec 24, 2020
OpenDILab RL Object Store

Introduction Decision AI Store Installation Prerequisites Linux Python >= 3.6 pip install . Quick Start Start Etcd Server di_store etcd_server ./conf/

null 31 Jul 18, 2021