Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time

Related tags

Network basenine
Overview

Basenine

GitHub Latest Release GitHub License GitHub Workflow Tests Code Coverage (Codecov)

Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time.

Featured Aspects

  • Has the fastest possible write speed.
  • Has a read speed that scales linearly.
  • Schema-free.
  • Only allows create and read.
  • Accepts JSON as the record format.
  • Let's you query based on JSONPath.
  • Has a rich filtering syntax for querying.
  • Defines a TCP-based protocol.
  • Has long lasting TCP connections.
  • Watches the database and streams back the new records.

Server

Run the server:

make && ./basenine -port 9099

Protocol

The database server has six modes:

  • Insert mode provies a long lasting TCP connection to insert data into the data_*.bin binary files on server's directory. A client can elevate itself to insert mode by sending /insert command.

  • Query mode let's you filter the records in the database based on a filtering syntax named BFL. Query mode streams the results to the client and is able to keep up where it left off even if the database have millions of records. The TCP connection in this mode is long lasting as well. The filter cannot be changed without establishing a new connnection.

  • Single mode is a short lasting TCP connection that returns a single record from the database based on the provided index value.

  • Validate mode checks the query against syntax errors. Returns the error if it's syntactically invalid otherwise returns OK.

  • Macro mode let's you define a macro for the query language like http~proto.name == "http".

  • Limit mode allows you to set a hard-limit for the database size in bytes like 100000000 (100MB). The disk usage ranges between 50000000 (50MB) and 100000000 (100MB). So the actual effective limit is the half of this value.

Query

Querying achieved through a filter syntax named Basenine Filter Language (BFL). It enables the user to query the traffic logs efficiently and precisely.

42 or request.headers["x"] == "y") ">
http and request.method == "GET" and request.path != "/example" and (request.query.a > 42 or request.headers["x"] == "y")

Please see the syntax reference for more info.

Client

Go

Insert:

// Establish a new connection to a Basenine server at localhost:9099
c, err := NewConnection("localhost", "9099")
if err != nil {
    panic(err)
}

// Elevate to INSERT mode
c.InsertMode()

// There can be many Send and SendText calls
c.SendText(`{"brand":{"name":"Chevrolet"},"model":"Camaro","year":2019}`)
c.Send([]byte(`{"brand":{"name":"Chevrolet"},"model":"Camaro","year":2020}`))
c.SendText(`{"brand":{"name":"Chevrolet"},"model":"Camaro","year":2021}`)

// Close
c.Close()

Single:

// Retrieve the record with ID equals to 42
data, err := Single("localhost", "9099", 42)
if err != nil {
    panic(err)
}

Query:

// Establish a new connection to a Basenine server at localhost:9099
c, err := NewConnection("localhost", "9099")
if err != nil {
    panic(err)
}

// Make a []byte channel to recieve the data
data := make(chan []byte)

// Define a function to handle the stream
handleDataChannel := func(wg *sync.WaitGroup, c *Connection, data chan []byte) {
    defer wg.Done()
    for {
        bytes := <-data

        // Do something with bytes

        c.Close()
    }
}

var wg sync.WaitGroup
go handleDataChannel(&wg, c, data)
wg.Add(1)

c.Query(`brand.name == "Chevrolet"`, data)

wg.Wait()

Validate:

err := Validate("localhost", "9099", `brand.name == "Chevrolet"`)
if err != nil {
    // err should be nil, otherwise a connection error or a syntax error
}

Macro:

// Define a macro `chevy` expands into `brand.name == "Chevrolet"`
err := Macro("localhost", "9099", "chevy", `brand.name == "Chevrolet"`)
if err != nil {
    // err can only be a connection error
}

Limit:

// Set the database size limit to 100MB
err := Limit("localhost", "9099", 100000000)
if err != nil {
    // err can only be a connection error
}
You might also like...
🐶  A modern alternative network traffic sniffer.
🐶 A modern alternative network traffic sniffer.

sniffer A modern alternative network traffic sniffer inspired by bandwhich(Rust) and nethogs(C++). sniffer.mov Introduction 中文介绍 sniffer is designed f

Capdns is a network capture utility designed specifically for DNS traffic. This utility is based on tcpdump.
Capdns is a network capture utility designed specifically for DNS traffic. This utility is based on tcpdump.

Capdns is a network capture utility designed specifically for DNS traffic. This utility is based on tcpdump. Some of its features include: Unde

A cloud native distributed streaming network telemetry.
A cloud native distributed streaming network telemetry.

Panoptes Streaming Panoptes Streaming is a cloud native distributed streaming network telemetry. It can be installed as a single binary or clustered n

Openp2p - an open source, free, and lightweight P2P sharing network
Openp2p - an open source, free, and lightweight P2P sharing network

It is an open source, free, and lightweight P2P sharing network. As long as any device joins in, you can access them anywhere

A Realtime API Gateway used with NATS to build REST, real time, and RPC APIs, where all your clients are synchronized seamlessly.
A Realtime API Gateway used with NATS to build REST, real time, and RPC APIs, where all your clients are synchronized seamlessly.

Realtime API Gateway Synchronize Your Clients Visit Resgate.io for guides, live demos, and resources. Resgate is a Go project implementing a realtime

Generate HTTP load and plot the results in real-time
Generate HTTP load and plot the results in real-time

ali A load testing tool capable of performing real-time analysis, inspired by vegeta and jplot. ali comes with an embedded terminal-based UI where you

Real-time HTTP Intrusion Detection
Real-time HTTP Intrusion Detection

teler Real-time HTTP Intrusion Detection Contribute · What's new · Report Bug · Request Feature teler is an real-time intrusion detection and threat a

PinGo is a standalone and feature-rich tool for common IP-based reachability checking tasks. Ping or Trace and Observe in real-time the statistics.

pingo As a network champion from designing and implementing to troubleshooting large scale networks - I know that is usually not easy for administrato

SubCenter is a middleware that integrate task subscriptions and real-time push

Subscription Center SubCenter是一个集成各种任务并进行实时推送的中间件,本身不提供数据与推送服务。

Releases(v0.8.3)
Owner
UP9
UP9
Walrus 🕑 Real-time event streaming platform built on top of gRPC streams

Walrus ?? Real-time event streaming platform built on top of gRPC streams Table of Contents About the project Built With How it works Getting Started

Matheus Mosca 14 Sep 24, 2022
Tool for monitoring network devices (mainly using SNMP) - monitoring check plugin

Thola Description A tool for monitoring network devices written in Go. It features a check mode which complies with the monitoring plugins development

inexio 262 Nov 10, 2022
A simple network analyzer that capture http network traffic

httpcap A simple network analyzer that captures http network traffic. support Windows/MacOS/Linux/OpenWrt(x64) https only capture clienthello colorful

null 2 Oct 25, 2022
Server and client implementation of the grpc go libraries to perform unary, client streaming, server streaming and full duplex RPCs from gRPC go introduction

Description This is an implementation of a gRPC client and server that provides route guidance from gRPC Basics: Go tutorial. It demonstrates how to u

Joram Wambugu 0 Nov 24, 2021
Reducing Malloc/Free traffic to cgo

CGOAlloc Reducing Malloc/Free traffic to cgo Why? Cgo overhead is a little higher than many are comfortable with (at the time of this writing, a simpl

Stephen Baynham 9 Nov 25, 2022
The Akita CLI for watching network traffic, automatically generating API specs, and diffing API specs.

Catch breaking changes faster Akita builds models of your APIs to help you: Catch breaking changes on every pull request, including added/removed endp

Akita Software 223 Dec 1, 2022
Apache Traffic Control is an Open Source implementation of a Content Delivery Network

Apache Traffic Control Apache Traffic Control is an Open Source implementation of a Content Delivery Network. Documentation Intro CDN Basics Traffic C

The Apache Software Foundation 837 Dec 1, 2022