nanoQ — high-performance brokerless Pub/Sub for streaming real-time data

Related tags

Messaging nq
Overview

nanoQ — high-performance brokerless Pub/Sub for streaming real-time data

nanoQ is a very minimalistic (opinionated/limited) Pub/Sub transport library.

Instant "fire and forget" publish with only best-effort delivery guarantee.

Do I need it?

For telecommunications, media, IoT, gaming, metrics, clicks, etc.: it's okay to loose data to get the most up to date messages.

Brokerless means no central broker server; publishers connect directly to subscribers.

No open Internet. nanoQ is for the private backend infrastructure networks only. There is no integrity / loop detections and others safety mechanisms.

Bandwidth vs latency is configurable, but ultimately nanoQ prefers bandwidth — it is designed to carry hundreds and thousands parallel streams through the backend infrastructure.

In a Nutshell

  • tiny: under 1K LOC
  • high-performance: no allocations on critical paths, granular locking, and other optimizations
  • low overhead simple protocol: 1-5 bytes of metadata (varint length prefix) per message, depending on the message size
  • battle-tested: running in production
  • data oblivious: JSON, protobuf, thrift, gzip, or XML - all goes
  • kubernetes and cloud-native: easy to integrate with existing load-balancers
  • scalable: no coordinator or a central server
  • simple URL address scheme: tcp4://localhost:1234 or unix:///var/run/sockety.sock
  • go native: context, modules and errors wrapping
  • transparent: with Prometheus metrics and pluggable logging

Quick start

Subscriber

import (
    "context"
    "log"
    "time"

    "github.com/aigent/nq"
)

func main() {
    opts := nq.SubOpts{
        KeepaliveTimeout: 5 * time.Second,
        Printf:           log.Printf,
    }
    sub := nq.NewSub("tcp4://:1234", opts, nq.NewDefaultMetrics())

    go func() {
        buf := make([]byte, 4096)
        for {
            if msg, stream, err := sub.Receive(context.TODO(), buf); err != nil {
                log.Println("Error while receiving:", err)
                continue
            } else {
                log.Printf("message from stream '%v' is: %s\n", stream, msg)
            }
        }
    }()
    if err := sub.Listen(context.TODO()); err != nil {
        log.Println("Listen error:", err)
    }
}

Publisher

import (
    "context"
    "log"
    "time"

    "github.com/aigent/nq"
)

func main() {
        opts := nq.PubOpts{
        KeepaliveTimeout: 5 * time.Second,
        ConnectTimeout:   3 * time.Second,
        WriteTimeout:     3 * time.Second,
        FlushFrequency:   100 * time.Millisecond,
        NoDelay:          true,
        Printf:           log.Printf,
    }
    pub := nq.NewPub("tcp4://localhost:1234", opts, nq.NewDefaultMetrics())
    for {
        // Publish the message using 100 connections
        for i := 1; i <= 100; i++ {
            if err := pub.Publish(context.TODO(), []byte("Hello nanoQ"), i); err != nil {
                log.Println("Error while publishing:", err)
            }
        }
    }
}
You might also like...
A brief demo of real-time plotting with Plotly, Go, and server-sent events
A brief demo of real-time plotting with Plotly, Go, and server-sent events

Golang SSE Demo A brief demo of real-time plotting with Plotly, Go, and server-side events. Overview I first learned about Server-Sent Events from @mr

franz-go contains a high performance, pure Go library for interacting with Kafka from 0.8.0 through 2.7.0+. Producing, consuming, transacting, administrating, etc.

franz-go - Apache Kafka client written in Go Franz-go is an all-encompassing Apache Kafka client fully written Go. This library aims to provide every

High-Performance server for NATS, the cloud native messaging system.
High-Performance server for NATS, the cloud native messaging system.

NATS is a simple, secure and performant communications system for digital systems, services and devices. NATS is part of the Cloud Native Computing Fo

Declarative streaming ETL for mundane tasks, written in Go
Declarative streaming ETL for mundane tasks, written in Go

Benthos is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform h

Build event-driven and event streaming applications with ease

Commander 🚀 Commander is Go library for writing event-driven applications. Enabling event sourcing, RPC over messages, SAGA's, bidirectional streamin

replicate messages from streaming channel to jetstream

NATS Streaming/Jetstream Replicator [SJR] Introduction This project replicates messages from streaming channels to jetstream. but why? At Snapp when w

Service responsible for streaming Kafka messages.

kafka-stream 🧙🏼‍♂️ Service responsible for streaming Kafka messages. What it does? This service reads all messages from the input topic and sends th

Мост между NATS streaming и MQ Series

Мост между NATS streaming и MQ Series Оригинальный репозиторий https://github.com/nats-io/nats-mq NATS-MQ Bridge This project implements a simple, but

Streamhub: a toolkit crafted for streaming-powered applications written in Go

✉️ Streamhub Streamhub is a toolkit crafted for streaming-powered applications w

Owner
Aigent
Aigent
golang long polling library. Makes web pub-sub easy via HTTP long-poll server :smiley: :coffee: :computer:

golongpoll Golang long polling library. Makes web pub-sub easy via an HTTP long-poll server. New in v1.1 Deprecated CreateManager and CreateCustomMana

J Cuga 607 Sep 18, 2022
Simple synchronous event pub-sub package for Golang

event-go Simple synchronous event pub-sub package for Golang This is a Go language package for publishing/subscribing domain events. This is useful to

itchyny 19 Jun 16, 2022
ntfy is a super simple pub-sub notification service. It allows you to send desktop notifications via scripts.

ntfy ntfy (pronounce: notify) is a super simple pub-sub notification service. It allows you to send desktop and (soon) phone notifications via scripts

Philipp C. Heckel 2.9k Sep 22, 2022
A basic pub-sub project using NATS

NATS Simple Pub-Sub Mechanism This is a basic pub-sub project using NATS. There is one publisher who publishes a "Hello World" message to a subject ca

Prosonul Haque 1 Dec 13, 2021
A simple in-process pub/sub for golang

go-pub-sub A simple in-process pub/sub for golang Motivation Call it somewhere between "I spent no more than 5 minutes looking for one that existed" a

Ryan Catherman 0 Jan 25, 2022
Kafka, Beanstalkd, Pulsar Pub/Sub framework

go-queue Kafka, Beanstalkd, Pulsar Pub/Sub framework.

chenquan 3 Sep 17, 2022
CLI tool for generating random messages with rules & publishing to the cloud services (SQS,SNS,PUB/SUB and etc.)

Randomsg A CLI tool to generate random messages and publish to cloud services like (SQS,SNS,PUB/SUB and etc.). TODO Generation of nested objects is no

Kerem Dokumacı 6 Sep 22, 2022
Simple, high-performance event streaming broker

Styx Styx is a simple and high-performance event streaming broker. It aims to provide teams of all sizes with a simple to operate, disk-persisted publ

Dataptive 47 Sep 6, 2022
💨 A real time messaging system to build a scalable in-app notifications, multiplayer games, chat apps in web and mobile apps.

Beaver A Real Time Messaging Server. Beaver is a real-time messaging server. With beaver you can easily build scalable in-app notifications, realtime

Ahmed 1.4k Sep 27, 2022
Scalable real-time messaging server in language-agnostic way

Centrifugo is a scalable real-time messaging server in language-agnostic way. Centrifugo works in conjunction with application backend written in any

Centrifugal 6.4k Sep 26, 2022