Lightweight, fault-tolerant message streams.

Overview

Liftbridge Logo

Build License ReportCard Coverage

Liftbridge provides lightweight, fault-tolerant message streams by implementing a durable stream augmentation for the NATS messaging system. It extends NATS with a Kafka-like publish-subscribe log API that is highly available and horizontally scalable. The goal of Liftbridge is to provide a message-streaming solution with a focus on simplicity and usability. Use it as a simpler and lighter alternative to systems like Kafka and Pulsar or to add streaming semantics to an existing NATS deployment.

See the introduction post on Liftbridge and this post for more context and some of the inspiration behind it.

Documentation

Community

Comments
  • Improve authorization feature

    Improve authorization feature

    What

    As mentioned in the docs, there are a few issues remained in the authorization feature (which is experimental):

    • (1) Authz is not synced across different nodes in the server.
    • (2) Add/Remove a specific permission is done manually.

    Why

    I want to create this issue to discuss possible routes for improvement on these 2 points.

    IMHO, point (1) is more urgent than point (2). If (1) is implemented, then (2) can be considered as acceptable (aka: it will not be more painful than manually edit JSON-based IAM policies on AWS anyway)

    opened by LaPetiteSouris 3
  • SetCursor: context deadline exceeded

    SetCursor: context deadline exceeded

    Every now and then SetCursor API fails with "context deadline exceeded". Could it be something related to a paused cursors stream? Liftbridge v1.7.1 on kubernetes.

    liftbridge-0:

    time="2022-03-07 11:00:29" level=debug msg="fsm: Paused stream __cursors"
    time="2022-03-07 11:00:34" level=debug msg="Server becoming follower for partition [subject=liftbridge-default.cursors, stream=__cursors, partition=0], epoch: 28930"
    time="2022-03-07 11:00:34" level=debug msg="Truncating log for partition [subject=liftbridge-default.cursors, stream=__cursors, partition=0] to 1557767"
    time="2022-03-07 11:00:34" level=debug msg="Replicating partition [subject=liftbridge-default.cursors, stream=__cursors, partition=0] from leader cluster-liftbridge-1"
    time="2022-03-07 11:00:34" level=debug msg="fsm: Resumed partition [subject=liftbridge-default.cursors, stream=__cursors, partition=0]"
    

    liftbridge-1:

    time="2022-03-07 11:00:29" level=debug msg="fsm: Paused stream __cursors"
    time="2022-03-07 11:00:31" level=debug msg="api: PublishAsync [stream=ems.helb.sensors, partition=0]"
    time="2022-03-07 11:00:34" level=debug msg="api: PublishAsync [stream=ems.helb.sensors, partition=0]"
    time="2022-03-07 11:00:34" level=error msg="api: Failed to publish message: context deadline exceeded"
    time="2022-03-07 11:00:34" level=debug msg="api: SetCursor [stream=ems.helb.sensors, partition=0, cursorId=sensors, offset=125103]"
    time="2022-03-07 11:00:34" level=debug msg="api: Publish [stream=__cursors, partition=0]"
    time="2022-03-07 11:00:34" level=debug msg="Server becoming leader for partition [subject=liftbridge-default.cursors, stream=__cursors, partition=0], epoch: 28930"
    time="2022-03-07 11:00:34" level=debug msg="fsm: Resumed partition [subject=liftbridge-default.cursors, stream=__cursors, partition=0]"
    time="2022-03-07 11:00:34" level=debug msg="api: SetCursor [stream=ems.helb.sensors, partition=0, cursorId=sensors, offset=125104]"
    time="2022-03-07 11:00:34" level=debug msg="api: Publish [stream=__cursors, partition=0]"
    

    liftbridge-2:

    time="2022-03-07 11:00:34" level=error msg="Error sending replication request for partition [subject=liftbridge-default.cursors, stream=__cursors, partition=0]: nats: no responders available for request"
    time="2022-03-07 11:00:34" level=error msg="Error sending replication request for partition [subject=liftbridge-default.cursors, stream=__cursors, partition=0]: nats: no responders available for request"
    time="2022-03-07 11:00:34" level=error msg="Error sending replication request for partition [subject=liftbridge-default.cursors, stream=__cursors, partition=0]: nats: no responders available for request"
    time="2022-03-07 11:00:34" level=error msg="Error sending replication request for partition [subject=liftbridge-default.cursors, stream=__cursors, partition=0]: nats: no responders available for request"
    time="2022-03-07 11:00:34" level=error msg="Error sending replication request for partition [subject=liftbridge-default.cursors, stream=__cursors, partition=0]: nats: no responders available for request"
    time="2022-03-07 11:00:34" level=error msg="Error sending replication request for partition [subject=liftbridge-default.cursors, stream=__cursors, partition=0]: nats: no responders available for request"
    ...
    
    opened by abunjevac 13
  • Performance against Redpanda

    Performance against Redpanda

    Since both Liftbridge and Redpanda have a similar goal of simplifying the streaming experience (to different degrees), it would be very interesting to run the same set of producer, consumer, and e2e latency tests on clusters running on identical hardware to see which one can perform better in various scenarios, and any tradeoffs that might come with reduced complexity.

    This would need to wait for #222 and #46 (Metrics and Consumer groups) to be a fair comparison.

    opened by danthegoodman1 1
  • standard emerging for these types of systems

    standard emerging for these types of systems

    https://github.com/asyncapi

    Its part of the Linux foundation. Its basically openAPI but designed for async and for broker agnostics.

    The golang code is really simple. 4 repos that make it easy to understand. https://github.com/orgs/asyncapi/repositories?q=&type=&language=go&sort=

    Uses JS for Designers on top to make it easier to use.

    The server uses kafka under the hood which sucks IMHO :) https://github.com/asyncapi/event-gateway

    I like it alot. Validation of messages by writing some JS at runtime into the event gateway. changing the flow of data at runtime.

    For gophers he equivalent is benthos.

    thoughts ?

    opened by gedw99 0
  • MQTT bridge connector

    MQTT bridge connector

    Implement a bridge process that maps Kafka topics to Liftbridge streams to better support the connection to an MQTT broker. As there are several go MQTT server libs, could potentially embed an MQTT broker for websocket connections to Liftbridge.

    opened by alexisread 0
Releases(v1.9.0)
Owner
Liftbridge
Lightweight, fault-tolerant message streams.
Liftbridge
Dkron - Distributed, fault tolerant job scheduling system https://dkron.io

Dkron - Distributed, fault tolerant job scheduling system for cloud native environments Website: http://dkron.io/ Dkron is a distributed cron service,

Distributed Works 3.3k Oct 4, 2022
A distributed fault-tolerant order book matching engine

Go - Between A distributed fault-tolerant order book matching engine. Features Limit orders Market orders Order book depth Calculate market price for

Daniel Gatis 1 Dec 24, 2021
Easy to use Raft library to make your app distributed, highly available and fault-tolerant

An easy to use customizable library to make your Go application Distributed, Highly available, Fault Tolerant etc... using Hashicorp's Raft library wh

Richard Bertok 59 Sep 5, 2022
An implementation of a distributed KV store backed by Raft tolerant of node failures and network partitions 🚣

barge A simple implementation of a consistent, distributed Key:Value store which uses the Raft Concensus Algorithm. This project launches a cluster of

Shehjad Khan 0 Nov 24, 2021
Sandglass is a distributed, horizontally scalable, persistent, time sorted message queue.

Sandglass is a distributed, horizontally scalable, persistent, time ordered message queue. It was developed to support asynchronous tasks and message

Sandglass 1.5k Sep 23, 2022
The lightweight, distributed relational database built on SQLite

rqlite is a lightweight, distributed relational database, which uses SQLite as its storage engine. Forming a cluster is very straightforward, it grace

rqlite 11.2k Oct 5, 2022
Scalable, fault-tolerant application-layer sharding for Go applications

ringpop-go (This project is no longer under active development.) Ringpop is a library that brings cooperation and coordination to distributed applicat

Uber Open Source 752 Sep 26, 2022
Distributed, fault-tolerant key-value storage written in go.

A simple, distributed, fault-tolerant key-value storage inspired by Redis. It uses Raft protocotol as consensus algorithm. It supports the following data structures: String, Bitmap, Map, List.

Igor German 360 Aug 4, 2022
A distributed, fault-tolerant pipeline for observability data

Table of Contents What Is Veneur? Use Case See Also Status Features Vendor And Backend Agnostic Modern Metrics Format (Or Others!) Global Aggregation

Stripe 1.6k Oct 3, 2022
Super fault-tolerant gateway for HTTP clusters, written in Go. White paper for reference - https://github.com/gptankit/serviceq-paper

ServiceQ ServiceQ is a fault-tolerant gateway for HTTP clusters. It employs probabilistic routing to distribute load during partial cluster shutdown (

Ankit Gupta 66 Jul 16, 2022
Fault tolerant, sharded key value storage written in GoLang

Ravel is a sharded, fault-tolerant key-value store built using BadgerDB and hashicorp/raft. You can shard your data across multiple clusters with mult

Aditya Meharia 75 May 11, 2022
Dkron - Distributed, fault tolerant job scheduling system https://dkron.io

Dkron - Distributed, fault tolerant job scheduling system for cloud native environments Website: http://dkron.io/ Dkron is a distributed cron service,

Distributed Works 3.3k Oct 4, 2022
A distributed fault-tolerant order book matching engine

Go - Between A distributed fault-tolerant order book matching engine. Features Limit orders Market orders Order book depth Calculate market price for

Daniel Gatis 1 Dec 24, 2021
Tendermint Core is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine

Tendermint Core is a Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine - written in any programming language - and securely replicates it on many machines.

null 0 Sep 8, 2022
Easy to use Raft library to make your app distributed, highly available and fault-tolerant

An easy to use customizable library to make your Go application Distributed, Highly available, Fault Tolerant etc... using Hashicorp's Raft library wh

Richard Bertok 59 Sep 5, 2022
Tendermint Core - A Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine

Tendermint Core - A Byzantine Fault Tolerant (BFT) middleware that takes a state transition machine

y 0 Jan 25, 2022
Prometheus Common Data Exporter can parse JSON, XML, yaml or other format data from various sources (such as HTTP response message, local file, TCP response message and UDP response message) into Prometheus metric data.

Prometheus Common Data Exporter Prometheus Common Data Exporter 用于将多种来源(如http响应报文、本地文件、TCP响应报文、UDP响应报文)的Json、xml、yaml或其它格式的数据,解析为Prometheus metric数据。

null 7 May 18, 2022
A Go (golang) package providing high-performance asynchronous logging, message filtering by severity and category, and multiple message targets.

ozzo-log Other languages 简体中文 Русский Description ozzo-log is a Go package providing enhanced logging support for Go programs. It has the following fe

Ozzo Framework 121 Sep 26, 2022
Provide cloud-edge message synergy solutions for companies and individuals.the cloud-edge message system based on NATS.

Swarm This project is a cloud-edge synergy solution based on NATS. quikly deploy cloud deploy on k8s #pull the project. git clone https://github.com/g

null 1 Jan 11, 2022
Alertmanager go message broker - A simple message broker made to integrate with alertmanager/prometheus

Alertmanager message broker Prerequisites Go 1.16+ Sqllite driver About: The alertmanager message broker is a project made to meet some of my needs to

Davi Araújo 0 Dec 27, 2021