Easy to use distributed event bus similar to Kafka

Related tags

Messaging chukcha
Overview

chukcha

Easy to use distributed event bus similar to Kafka.

The event bus is designed to be used as a persistent intermediate storage buffer for any kinds of events or logs that you might want.

Possible usages for Chukcha are basically the same as for Kafka, including, but not limited to:

  • Centralised logs collection,
  • Collecting structured or unstructured events for machine learning or analytics (e.g. Hadoop, ClickHouse, etc),
  • Change log buffer to synchronise the contents of one or more storages with each other (e.g. update a full-text index upon database update).

The messages are expected to be delimited by a new line character, although there are no other limitations in terms of events format: it can be JSON, plain text, Protobuf (with \ and new line character escaped), etc.

The youtube playlist to see live development: https://www.youtube.com/watch?v=t3FdULDRfRM&list=PLWwSgbaBp9XqeuIuTWqpNtvf_EL0I4TJ2

Requirements

Go 1.16+ is needed to build Chukcha.

Features (work in progress)

  1. Easy to configure out of the box to not lose data unexpectedly.
  2. Distributed, with asynchronous replication by default.
  3. Explicit acknowledgement of data that was read.

Limitations

  1. The maximum supported size is 1 MiB per message.
  2. Each message has to be separated by a new line character (\n).
  3. The maximum default write batch size is 4 MiB.
  4. Maximum memory consumption is around (write batch size) * 2 (8 MiB with default settings) per connection. TODO: memory consumption per connection can be brought down significantly if using a custom protocol on top of HTTP.
  5. The default upper bound for the number of open connections is 262144.
  6. Kafka protocol is not likely to be ever supported unless someone decides to contribute it.

Differences to Kafka

Compared to Kafka, Chukcha does not guarantee that the messages will be delivered in order or exactly once. You can come close to getting a total order of messages if you always write a single instance, but in that case if that instance fails there would be no way of knowing whether or not data was actually flushed to disk and/or replicated so you would have to retry sending the message to another node in the cluster, so messages can be duplicated. Arguably, the same applies to Kafka, however Kafka provides much stricter guarantees by basically forcing you to always write to the partition leader instead of a random node. Since there is no such concept as a partition leader in Chukcha and all nodes are equal, achieving total messages order is much more difficult.

In Chukcha you are encouraged to spread the write and read load across all the nodes in Chukcha cluster so events order will not be preserved because of async replication being the default. Also please keep in mind that in order to achieve the best performance you would probably want to write data in batches and not just write a single message at a time, so because of that events from every client will need to be buffered and there will be no total event order in that case either. You can see the illustration below that covers this.

data-duplication

Handling data duplication and not having a strict order of events

In many situations duplicating data (or losing it entirely) in case of errors is OK and you don't need to do anything special about that scenario. However, in many cases it actually does matter and the clients need to be able to handle it. Even if you think you don't have that problem in Kafka you probably just lose data instead of having duplicates, which is not ideal in many situations either. In Kafka it's actually possible to both lose data and have duplicates and out-of-order events too in the presence of failures if you are not careful!

Idempotent events handling in databases

Basically the solution to both data being duplicated and loose order of events is the same: idempotency. For example, in the data duplication scenario above you don't want to send just an event that "something happened", you also want to attach a unique event ID to each event that you are sending. When processing each event, e.g. recording it to a database, you then make sure that in a single transaction you first make sure that you haven't processed that event before, e.g. by using a separate table for that, and then doing the actual processing.

Idempotent full-text indexing (e.g. updating Elastic or Manticore Search index)

If you are doing things that are idempotent by nature, e.g. indexing documents, make sure to only send events like "entity with ID changed" and not the entity contents. This way you would have to fetch the most recent contents of that entity from your persistent storage when processing the event and thus you always have up-to-date index. Otherwise you can end up in a situation when the newer versions of the entity are processed before the older ones (this can happen e.g. if you manage to read events that were sent to two different Chukcha nodes at different times) and having a stale index instead. Make sure to update the persistent storage before sending an indexing event too.

Idempotent analytics (e.g. in ClickHouse)

If you are using e.g. ClickHouse make sure you are using e.g. a *ReplacingMergeTree engine family and collapse events by a unique event id generated in your client when sending the event so that duplicates are removed when doing merges.

Idempotent sending of e-mails

There is no such thing as sending e-mail idempotently. You would have to resort to keeping track of e-mails that you already sent successfully in a separate database and have a small chance of sending the last e-mail in the queue more than more once in case of failures. You can limit the number of attempts by consulting the database both before and after attempting to send an e-mail and e.g. limit the number of attempts before giving up on that particular entry.

Data loss scenarios

By default the data is not flushed to disk upon each write, but this can be changed in the server config or per request.

The data durability guarantees are thus the following:

By default data written to a single node can be lost, if:

  • The machine was not shut down cleanly, e.g. a kernel panic occurred or a host experienced any other hardware failures, or
  • The machine was completely lost and never returned into the cluster

(TODO): When min_sync_replicas is greater than 0, data will only be lost if:

  • More than the min_sync_replicas machines are not shut down cleanly simultaneously, or
  • More than min_sync_replicas machines are lost completely and never returned to the cluster.

(TODO2): When sync_each_write=true is set

  • Data is only lost when machine's hard drives/SSDs fail.

(TODO2): When both sync_each_write=true and min_sync_replicas > 0

  • Data is only lost if more than min_sync_replicas machines are totally lost and never re-join the cluster.

Design (work in progress)

  1. Data is split into chunks and is stored as files on disk. Each server owns the chunks being written into and the ownership of each individual chunk is never changed. The files are replicated to all other servers in the cluster.
  2. Readers explicitly acknowledge data that was read and processed successfully. Readers are responsible for reading the chunks starting with the appropirate offsets. Data can expire after a set period of time if so desired (TODO). By default data does not expire at all.

Replication

  1. Replication is asynchronous by default with an option to wait until the data written is replicated to at least N other servers.
  2. Every file in data directory looks like the following: <category>/<server_name>-chunkXXXXXXXXX.
  3. No leaders and followers: all chunks are replicated into all servers in the cluster, all nodes are equal (inspired by ClickHouse replication)
  4. Each instance in the cluster must have a unique name and it will be used as a prefix to all files in the category.
  5. Clients should only connect to a single instance and consume chunks written to all the servers because all data is replicated to every node.
  6. If a node permanently goes away the last chunk will be marked as complete after a (big) timeout.

replication

Installation

  1. Install Go at https://golang.org/
  2. Run the following command in your terminal: $ go install -v github.com/YuriyNasretdinov/[email protected]
  3. The binary built should be located at ~/go/bin/chukcha or at $GOPATH/bin/chukcha if $GOPATH was set.

Usage

It's too early to start using it yet! If you really want to, you can, of course, but keep in mind that this is a project still in the early days of it's development and it's API will probably change a lot before stabilising.

If you really want to use Chukcha, please refer to the simple Go client library at https://pkg.go.dev/github.com/YuriyNasretdinov/chukcha/client.

TODOs:

  1. Limit for the maximum message size is 1 MiB, otherwise we can no longer serve results from disk because we read from disk in 1 MiB chunks.
  2. Handle situations when we run out of disk space or the number of inodes.
  3. Compute code coverage.
  4. Introduce replication.
  5. Write replication tests
  6. Rotate chunks not only based on the size but also based on time passed, to allow chunks downloaded from other servers to be forcefully finalised upon failure.
Owner
Yuriy Nasretdinov
Currently SRE at Google
Yuriy Nasretdinov
pubsub controller using kafka and base on sarama. Easy controll flow for actions streamming, event driven.

Psub helper for create system using kafka to streaming and events driven base. Install go get github.com/teng231/psub have 3 env variables for config

Te Nguyen 5 May 29, 2022
provider-kafka is a Crossplane Provider that is used to manage Kafka resources.

provider-kafka provider-kafka is a Crossplane Provider that is used to manage Kafka resources. Usage Create a provider secret containing a json like t

Crossplane Contrib 15 Jul 18, 2022
A CLI tool for interacting with Kafka through the Confluent Kafka Rest Proxy

kafkactl Table of contents kafkactl Table of contents Overview Build Development Overview kafkactl is a CLI tool to interact with Kafka through the Co

Alexandre Barone 0 Nov 1, 2021
Example Golang Event-Driven with kafka Microservices Choreography

Microservices Choreography A demonstration for event sourcing using Go and Kafka example Microservices Choreography. To run this project: Install Go I

Muhammad Nasrul 0 Dec 2, 2021
Basic Event Streaming - Fundamentals of Kafka Studies (BESt-FunKS)

Apache Kafka My study repo for Apache Kafka. Based on this tutorial. Contents Overview Key Terms Event Topic Producer Consumer Partition Getting Start

João Saraceni 2 Mar 2, 2022
Build event-driven and event streaming applications with ease

Commander ?? Commander is Go library for writing event-driven applications. Enabling event sourcing, RPC over messages, SAGA's, bidirectional streamin

Jeroen Rinzema 60 Jun 13, 2022
Event-planning-go - GRAPHQL Project for Event Planning

About The Project GRAPHQL Project for Event Planning Building the project with l

Muhamad Hilmi Hibatullah 2 Mar 13, 2022
🔊Minimalist message bus implementation for internal communication

?? Bus Bus is a minimalist event/message bus implementation for internal communication. It is heavily inspired from my event_bus package for Elixir la

Mustafa Turan 272 Aug 5, 2022
Native Go bindings for D-Bus

dbus dbus is a simple library that implements native Go client bindings for the D-Bus message bus system. Features Complete native implementation of t

null 740 Aug 13, 2022
Go simple async message bus

?? message-bus Go simple async message bus. ?? ABOUT Contributors: Rafał Lorenz Want to contribute ? Feel free to send pull requests! Have problems, b

Rafał Lorenz 225 Jul 18, 2022
Gin best practices, gin development scaffolding, too late to explain, get on the bus.

Table of Contents generated with DocToc gin_scaffold 现在开始 文件分层 log / redis / mysql / http.client 常用方法 swagger文档生成 gin_scaffold Gin best practices, gin

niuyufu 523 Aug 13, 2022
Messagebus - Simple Message Bus Written in Golang

MessageBus Simple Message Bus Written in Golang How to Use go get gopkg.io/Usada

Usada Pekora 2 Apr 21, 2022
go broker interface,you can use kafka,redis,pulsar etc.

broker go broker interface,you can use kafka,redis,pulsar etc. pulsar in docker run pulsar in docker docker run -dit \ --name pulsar-sever \ -p 6650:

go-god 11 Jul 29, 2022
Confluent's Apache Kafka Golang client

Confluent's Golang Client for Apache KafkaTM confluent-kafka-go is Confluent's Golang client for Apache Kafka and the Confluent Platform. Features: Hi

Confluent Inc. 3.4k Aug 5, 2022
Sarama is a Go library for Apache Kafka 0.8, and up.

sarama Sarama is an MIT-licensed Go client library for Apache Kafka version 0.8 (and later). Getting started API documentation and examples are availa

Shopify 9k Aug 11, 2022
Implementation of the NELI leader election protocol for Go and Kafka

goNELI Implementation of the NELI leader election protocol for Go and Kafka. goNELI encapsulates the 'fast' variation of the protocol, running in excl

Obsidian Dynamics 58 Jul 6, 2022
ChanBroker, a Broker for goroutine, is simliar to kafka

Introduction chanbroker, a Broker for goroutine, is simliar to kafka In chanbroker has three types of goroutine: Producer Consumer(Subscriber) Broker

沉风 61 Aug 12, 2021
Apache Kafka Web UI for exploring messages, consumers, configurations and more with a focus on a good UI & UX.

Kowl - Apache Kafka Web UI Kowl (previously known as Kafka Owl) is a web application that helps you to explore messages in your Apache Kafka cluster a

CloudHut 2.6k Aug 8, 2022
franz-go contains a high performance, pure Go library for interacting with Kafka from 0.8.0 through 2.7.0+. Producing, consuming, transacting, administrating, etc.

franz-go - Apache Kafka client written in Go Franz-go is an all-encompassing Apache Kafka client fully written Go. This library aims to provide every

Travis Bischel 703 Aug 8, 2022