Styx is a simple and high-performance event streaming broker. It aims to provide teams of all sizes with a simple to operate, disk-persisted publish-subscribe system for event streams. Styx is deployed as a single binary with no dependencies, it exposes a high-performance binary protocol as well as HTTP and WebSockets APIs for both event production and consumption.
Designed around the concept of a Commit Log as popularized by projects like Apache Kafka or Apache Pulsar, Styx provides durable storage and atomicity on tail-able event logs.
- Deployed as a single binary with no dependency
- Designed to run and provide durability as a single-node system
- Out-of-the-box Prometheus and Statsd monitoring
- Full-featured CLI for event log management, backup/restore and flexible bulk load/export
- Event records are immutable, atomic, durable, and fsynced to permanent storage before being acked
- Configurable retention policies
- Small footprint. Very low memory and CPU consumption
- Native REST API for event log management
- Native REST API for event production and consumption, including batched with long-polling support
- Native WebSockets API both for event production and consumption
- Millions of events/second, GB/s throughput, low latency
- Scales to thousands of producers, consumers, and event logs
- Decoupled storage engine that can be used standalone
Styx is designed to be a no-brainer for teams that don't want to invest resources in complex (and sometimes fragile) clustered solutions, while providing data safety, plenty of room to scale and instant integration in any language.
How does it work ?
Styx is a Publish-Subscribe system.
Producers publish events to event
consumers subscribe to these logs for further processing.
+--------------+ | | +---->+ Consumer | | | | | +--------------+ +------------+ +------------+ | +--------------+ | | | +----+ | | | Producer +---->+ Styx log +--------->+ Consumer | | | | +----+ | | +------------+ +------------+ | +--------------+ | +--------------+ | | | +---->+ Consumer | | | +--------------+
Usage of the word
log in Styx should not be confused with textual application logs, and designate an ordered sequence of event
records that are persisted to disk.
+------------+------------+------------+-----------+------------+ | record 0 | record 1 | record 2 | ....... | record N | +------------+------------+------------+-----------+------------+
logs are append-only and immutable. As they are persisted to disk, event
logs can be consumed in real-time by susbcribers, and can also be replayed from the beggining or from any arbitrary
position. This unlocks keeping a complete history of published events, regenerating downstream data when updating your app, or replaying complete streams through new processing logic as unexpected needs surface.
Styx usage can differ from traditional Message Queues and Streaming Platforms. In particular we encourage usage patterns that remove the need for consumer acks while providing stronger guarantees. You should consider reading the Reliable Processing Howto as an introduction.
Reliable event production from external sources to Styx logs can also be achieved with data sources exhibiting particular properties. We plan on developing this subject and other usage patterns in Howto's and tutorial as soon as possible.
You can launch a Styx container to try it out with
$ docker pull dataptive/styx $ docker run --name styx -p 7123:7123 dataptive/styx 2021-03-31T08:06:06.987817093Z INFO server: starting Styx server version 0.1.4 2021-03-31T08:06:06.987971158Z INFO logman: starting log manager (data_directory=./data) 2021-03-31T08:06:06.988911129Z INFO server: listening for client connections on 0.0.0.0:7123
Styx will now be reachable at http://localhost:7123/.
The Styx CLI is available from the docker container
$ docker exec -it styx styx Usage: styx COMMAND A command line interface (CLI) for the Styx API. Commands: logs Manage logs Global Options: -f, --format string Output format [text|json] (default "text") -H, --host string Server to connect to (default "http://localhost:7123") -h, --help Display help
Precompiled binaries and packages are available from the Releases section for various operating systems and architectures.
Installing on Debian-based systems
wget https://github.com/dataptive/styx/releases/download/v0.1.4/styx-0.1.4-amd64.deb dpkg -i styx-0.1.4-amd64.deb
Installing on Redhat-based systems
wget https://github.com/dataptive/styx/releases/download/v0.1.4/styx-0.1.4-amd64.rpm rpm -i styx-0.1.4-amd64.rpm
Building from source
To build or install Styx from source, you will need to have a working Go environment.
$ go get github.com/dataptive/styx/cmd/styx-server $ go get github.com/dataptive/styx/cmd/styx
Create a directory to host event log data, and update your
config.toml to make
data_directory point to your newly created directory (a sample config file is provided at the root of this repository).
Assuming you're hosting both
config.toml in your current working directory, run
$ mkdir $PWD/data
Then run the Styx server with this command
$ styx-server --config=$PWD/config.toml
The Styx CLI should be available as the
$ styx Usage: styx COMMAND A command line interface (CLI) for the Styx API. Commands: logs Manage logs Global Options: -f, --format string Output format [text|json] (default "text") -H, --host string Server to connect to (default "http://localhost:7123") -h, --help Display help
To build the binaries without installing them, run
$ git clone https://github.com/dataptive/styx.git $ cd styx $ go build -o styx-server cmd/styx-server/main.go $ go build -o styx cmd/styx/main.go
styx binaries should be available in your working directory.
Producing and consuming records
While the server is running, create a
test event log and tail its contents using the CLI
$ styx logs create test $ styx logs consume test --follow
The CLI will hang, waiting for incoming events (you can quit with Ctrl+D)
In another terminal, try pushing events in the log with curl
$ curl localhost:7123/logs/test/records -X POST -d 'Hello, world !'
All CLI commands support the
--help flag, play with them to get a quick tour of Styx's features. For example
$ styx logs create --help Usage: styx logs create NAME [OPTIONS] Create a new log Options: --max-record-size bytes Maximum record size --index-after-size bytes Write a segment index entry after every size --segment-max-count records Create a new segment when current segment exceeds this number of records --segment-max-size bytes Create a new segment when current segment exceeds this size --segment-max-age seconds Create a new segment when current segment exceeds this age --log-max-count records Expire oldest segment when log exceeds this number of records --log-max-size bytes Expire oldest segment when log exceeds this size --log-max-age seconds Expire oldest segment when log exceeds this age Global Options: -f, --format string Output format [text|json] (default "text") -H, --host string Server to connect to (default "http://localhost:7123") -h, --help Display help
benchmark command from the CLI to run a benchmark on your own server.
The command will launch a batch of produce and consume tasks with various payload sizes and produce a report. You should have about 10GB of free storage for everything to run smoothly. Used disk space will automatically be freed when the benchmark ends.
Although you'll get better numbers when running outside of docker, the following commands will give you a quick estimate of what you can expect from Styx in your own context.
$ docker pull dataptive/styx $ docker run --name styx -d -p 7123:7123 dataptive/styx $ docker exec styx styx benchmark
In case you've built Styx from source, simply run
$ styx benchmark
The documentation is a work in progress, please open an issue if you find something unclear or missing.
- Getting started
- API reference
- Code examples
- Using the Golang client Producer
- Using WebSockets in python to consume a real-time event feed
- Using python+requests to produce concurrently
- Building a reliable processing pipeline using python and WebSockets
- Building a HTTP->Styx event gateway in NodeJS
- Using NodeJS to build a simple event processing pipeline
You're more than welcome to open issues should you encounter any bug or instability in the code. Feature suggestions are welcome. We may restrict PRs until we have setup a CLA.
You're welcome to join our Slack to discuss the project or ask for help !