Build platforms that flexibly mix SQL, batch, and stream processing paradigms

• Optional consumer.MessageProducer interface allows applications to provide an alternate driver of message reading & parsing, outside of the built-in mechanisms for reading journals named in the ShardSpec.
• Binary-compatible enhancement of recovery logs allows for cheap "forks", where historical content of one log is used from within another. This introduced a breaking change to the recoverylog.Recorder, which now has a dedicated constructor and non-exported fields.
• Allow journal names in Read and Append RPCs to be extended with a 'name;suffix', where the suffix is captured into consumer checkpoint offsets. This allows a single consumer to have multiple, concurrent reads of a journal.
• New instrumentation for testing and building custom consumer executables.
• Fix a memory leak in append flow control.
• Fix an allocator bug which could cause delayed convergence under pretty uncommon conditions.
• Fix some corner cases on timely reporting of shard progress in the Stat RPC.
• Update the versions of all dependencies.

• Behavior change: journal registers are a small set of labels which are used for cooperative, transactional locking. Where before a set of checked registers must always match current journal registers for an append to proceed, now an exception exists in the case where a journal has a fully empty set of registers (as is the case on recovery if journal consistency is lost).
• gazctl journals reset-head now issues concurrent resets to enumerated journals, and no longer fails if an attempt to reset a consistent journal loses an append race.
• Makefiles now build local Docker images gazette/broker:latest and gazette/examples:latest, instead of gazette-broker:latest and gazette-examples:latest.
• Additional sanity checks to ensure badly-behaved clients or peers cannot block an exiting broker from draining its server.
• Gazette no longer logs "wrong fragment format" warnings when encountering "directory" objects (those where the object key ends with a '/') in cloud storage. These were just adding noise and were not actually anything to worry about.
• Fix various test flakes.

• BUGFIX: Add a catch-all timeout on server GracefulStop, to ensure poorly behaved clients cannot keep a terminating broker around indefinitely (GH-266).

• % is now an allowed character in label values and journal names. This makes their grammar compatible with URL-encoding.
• Added grpc-prometheus instrumentation to client & server gRPC's.
• Several new metrics added:
  • gazette_shard_*: operational metrics of consumer shards
  • gazette_spool_*: metrics of journal fragment spools
  • gazette_journal_server_*: metrics of gRPC JournalServer invocations
• Some metrics are marked as DEPRECATED, and will be removed in the future:
  • gazette_commits_total
  • gazette_committed_bytes_total
  • gazette_consumer_*
• Some already-broken or unused metrics were removed.
• Improvements to benchmarks and fixed test flakes.

• BUGFIX: Fix potential UUID sequencing race in concurrent calls to PublishCommitted of the same message.Publisher, and add new regression testing (GH-259).
• Instrument message.Sequencer with Prometheus metrics.
• Add curl to release images, making it much simpler for node-zone.sh mappings to directly query cloud metadata APIs.
• Fix omitted registration of broker WriteHead metric.
• Various improvements to tests, addressing or instrumenting flakes.

• Update RocksDB to 6.7.3, and Go to 1.14.2, along with many other package dependencies.
• The etcdtest package no longer embeds an Etcd server. Instead, an etcd binary must be available on the PATH and is invoked as a sub-process, using Unix domain sockets. Users who use etcdtest themselves, or are running Gazette tests outside of the hermetic Docker build environment, must provide a reasonably recent version of etcd.
• Relatedly, the gazette client package no longer depends on Etcd (including the Etcd client).
• JSON-framed messages are now able to use custom marshal/demarshal routines.
• gazctl journals/shards apply now interprets revision -1 (previously disallowed) to mean "don't care", allowing specs to be applied which will always overwrite what's in Etcd. This makes sense for specs managed in another source-of-truth (eg, git).
• Mitigations and improved logging for issue GH-248.
• Remove explicit TCP keep-alive management, as this is now Go default.
• Various doc improvements & cleanups.

• Fix: as-ci target downloads go modules before invoking inner make, to allow external repos to include Makefiles of the gazette/core repo which are dynamically determined using the go mod tool.
• Update to Go 1.13.4

• Added MaxAppendRate JournalSpec field and global broker flag. Append RPCs now use a token-bucket flow control strategy, where RPC chunks are evaluated and potentially throttled or policed against maximum and minimum allowed flow-rates.
• Added PathPostfixTemplate JournalSpec field. Path postfixes are evaluated and applied to individual Fragments as they're persisted. A primary use case is to support Hive-compatible partitioning of Fragments based on their creation time. Journal names and labels may now include the '=' rune, to facilitate the layout of multiple journals as a Hive-partitioned table.
• Reworked almost all documentation into reStructuredText / Sphinx / ReadTheDocs format.
• Make-based build system is refactored to make it easier to integrate and reuse in external repositories and consumer application projects.
• Add DisableWaitForAck ShardSpec field, which toggles the consumer transaction behavior of waiting for ACKs of read pending messages. Most applications won't want to set this, but it can be helpful to avoid stalls in applications with cyclic message flows.

Add ca-certificates to release images (Issue #227)

SQLite is now a supported consumer store! (Issue #212).

A bike-share example and extensive documentation have been added, along with new kustomize manifests for deploying existing examples (Issue #97).

Official Images

Docker images are available on Docker Hub: gazette/broker:v0.84.1 & gazette/examples:v0.84.1.

Other Changes

• Instances of message.Framing may now be dynamically registered. Support for text/csv is added.
• Added the gazctl attach-uuids sub-command.
• Automated partition crash-tests are re-enabled, after adding Urkel DaemonSet kustomize manifests to properly support them (Issue #222).
• The ListFragments RPC now properly respects fragment stores which use re-write rules (Issue #223).
• The journals of ShardSpecs are now verified by consumer servers and gazctl, to actually exist and have appropriate content-types (Issue #216).
• Extensive Godoc and documentation improvements.
• Various minor logging improvements and bug fixes.
• A CHANGELOG.md has been added to the project, and will be updated on a go-forward basis.

This release introduces exactly-once processing semantics to Gazette!

This is a breaking change to many of the consumer package interfaces, notably Shard, Application and Store, as well as the message interfaces. Updates to consumer applications will be required.

This release also introduces Kustomize manifests for deploying brokers, consumers, and dependencies. Helm charts of the repo are deprecated and will be removed in a future release.

Official Images

Docker images are available on Docker Hub: gazette/broker:v0.83.2 & gazette/examples:v0.83.2.

Rolling Upgrades

A rolling upgrade from v0.82 => v0.83 is supported and tested, with the following caveats:

• Brokers must be fully migrated to v0.83 before any consumers may be migrated. This is required as v0.83 brokers introduce journal "registers" which v0.83 consumers rely on. The v0.83 broker is fully compatible with v0.82 consumers.
• v0.83 consumers will migrate the means of storing offsets within RocksDB from now-legacy keys/values to new consumer Checkpoints introduced with v0.83. Legacy offsets are not removed, but are also not updated. This means downgrading from v0.83 => v0.82 will re-process portions of source journals read by the v0.83 consumer. Similarly, a subsequent re-upgrade from v0.82 => v0.83 will not migrate offsets again (and portions read by the downgraded v0.82 consumer will be re-processed).

Release v0.82.2 is a patch release of the v0.82 branch

It includes fixes cherry-picked from master since v0.82.1 was cut:

36a01b6e48d3cc7605bc7ceda49b91659a93baea consumer: fix some spurious shard recovery errors ac3a329843af9d9787875fc1bd17bdf08597863c broker: add more context cancellation checks for log supression 35632e1e9aa8af0de0669da5823d36c9bd3b8d7c broker: proxyAppend should take AppendRequest by value (not reference)
4c6fa33706809c75a1de61eafc082a26de215c2f client: RouteCache should account for empty Route ef7098e8e1b644888c3dcfdec0cc5a4ac7f33183 allocator: update some logging

Official Images

Docker images are available on Docker Hub: gazette/broker:v0.82.2 & gazette/examples:v0.82.2.