Modern CLI for Apache Kafka, written in Go.

Related tags

Messaging kaf
Overview

Kaf

Kafka CLI inspired by kubectl & docker

Actions Status GoReportCard GoDoc

asciicinema

Install

Install from source:

go get -u github.com/birdayz/kaf/cmd/kaf

Install binary:

curl https://raw.githubusercontent.com/birdayz/kaf/master/godownloader.sh | BINDIR=$HOME/bin bash

Install on Archlinux via AUR:

yay -S kaf

Usage

Add a local Kafka with no auth

kaf config add-cluster local -b localhost:9092

Select cluster from dropdown list

kaf config select-cluster

Describe and List nodes

kaf node ls

List topics, partitions and replicas

kaf topics

Describe a given topic called mqtt.messages.incoming

kaf topic describe mqtt.messages.incoming

List consumer groups

kaf groups

Describe a given consumer group called dispatcher

kafa group describe dispatcher

Write message into given topic from stdin

echo test | kaf produce mqtt.messages.incoming

Configuration

See the examples folder

Shell autocompletion

Source the completion script in your shell commands file:

Bash Linux:

kaf completion bash > /etc/bash_completion.d/kaf

Bash MacOS:

kaf completion bash > /usr/local/etc/bash_completion.d/kaf

Zsh

kaf completion zsh > "${fpath[1]}/_kaf"

Fish

kaf completion fish > ~/.config/fish/completions/kaf.fish

Powershell

Invoke-Expression (@(kaf completion powershell) -replace " ''\)$"," ' ')" -join "`n")

Comments
  • strimzi SSL

    strimzi SSL

    Default Strimzi operator installs with TLS enabled with a crt certificate but no user or password at all. I could not make Kaf to work with just the TLS certificate. I could not find anything in the documentation or examples folder explaining this connectivity mode.

    So far, I am configuring my cluster like:

    - name: strimzi
      brokers:
      - my-cluster-kafka-bootstrap-kafka-strimzi.apps.openshift.mydomain.com:443
      SASL: null
      TLS:
        cafile: /Users/user/ca.crt
        clientfile: ""
        clientkeyfile: ""
        insecure: true
      security-protocol: SASL_SSL
      schema-registry-url: ""
    

    The output for that configuration is:

    $ kaf topics               
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1641e92]
    
    goroutine 1 [running]:
    main.getConfig(0xc000161260)
    	/home/j0e/projects/kaf/cmd/kaf/kaf.go:88 +0x172
    main.getClusterAdmin(0xc0002540e8, 0xc000161260)
    	/home/j0e/projects/kaf/cmd/kaf/kaf.go:177 +0x26
    main.glob..func16(0x1e183e0, 0x1e3c468, 0x0, 0x0)
    	/home/j0e/projects/kaf/cmd/kaf/topic.go:53 +0x37
    github.com/spf13/cobra.(*Command).execute(0x1e183e0, 0x1e3c468, 0x0, 0x0, 0x1e183e0, 0x1e3c468)
    	/home/j0e/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:830 +0x2aa
    github.com/spf13/cobra.(*Command).ExecuteC(0x1e165e0, 0x17d2f3e, 0xc000207f50, 0x100543f)
    	/home/j0e/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
    github.com/spf13/cobra.(*Command).Execute(...)
    	/home/j0e/gopath/pkg/mod/github.com/spf13/[email protected]/command.go:864
    main.main()
    	/home/j0e/projects/kaf/cmd/kaf/kaf.go:105 +0x31
    

    If I change the secutiry-protocol to SSL I get the following:

    $ kaf topics      
    Unable to read Clientfile :open : no such file or directory
    

    Thanks

    opened by jesusmah 10
  • cmd/consume: extend offsets flag and add count

    cmd/consume: extend offsets flag and add count

    This extends the consume command with flags for setting specific offsets, or offsets relative to newest/oldest, setting a count of messages to read, and setting a flag to exit when the end of the partition is reached.

    The changes are roughly based on https://github.com/birdayz/kaf/issues/39, except it does support setting different offsets and counts on different partitions - you just need to make sure your lists of partitions, offsets, and counts are the same length.

    This incidentally fixes a race condition on --follow which was due to the offset variable being shared between goroutines. It mostly showed up for me when partitions had widely different offsets. I can make a PR just fixing that if you don't want to accept this entire change right away.

    opened by SteveRuble 9
  • Adding support to produce avro encoded messages

    Adding support to produce avro encoded messages

    As it turns out, most of the heavy lifting was already done by the schemaCache implementation. If you think this approach is fine I would like to add support for avro keys as well.

    Fixes #181

    opened by fabiojmendes 7
  • Only allow group commit on empty consumergroups

    Only allow group commit on empty consumergroups

    Adds a simple check to the kaf group commit cmd that exits if the consumer group is active. In local testing it was possible for me to set the offset for an active consumer 😬 , thus the change.

    Also added a small example in the readme so its more explicit

    I've been having problems running the tests locally so no tests atm, if anyone has any advice, please share! 🙇 Would be happy to add some test coverage in a follow up MR

    opened by aliellis 7
  • Add integration tests for some of the commands

    Add integration tests for some of the commands

    This commit adds integration tests for a part of available commands. This is achieved by using Gnomock Kafka preset which spins up a temporary Kafka container, and executing CLI commands against it.

    To support integration tests, I had to make a couple of changes to non-test code:

    1. I added context to a couple of places to support cobra's ExecuteContext function. The reason behind it was that consume command does not exit automatically, and there needed to be a way to cancel it after some time. For that I used context.WithTimeout, and pushed this context into consumer code. Now consumer code either passes context forward, or listens to cancellation event itself.
    2. I changed all of the commands to use configurable stdin, stdout and stderr streams. Cobra supports using custom input/output objects, but all the actual commands used os.Stdout directly. I implemented a PersistentPreRun function on rootCmd that decides which reader/writer to use: by default we always use os descriptors.
    opened by orlangure 7
  • Possibility to produce avro encoded message to schema regustry powered topic

    Possibility to produce avro encoded message to schema regustry powered topic

    Hello

    Some tools allow to produce avro encoded messges to topic using schema from schema registry using JSON file as input message.

    For example this https://github.com/tchiotludo/akhq

    opened by eshepelyuk 6
  • Add bash/zsh autocompletion

    Add bash/zsh autocompletion

    Issue https://github.com/birdayz/kaf/issues/14

    I've added bash autocompletion and fixed some minor issues.

    note that it can be slow to get the list of topics. maybe add caching?

    Not sure if this is a good idea. How would you manage such cache? What user should do to update his list of topics?

    opened by tetafro 6
  • Consumption Improvements

    Consumption Improvements

    • I was getting "broker not connected" errors frequently when starting up that end up being a race between requesting partition information and having the broker's connection established. This implements a simply retry+backoff so that it can hopefully succeed.
    • I was also having issues with the output being interleaved on account of the multiple goroutines writing to same writer without synchronization (in some places). I fixed that with a single channel which handles all the writing. This is mentioned in #23
    • I wonder if we can use sarama's newer ConsumerGroup interface which might simplify this -- I have to dig a bit more into that one...

    Thoughts welcome... :-)

    opened by sergiosalvatore 6
  • Add version flag to show the current version of the tool

    Add version flag to show the current version of the tool

    Hi @birdayz

    I finally decided to try a first contribution for this project that has been so useful in the last year of my work. I'm no go expert but I hope this is a good contribution and a good start.

    As requested in #158 this PR:

    Output:

    $> go build -v -ldflags "-s -w -X main.version=v2.0 -X main.commit=abc" ./cmd/kaf/
    $> ./kaf --version
    kaf version v2.0 (abc)
    
    opened by mxcoder 5
  • add TLS config

    add TLS config

    clusters:
    - name: test
      brokers:
      - localhost:9092
      SASL:
        mechanism: PLAIN
        username: admin
        password: mypasswordisnotsosimple
      TLS:
        cafile: /path/ca.pem
        verify: true
      security-protocol: SASL_SSL
    
    opened by vixns 5
  • Add ability to reset consumer group offsets

    Add ability to reset consumer group offsets

    Hey there! First of all, thanks for the great project, the kubectl syntax is a stroke of genius, makes it really friendly to use and stand out compared to the other tools out there.

    I was wondering if there was any demand for adding the ability to reset consumer group offsets? Some other admin-related tools offer this ability, notably kowl and I think it would provide a lot of value to the project.

    I'm happy to own this as I require this functionality, just want to run it past y'all before commencing work :)

    opened by aliellis 4
  • Bump github.com/Shopify/sarama from 1.36.0 to 1.37.0

    Bump github.com/Shopify/sarama from 1.36.0 to 1.37.0

    Bumps github.com/Shopify/sarama from 1.36.0 to 1.37.0.

    Release notes

    Sourced from github.com/Shopify/sarama's releases.

    Version 1.37.0 (2022-09-28)

    What's Changed

    :tada: New Features / Improvements

    :bug: Fixes

    :package: Dependency updates

    :wrench: Maintenance

    New Contributors

    Full Changelog: https://github.com/Shopify/sarama/compare/v1.36.0...v1.37.0

    Commits
    • 619d4ef Merge pull request #2327 from joewreschnig/mock-config-validate
    • 9a14762 Validate the Config when creating a mock producer/consumer
    • b923960 Merge pull request #2350 from Shopify/deps/github.com-pierrec-lz4-v4-4.1.x
    • fcc0267 Merge pull request #2349 from Shopify/deps/github.com-klauspost-compress-1.15.x
    • c9db08b Merge pull request #2336 from Shopify/deps/github.com-shopify-toxiproxy-v2-2.x
    • 802274a Merge pull request #2348 from Shopify/deps/golang.org-x-net-digest
    • f0045f2 chore(deps): bump module github.com/shopify/toxiproxy/v2 to v2.5.0
    • 73a6257 chore(deps): bump module github.com/pierrec/lz4/v4 to v4.1.17
    • 3584bd2 chore(deps): bump module github.com/klauspost/compress to v1.15.11
    • 49606c1 chore(deps): bump golang.org/x/net digest to f486391
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
  • Bump github.com/orlangure/gnomock from 0.21.1 to 0.22.0

    Bump github.com/orlangure/gnomock from 0.21.1 to 0.22.0

    Bumps github.com/orlangure/gnomock from 0.21.1 to 0.22.0.

    Release notes

    Sourced from github.com/orlangure/gnomock's releases.

    v0.22.0

    What's Changed

    Full Changelog: https://github.com/orlangure/gnomock/compare/v0.21.1...v0.22.0

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
  • Question/Feature request: Configure topic with retention period

    Question/Feature request: Configure topic with retention period

    Hello, thanks for providing this tool!

    Is it possible to specify the retention period when creating a topic?

    If not, would it make sense to extend the create and update topic command with multiple config values that will be appended when to the command? This way, kaf will not need to be aware of each configuration. I would expect that Kafka Admin API validates the configuration entries and will provide feedback without kaf needing to validate them.

    Thanks, any feedback would be appreciated.

    opened by sermojohn 0
  • Support for IAM + AWS Managed Kafka (MSK)

    Support for IAM + AWS Managed Kafka (MSK)

    Neat project 👋 hopefully I'm not missing something obvious here (I checked the examples folder) but it doesn't seem like there is any way to configure kaf to use IAM in order to connect to an AWS managed kafka cluster.

    This would be an incredibly useful feature. I am happy to try my best to contribute :) though I have never written golang beyond a couple tutorials

    opened by unredundant 4
Releases(v0.2.3)
Owner
Johannes Brüderl
Software Engineer
Johannes Brüderl
franz-go - A complete Apache Kafka client written in Go

franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 2.8.0+. Producing, consuming, transacting, administrating, etc.

Travis Bischel 759 Sep 25, 2022
Study project that uses Apache Kafka as syncing mechanism between two databases, with producers and consumers written in Go.

Kafka DB Sync Study project that uses Apache Kafka as syncing mechanisms between a monolith DB and a microservice. The main purpose of this project is

Diego Hordi 0 Dec 5, 2021
A CLI tool for interacting with Kafka through the Confluent Kafka Rest Proxy

kafkactl Table of contents kafkactl Table of contents Overview Build Development Overview kafkactl is a CLI tool to interact with Kafka through the Co

Alexandre Barone 0 Nov 1, 2021
Confluent's Apache Kafka Golang client

Confluent's Golang Client for Apache KafkaTM confluent-kafka-go is Confluent's Golang client for Apache Kafka and the Confluent Platform. Features: Hi

Confluent Inc. 3.6k Sep 27, 2022
Sarama is a Go library for Apache Kafka 0.8, and up.

sarama Sarama is an MIT-licensed Go client library for Apache Kafka version 0.8 (and later). Getting started API documentation and examples are availa

Shopify 9.2k Sep 23, 2022
Apache Kafka Web UI for exploring messages, consumers, configurations and more with a focus on a good UI & UX.

Kowl - Apache Kafka Web UI Kowl (previously known as Kafka Owl) is a web application that helps you to explore messages in your Apache Kafka cluster a

CloudHut 2.7k Oct 1, 2022
Cluster extensions for Sarama, the Go client library for Apache Kafka 0.9

Cluster extensions for Sarama, the Go client library for Apache Kafka 0.9 (and later).

Black Square Media 1k Sep 27, 2022
A quick introduction to how Apache Kafka works and differs from other messaging systems using an example application.

Apache Kafka in 6 minutes A quick introduction to how Apache Kafka works and differs from other messaging systems using an example application. In thi

bagher sohrabi 2 Oct 27, 2021
Testing Apache Kafka using Go.

Apache Kafka Go Testing Apache Kafka using Go. Instructions Provision the single node Kafka cluster using Docker: docker-compose -p apache-kafka-go up

Gabriel Kim 0 Dec 17, 2021
provider-kafka is a Crossplane Provider that is used to manage Kafka resources.

provider-kafka provider-kafka is a Crossplane Provider that is used to manage Kafka resources. Usage Create a provider secret containing a json like t

Crossplane Contrib 17 Sep 22, 2022
Apache Pulsar Go Client Library

Apache Pulsar Go Client Library A Go client library for the Apache Pulsar project. Goal This projects is developing a pure-Go client library for Pulsa

The Apache Software Foundation 499 Sep 27, 2022
Implementation of the NELI leader election protocol for Go and Kafka

goNELI Implementation of the NELI leader election protocol for Go and Kafka. goNELI encapsulates the 'fast' variation of the protocol, running in excl

Obsidian Dynamics 58 Jul 6, 2022
ChanBroker, a Broker for goroutine, is simliar to kafka

Introduction chanbroker, a Broker for goroutine, is simliar to kafka In chanbroker has three types of goroutine: Producer Consumer(Subscriber) Broker

沉风 61 Aug 12, 2021
franz-go contains a high performance, pure Go library for interacting with Kafka from 0.8.0 through 2.7.0+. Producing, consuming, transacting, administrating, etc.

franz-go - Apache Kafka client written in Go Franz-go is an all-encompassing Apache Kafka client fully written Go. This library aims to provide every

Travis Bischel 761 Sep 29, 2022
Easy to use distributed event bus similar to Kafka

chukcha Easy to use distributed event bus similar to Kafka. The event bus is designed to be used as a persistent intermediate storage buffer for any k

Yuriy Nasretdinov 74 Sep 19, 2022
Kafka implemented in Golang with built-in coordination (No ZooKeeper, single binary install, Cloud Native)

Jocko Distributed commit log service in Go that is wire compatible with Kafka. Created by @travisjeffery, continued by nash. Goals: Protocol compatibl

Nash.io 104 Aug 9, 2021
kafka watcher for casbin library

Casbin Kafka Watcher Casbin watcher for kafka This watcher library will enable users to dynamically change casbin policies through kakfa messages Infl

Aruna Prabashwara 3 May 8, 2021
Go gRPC Kafka CQRS microservices with tracing

Golang CQRS Kafka gRPC Postgresql MongoDB Redis microservices example ?? ??‍?? Full list what has been used: Kafka as messages broker gRPC Go implemen

Alexander 99 Sep 8, 2022
pubsub controller using kafka and base on sarama. Easy controll flow for actions streamming, event driven.

Psub helper for create system using kafka to streaming and events driven base. Install go get github.com/teng231/psub have 3 env variables for config

Te Nguyen 5 Aug 23, 2022