Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

Related tags

Distributed Systems go distributed-systems streaming kafka queue messaging
Overview

Jocko

ci gitter codecov

Kafka/distributed commit log service in Go.

Goals of this project:

  • Implement Kafka in Go
  • Protocol compatible with Kafka so Kafka clients and services work with Jocko
  • Make operating simpler
  • Distribute a single binary
  • Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper)
  • Smarter configuration settings
    • Able to use percentages of disk space for retention policies rather than only bytes and time kept
    • Handling size configs when you change the number of partitions or add topics
  • Learn a lot and have fun

TODO

  • Producing
  • Fetching
  • Partition consensus and distribution
  • Protocol
    • Produce
    • Fetch
    • Metadata
    • Create Topics
    • Delete Topics
    • Consumer group [current task]
  • Discovery
  • API versioning [more API versions to implement]
  • Replication [first draft done - testing heavily now]

Hiatus Writing Book

I’m writing a book for PragProg called Building Distributed Services with Go. You can sign up on this mailing list and get updated when the book’s available. It walks you through building a distributed commit log from scratch. I hope it will help Jocko contributors and people who want to work on distributed services.

Reading

Project Layout

├── broker        broker subsystem
├── cmd           commands
│   └── jocko     command to run a Jocko broker and manage topics
├── commitlog     low-level commit log implementation
├── examples      examples running/using Jocko
│   ├── cluster   example booting up a 3-broker Jocko cluster
│   └── sarama    example producing/consuming with Sarama
├── protocol      golang implementation of Kafka's protocol
├── prometheus    wrapper around Prometheus' client lib to handle metrics
├── server        API subsystem
└── testutil      test utils
    └── mock      mocks of the various subsystems

Building

Local

  1. Clone Jocko

    $ go get github.com/travisjeffery/jocko

  2. Build Jocko

    $ cd $GOPATH/src/github.com/travisjeffery/jocko
$ make build

    (If you see an error about dep not being found, ensure that $GOPATH/bin is in your PATH)

Docker

docker build -t travisjeffery/jocko:latest .

Contributing

See CONTRIBUTING for details on submitting patches and the contribution workflow.

License

Jocko is under the MIT license, see the LICENSE file for details.

Comments
  • broker tests #67

    broker tests #67

    • Test broker: New()
      • Broker struct is initialized properly
      • Serf.Bootstrap is called correctly
      • Raft.Bootstrap is called correctly
      • Handle all error cases
    • Test broker: Join()
      • Add test for serf error
    • Test broker: partition()
      • Add test for empty topic
    • Test broker: startReplica()
      • Add test for starting as leader
      • Add test for starting as follower
      • Add test for existing topic
      • Add test for duplicate partition
      • Add test for commitlog New() error
    opened by ryanstinson 11
  • Cleanup cluster example

    Cleanup cluster example

    While trying to reproduce https://github.com/travisjeffery/jocko/issues/20 I've noticed that:

    • the cluster example is using outdated config flags - I found replacement for log-dir and prometheus-addr to be data-dir and http-addr, but I couldn't find replacement for debug so its usage is just removed (maybe DevMode is planned replacement, but that one is not configurable through CLI at the moment and it doesn't change zap logger so probably should be in separate commit)
    • broker-id and data-dir config flags were configuring config struct properties which weren't actually used, there were duplicate struct properties which were used but not configurable throgh CLI
    • jocko binary produced from both make build and by following cluster example is not gitignored so could be by accident committed to version control

    This PR fixes the 3 mentioned issues in 3 separate commits.

    opened by sslavic 6
  • Move leadership from broker to raft

    Move leadership from broker to raft

    Changes:

    • moved leader.go from broker package to raft package to finish the separation between broker and raft

    Next steps:

    • add raft test (mock out serf)
    • add broker tests (mock out serf and raft)
    • modify server tests (mock out broker)
    opened by shwetabhgarg 5
  • Refactor tests

    Refactor tests

    Changes done:

    • Moved tests to their own package
    • Exposed type replicator, re-wrote replicator tests without internals (using mocks)
    • Re-wrote serf tests without broker
    • Minor code changes in commitlog and server: stop accepting connections after server shutdown

    Next steps (TODO):

    • Raft: move leader.go from broker to raft package to finish their separation, write tests for raft
    • Broker: write tests for broker (mocking out serf/raft) to cover more cases
    • Server: modify server tests by mocking out broker, add tests to cover remaining request/response cases
    opened by shwetabhgarg 5
  • Error opening raft store: missing port in address

    Error opening raft store: missing port in address

    Hi,

    I just followed the samara/README.md and got this error:

    $ go run main.go
Error opening raft store: missing port in address
exit status 1

    I'm using go version go1.7.5 darwin/amd64 on macOS Sierra 10.12.3 (16D32).

    I also tried go get -u github.com/kardianos/govendor and govendor sync - inspired by your Makefile - assuming some dependency version issues but got

    $ govendor sync
# cd .; git clone https://gopkg.in/alecthomas/kingpin.v2 /Users/stefanscheidt/Code/Go/.cache/govendor/gopkg.in/alecthomas/kingpin.v2
Cloning into '/Users/stefanscheidt/Code/Go/.cache/govendor/gopkg.in/alecthomas/kingpin.v2'...
error: RPC failed; HTTP 301 curl 22 The requested URL returned error: 301
fatal: The remote end hung up unexpectedly
Error: Remotes failed for:
	Failed for "gopkg.in/alecthomas/kingpin.v2" (failed to clone repo): exit status 128
    opened by stefanscheidt 5
  • Reader giving incorrect results

    Reader giving incorrect results

    It seems that the io.Reader created with NewReader can give incorrect results back. My understanding is that NewReader(offset, maxBytes) will create a reader that begins returning messages starting with the given offset. The following test shows a case where that is not true:

    func TestReader(t *testing.T) {
	var err error
	l := setup(t)
	defer cleanup(t)

	numMsgs := 10
	msgs := make([]commitlog.MessageSet, numMsgs)
	for i := 0; i < numMsgs; i++ {
		msgs[i] = commitlog.NewMessageSet(
			uint64(i), commitlog.NewMessage([]byte(strconv.Itoa(i))),
		)
	}
	for _, ms := range msgs {
		_, err := l.Append(ms)
		require.NoError(t, err)
	}
	idx := 4
	maxBytes := msgs[idx].Size()
	r, err := l.NewReader(int64(idx), maxBytes)
	require.NoError(t, err)

	p := make([]byte, maxBytes)
	_, err = r.Read(p)
	require.NoError(t, err)
	act := commitlog.MessageSet(p)
	require.Equal(t, int64(idx), act.Offset())
}

    --- FAIL: TestReader (0.00s)
        commitlog_test.go:55:
                        Error Trace:    commitlog_test.go:55
                        Error:          Not equal:
                                        expected: 4
                                        actual  : 7
                        Test:           TestReader

    This test inserts 10 MessageSets (offsets 0 to 9), then creates a reader at offset 4. We then check that the first MessageSet returned is offset 4. As you can see in this particular test, it's actually returning a message with offset 7.

    From a brief review of the code, there may be problems with findSegment and Segment.findEntry?

    opened by tylertreat 4
  • Working replication and broker failover

    Working replication and broker failover

    This PR marks a big milestone for Jocko adding a first working draft of replication and broker failover:

    • Working/tested replication/partition reassignment of failed brokers
    • Added broker healthchecks
    • Added tracing You can run Uber's jaegar with $ docker run -d -p5775:5775/udp -p6831:6831/udp -p6832:6832/udp -p5778:5778 -p16686:16686 -p14268:14268 -p9411:9411 jaegertracing/all-in-one, run jocko's tests with $ make test, and open localhost:16686 to see the traces.
    • Added verbose logs which you can enable with JOCKODEBUG="server=1 broker=1 fsm=1"

    Example of a trace:

    screen shot 2018-03-04 at 8 29 22 pm

    opened by travisjeffery 4
  • Different logging library

    Different logging library

    Is there a reason for using travisjeffery/simplelog library for logging? Logrus would give us much more logging features. For jocko specific, I think different logging levels and logging output options that Logrus provides would come in handy.

    opened by BasPH 4
  • Broker cleanup 2

    Broker cleanup 2

    Part 2 of Issue #22 @travisjeffery Made the changes suggested in PR #25

    Change set: Added wrapper over serf.MemberStatus and raft.FSM to cleanly hide hashicorp raft and serf from jocko behind interfaces

    Next steps: add/edit tests Will open a new issue for this

    opened by shwetabhgarg 4
  • Please consider adding vendor dir to next release

    Please consider adding vendor dir to next release

    Hi!

    I am developing a port for FreeBSD where network access is not allowed during build phase. Thus I have no possibility of running go get or deps ensure. Therefor it would be very cool if vendor dir could be added. It would not only benefit FreeBSD, but it'd ensure consistency with other operating systems as well.

    opened by dinapappor 3
  • Ditch the gommap

    Ditch the gommap

    Hi, I just wanted to test the commit log but it looks like the index depends on the https://github.com/tysonmote/gommap which hasn't been updated for almost 5 years and does not work on Windows at all.

    opened by ghost 3
  • example: fix Sarama example

    example: fix Sarama example

    Sarama example did not run correctly, notably because the second value returned by NewTestServer is not a cancel function but a string with the directory containing Kafka data.

    opened by vincentbernat 0
  • Project Status

    Project Status

    Hello team,

    Great job for putting such a lib. I would like to know whether the project is still ongoing or it is abandoned. I do know how difficult it is to combine day-job and open source project.

    opened by Tochemey 4
  • codahale/hdrhistogram repo url has been transferred under the github HdrHstogram umbrella

    codahale/hdrhistogram repo url has been transferred under the github HdrHstogram umbrella

    Problem

    The codahale/hdrhistogram repo has been transferred under the github HdrHstogram umbrella with the help from the original author in Sept 2020 (new repo url https://github.com/HdrHistogram/hdrhistogram-go). The main reasons are to group all implementations under the same roof and to provide more active contribution from the community as the original repository was archived several years ago.

    The dependency URL should be modified to point to the new repository URL. The tag "v0.9.0" was applied at the point of transfer and will reflect the exact code that was frozen in the original repository.

    If you are using Go modules, you can update to the exact point of transfer using the @v0.9.0 tag in your go get command.

    go mod edit -replace github.com/codahale/hdrhistogram=github.com/HdrHistogram/[email protected]

    Performance Improvements

    From the point of transfer, up until now (mon 16 aug 2021), we've released 3 versions that aim support the standard HdrHistogram serialization/exposition formats, and deeply improve READ performance. We recommend to update to the latest version.

    opened by filipecosta90 0
  • Write to CommitLog and Index does not in a transaction.

    Write to CommitLog and Index does not in a transaction.

    Thanks a lot for your project, it is very helpful for me to learn kafka. When I see the code, I find that, writeing a meesage to CommitLog and Index does not in a transaction, so how to ensure the consistency, or maybe I miss something? Thanks again. I am a Chinese student, so forgive my poor English.

    opened by yiippee 1
  • Add support for flex versions

    Add support for flex versions

    Flex version support as per the kafka protocol specification and as implemented in the confluent librdkafka library

    https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_buf.h

    opened by dearchap 2
  • bump gommap to 6edfc90 to support FreeBSD amd64

    bump gommap to 6edfc90 to support FreeBSD amd64

    I'd like to add jocko to FreeBSD ports - with this patch, we already get amd64 support for mmap. I will see what's missing to add arm64 support as well, but that will take a bit longer to work through.

    $ uname -a
FreeBSD 14.0-CURRENT main-n244674-a4f26914d4d5 GENERIC-NODEBUG  amd64

$ gmake
?   	github.com/travisjeffery/jocko/cmd/jocko	[no test files]
=== RUN   TestIndex
--- PASS: TestIndex (0.00s)
=== RUN   TestIndexScanner
--- PASS: TestIndexScanner (0.00s)
=== RUN   TestNewCommitLog
--- PASS: TestNewCommitLog (0.00s)
=== RUN   TestTruncate
--- PASS: TestTruncate (0.00s)
=== RUN   TestCleaner
--- PASS: TestCleaner (0.00s)
=== RUN   TestCompactCleaner
--- PASS: TestCompactCleaner (0.00s)
=== RUN   TestDeleteCleaner
--- PASS: TestDeleteCleaner (0.00s)
=== RUN   TestMessageSet
--- PASS: TestMessageSet (0.00s)
=== RUN   TestMessages
--- PASS: TestMessages (0.00s)
=== RUN   TestMessage
--- PASS: TestMessage (0.00s)
=== RUN   TestReader
=== RUN   TestReader/6
=== RUN   TestReader/60
=== RUN   TestReader/600
=== RUN   TestReader/6000
--- PASS: TestReader (0.00s)
    --- PASS: TestReader/6 (0.00s)
    --- PASS: TestReader/60 (0.00s)
    --- PASS: TestReader/600 (0.00s)
    --- PASS: TestReader/6000 (0.00s)
=== RUN   TestSegmentScanner
--- PASS: TestSegmentScanner (0.00s)
PASS
ok  	github.com/travisjeffery/jocko/commitlog	0.011s
=== RUN   TestNewBrokerLookup
--- PASS: TestNewBrokerLookup (0.00s)
=== RUN   TestBroker_Run
=== RUN   TestBroker_Run/api_versions
2021-02-11T12:58:35.975Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10002}]
2021-02-11T12:58:35.975Z [INFO]  raft: Node at 127.0.0.1:10002 [Leader] entering Leader state
2021/02/11 12:58:35.975651 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/api_versions-node-1 127.0.0.1
2021/02/11 12:58:35.975759 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:35.975835 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:35.975898 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 3, path: /tmp/jocko-test-server-1962560940/raft
2021/02/11 12:58:35.975916 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:35.975946 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/api_versions-node-1
2021/02/11 12:58:35.976237 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.002462 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.002517 broker_test: broker/1: run done
2021/02/11 12:58:36.002650 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.002667 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/create_topic_ok
2021-02-11T12:58:36.003Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10006}]
2021-02-11T12:58:36.004Z [INFO]  raft: Node at 127.0.0.1:10006 [Leader] entering Leader state
2021/02/11 12:58:36.004368 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/create_topic_ok-node-2 127.0.0.1
2021/02/11 12:58:36.004438 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.004439 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.004462 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-2946557211/raft
2021/02/11 12:58:36.004674 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.004707 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/create_topic_ok-node-2
2021/02/11 12:58:36.004962 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.029736 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.029788 broker_test: broker/1: run done
2021/02/11 12:58:36.029959 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.029974 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/create_topic_invalid_replication_factor_error
2021-02-11T12:58:36.032Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10010}]
2021-02-11T12:58:36.032Z [INFO]  raft: Node at 127.0.0.1:10010 [Leader] entering Leader state
2021/02/11 12:58:36.032578 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/create_topic_invalid_replication_factor_error-node-3 127.0.0.1
2021/02/11 12:58:36.032635 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.032643 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.032708 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-3266246590/raft
2021/02/11 12:58:36.032860 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.032883 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/create_topic_invalid_replication_factor_error-node-3
2021/02/11 12:58:36.033096 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.057713 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.057764 broker_test: broker/1: run done
2021/02/11 12:58:36.057905 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.057923 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/delete_topic
2021-02-11T12:58:36.059Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10014}]
2021-02-11T12:58:36.059Z [INFO]  raft: Node at 127.0.0.1:10014 [Leader] entering Leader state
2021/02/11 12:58:36.059707 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/delete_topic-node-4 127.0.0.1
2021/02/11 12:58:36.059762 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.059763 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.059785 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-4213653509/raft
2021/02/11 12:58:36.059921 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.059943 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/delete_topic-node-4
2021/02/11 12:58:36.060146 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.085370 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.085410 broker_test: broker/1: request: ctx: correlation id: 2, api key: 0, client: , size: 0
2021/02/11 12:58:36.085443 broker_test: broker/1: run done
2021/02/11 12:58:36.085563 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.085587 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/offsets
2021-02-11T12:58:36.087Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10018}]
2021-02-11T12:58:36.087Z [INFO]  raft: Node at 127.0.0.1:10018 [Leader] entering Leader state
2021/02/11 12:58:36.087735 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/offsets-node-5 127.0.0.1
2021/02/11 12:58:36.087787 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.087793 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.087820 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-5947463584/raft
2021/02/11 12:58:36.087946 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.087969 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/offsets-node-5
2021/02/11 12:58:36.088172 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.112922 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.113463 broker_test: broker/1: request: ctx: correlation id: 2, api key: 0, client: , size: 0
2021/02/11 12:58:36.113487 broker_test: broker/1: produce: &protocol.ProduceRequest{APIVersion:0, TransactionalID:(*string)(nil), Acks:0, Timeout:100000000, TopicData:[]*protocol.TopicData{(*protocol.TopicData)(0xc0001904e0)}}
2021/02/11 12:58:36.113499 broker_test: broker/1: produce to partition: 0: &{test-topic [0xc00000f860]}
2021/02/11 12:58:36.113576 broker_test: broker/1: request: ctx: correlation id: 3, api key: 0, client: , size: 0
2021/02/11 12:58:36.113605 broker_test: broker/1: request: ctx: correlation id: 4, api key: 0, client: , size: 0
2021/02/11 12:58:36.113628 broker_test: broker/1: run done
2021/02/11 12:58:36.113785 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.113802 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/fetch
2021-02-11T12:58:36.115Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10022}]
2021-02-11T12:58:36.115Z [INFO]  raft: Node at 127.0.0.1:10022 [Leader] entering Leader state
2021/02/11 12:58:36.115475 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/fetch-node-6 127.0.0.1
2021/02/11 12:58:36.115533 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.115535 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.115575 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-6039209599/raft
2021/02/11 12:58:36.115731 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.115774 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/fetch-node-6
2021/02/11 12:58:36.115987 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.140719 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.141226 broker_test: broker/1: request: ctx: correlation id: 2, api key: 0, client: , size: 0
2021/02/11 12:58:36.141246 broker_test: broker/1: produce: &protocol.ProduceRequest{APIVersion:0, TransactionalID:(*string)(nil), Acks:0, Timeout:100000000, TopicData:[]*protocol.TopicData{(*protocol.TopicData)(0xc0001907e0)}}
2021/02/11 12:58:36.141253 broker_test: broker/1: produce to partition: 0: &{test-topic [0xc00000f960]}
2021/02/11 12:58:36.141329 broker_test: broker/1: request: ctx: correlation id: 3, api key: 0, client: , size: 0
2021/02/11 12:58:36.141401 broker_test: broker/1: run done
2021/02/11 12:58:36.142078 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.142100 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/metadata
2021-02-11T12:58:36.143Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10026}]
2021-02-11T12:58:36.143Z [INFO]  raft: Node at 127.0.0.1:10026 [Leader] entering Leader state
2021/02/11 12:58:36.143552 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/metadata-node-7 127.0.0.1
2021/02/11 12:58:36.143605 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.143607 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.143636 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-7717650898/raft
2021/02/11 12:58:36.143757 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.143792 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/metadata-node-7
2021/02/11 12:58:36.143985 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.168698 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.169167 broker_test: broker/1: request: ctx: correlation id: 2, api key: 0, client: , size: 0
2021/02/11 12:58:36.169184 broker_test: broker/1: produce: &protocol.ProduceRequest{APIVersion:0, TransactionalID:(*string)(nil), Acks:0, Timeout:100000000, TopicData:[]*protocol.TopicData{(*protocol.TopicData)(0xc000190a20)}}
2021/02/11 12:58:36.169191 broker_test: broker/1: produce to partition: 0: &{test-topic [0xc00000fa80]}
2021/02/11 12:58:36.169265 broker_test: broker/1: request: ctx: correlation id: 3, api key: 0, client: , size: 0
2021/02/11 12:58:36.169333 broker_test: broker/1: run done
2021/02/11 12:58:36.169514 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.169531 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/produce_topic/partition_doesn't_exist_error
2021-02-11T12:58:36.170Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10030}]
2021-02-11T12:58:36.170Z [INFO]  raft: Node at 127.0.0.1:10030 [Leader] entering Leader state
2021/02/11 12:58:36.171629 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/produce_topic/partition_doesn't_exist_error-node-8 127.0.0.1
2021/02/11 12:58:36.171695 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.171708 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.171712 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-8714297609/raft
2021/02/11 12:58:36.171828 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.171864 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/produce_topic/partition_doesn't_exist_error-node-8
2021/02/11 12:58:36.172057 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.196770 broker_test: broker/1: request: ctx: correlation id: 2, api key: 0, client: , size: 0
2021/02/11 12:58:36.196796 broker_test: broker/1: produce: &protocol.ProduceRequest{APIVersion:0, TransactionalID:(*string)(nil), Acks:0, Timeout:100000000, TopicData:[]*protocol.TopicData{(*protocol.TopicData)(0xc000190bd0)}}
2021/02/11 12:58:36.196805 broker_test: broker/1: produce to partition: 0: &{another-topic [0xc00000fba0]}
2021/02/11 12:58:36.196838 broker_test: broker/1: produce to partition error: unknown topic
2021/02/11 12:58:36.196889 broker_test: broker/1: run done
2021/02/11 12:58:36.197004 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.197023 serf/1: [WARN] serf: Shutdown without a Leave
=== RUN   TestBroker_Run/find_coordinator
2021-02-11T12:58:36.198Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:127.0.0.1:10034}]
2021-02-11T12:58:36.198Z [INFO]  raft: Node at 127.0.0.1:10034 [Leader] entering Leader state
2021/02/11 12:58:36.198644 serf/1: [INFO] serf: EventMemberJoin: TestBroker_Run/find_coordinator-node-9 127.0.0.1
2021/02/11 12:58:36.198708 broker_test: broker/1: adding LAN server: 1
2021/02/11 12:58:36.198746 broker_test: broker/1: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-9243442388/raft
2021/02/11 12:58:36.198780 broker_test: leader/1: cluster leadership acquired
2021/02/11 12:58:36.198852 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.198888 broker_test: leader/1: member joined, marking health alive: TestBroker_Run/find_coordinator-node-9
2021/02/11 12:58:36.199090 broker_test: leader/1: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.223912 broker_test: broker/1: request: ctx: correlation id: 1, api key: 0, client: , size: 0
2021/02/11 12:58:36.224412 broker_test: broker/1: request: ctx: correlation id: 3, api key: 0, client: , size: 0
2021/02/11 12:58:36.231678 broker_test: broker/1: run done
2021/02/11 12:58:36.231852 broker_test: broker/1: shutting down broker
2021/02/11 12:58:36.231873 serf/1: [WARN] serf: Shutdown without a Leave
--- PASS: TestBroker_Run (0.26s)
    --- PASS: TestBroker_Run/api_versions (0.03s)
    --- PASS: TestBroker_Run/create_topic_ok (0.03s)
    --- PASS: TestBroker_Run/create_topic_invalid_replication_factor_error (0.03s)
    --- PASS: TestBroker_Run/delete_topic (0.03s)
    --- PASS: TestBroker_Run/offsets (0.03s)
    --- PASS: TestBroker_Run/fetch (0.03s)
    --- PASS: TestBroker_Run/metadata (0.03s)
    --- PASS: TestBroker_Run/produce_topic/partition_doesn't_exist_error (0.03s)
    --- PASS: TestBroker_Run/find_coordinator (0.03s)
=== RUN   TestBroker_Run_JoinSyncGroup
    broker_test.go:569: 
--- SKIP: TestBroker_Run_JoinSyncGroup (0.00s)
=== RUN   TestBroker_Shutdown
=== RUN   TestBroker_Shutdown/shutdown_ok
2021-02-11T12:58:36.233Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:10 Address:127.0.0.1:10038}]
2021-02-11T12:58:36.233Z [INFO]  raft: Node at 127.0.0.1:10038 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:36.233913 serf/10: [INFO] serf: EventMemberJoin: TestBroker_Shutdown/shutdown_ok-node-10 127.0.0.1
2021/02/11 12:58:36.233968 broker_test: server/10: run handler
2021/02/11 12:58:36.233974 broker_test: broker/10: shutting down broker
2021/02/11 12:58:36.233982 serf/10: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:36.234005 broker_test: broker/10: adding LAN server: 10
--- PASS: TestBroker_Shutdown (0.00s)
    --- PASS: TestBroker_Shutdown/shutdown_ok (0.00s)
=== RUN   TestBroker_JoinLAN
2021-02-11T12:58:36.235Z [INFO]  raft: Initial configuration (index=0): []
2021/02/11 12:58:36.235566 serf/11: [INFO] serf: EventMemberJoin: TestBroker_JoinLAN-node-11 127.0.0.1
2021-02-11T12:58:36.236Z [INFO]  raft: Node at 127.0.0.1:10042 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:36.236285 broker_test: broker/11: adding LAN server: 11
2021-02-11T12:58:36.236Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:36.236Z [INFO]  raft: Node at 127.0.0.1:10046 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:36.237309 serf/12: [INFO] serf: EventMemberJoin: TestBroker_JoinLAN-node-12 127.0.0.1
2021/02/11 12:58:36.237388 broker_test: broker/12: adding LAN server: 12
2021/02/11 12:58:36.237468 memberlist/12: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10043
2021/02/11 12:58:36.237491 memberlist/11: [DEBUG] memberlist: Stream connection from=127.0.0.1:22171
2021/02/11 12:58:36.237990 serf/11: [INFO] serf: EventMemberJoin: TestBroker_JoinLAN-node-12 127.0.0.1
2021/02/11 12:58:36.238066 broker_test: broker/11: adding LAN server: 12
2021/02/11 12:58:36.238115 serf/12: [INFO] serf: EventMemberJoin: TestBroker_JoinLAN-node-11 127.0.0.1
2021/02/11 12:58:36.238146 broker_test: broker/12: adding LAN server: 11
2021/02/11 12:58:36.238222 broker_test: broker/12: shutting down broker
2021/02/11 12:58:36.238234 serf/12: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:36.238468 broker_test: broker/11: shutting down broker
2021/02/11 12:58:36.238485 serf/11: [WARN] serf: Shutdown without a Leave
--- PASS: TestBroker_JoinLAN (0.00s)
=== RUN   TestBroker_RegisterMember
2021-02-11T12:58:36.239Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:13 Address:127.0.0.1:10050}]
2021-02-11T12:58:36.239Z [INFO]  raft: Node at 127.0.0.1:10050 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:36.239914 serf/13: [INFO] serf: EventMemberJoin: TestBroker_RegisterMember-node-13 127.0.0.1
2021/02/11 12:58:36.239970 broker_test: broker/13: adding LAN server: 13
2021/02/11 12:58:36.239988 broker_test: broker/13: raft data found, disabling bootstrap mode: index: 1, path: /tmp/jocko-test-server-13386614088/raft
2021-02-11T12:58:36.240Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:36.240Z [INFO]  raft: Node at 127.0.0.1:10054 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:36.241086 serf/14: [INFO] serf: EventMemberJoin: TestBroker_RegisterMember-node-14 127.0.0.1
2021/02/11 12:58:36.241172 broker_test: broker/14: adding LAN server: 14
2021/02/11 12:58:36.241193 broker_test: broker/14: maybe bootstrap: need more brokers: got: 1: expect: 3
2021/02/11 12:58:36.241229 memberlist/13: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10055
2021/02/11 12:58:36.241256 memberlist/14: [DEBUG] memberlist: Stream connection from=127.0.0.1:22172
2021/02/11 12:58:36.241924 serf/14: [INFO] serf: EventMemberJoin: TestBroker_RegisterMember-node-13 127.0.0.1
2021/02/11 12:58:36.242087 broker_test: broker/14: adding LAN server: 13
2021/02/11 12:58:36.242183 broker_test: broker/14; member TestBroker_RegisterMember-node-13 has bootstrap mode. expect disabled
2021/02/11 12:58:36.242189 serf/13: [INFO] serf: EventMemberJoin: TestBroker_RegisterMember-node-14 127.0.0.1
2021/02/11 12:58:36.242408 broker_test: broker/13: adding LAN server: 14
2021/02/11 12:58:36.340469 serf/14: [DEBUG] serf: messageJoinType: TestBroker_RegisterMember-node-13
2021/02/11 12:58:36.340604 serf/14: [DEBUG] serf: messageJoinType: TestBroker_RegisterMember-node-13
2021/02/11 12:58:36.341461 serf/13: [DEBUG] serf: messageJoinType: TestBroker_RegisterMember-node-13
2021/02/11 12:58:36.341502 serf/13: [DEBUG] serf: messageJoinType: TestBroker_RegisterMember-node-13
2021/02/11 12:58:36.490092 broker_test: broker/13: state:
	members:
		- 0:
			name: TestBroker_RegisterMember-node-13
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_RegisterMember-node-14
			addr: 127.0.0.1
			status: alive
	nodes:
	topics:
2021/02/11 12:58:36.491327 broker_test: broker/14: state:
	members:
		- 0:
			name: TestBroker_RegisterMember-node-14
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_RegisterMember-node-13
			addr: 127.0.0.1
			status: alive
	nodes:
	topics:
2021-02-11T12:58:36.498Z [WARN]  raft: no known peers, aborting election
2021-02-11T12:58:36.638Z [WARN]  raft: Heartbeat timeout from "" reached, starting election
2021-02-11T12:58:36.638Z [INFO]  raft: Node at 127.0.0.1:10050 [Candidate] entering Candidate state in term 2
2021-02-11T12:58:36.639Z [DEBUG] raft: Votes needed: 1
2021-02-11T12:58:36.639Z [DEBUG] raft: Vote granted from 13 in term 2. Tally: 1
2021-02-11T12:58:36.639Z [INFO]  raft: Election won. Tally: 1
2021-02-11T12:58:36.639Z [INFO]  raft: Node at 127.0.0.1:10050 [Leader] entering Leader state
2021/02/11 12:58:36.639176 broker_test: leader/13: cluster leadership acquired
2021/02/11 12:58:36.639282 broker_test: leader/13: join cluster: add voter: 14
2021-02-11T12:58:36.639Z [INFO]  raft: Updating configuration with AddStaging (14, 127.0.0.1:10054) to [{Suffrage:Voter ID:13 Address:127.0.0.1:10050} {Suffrage:Voter ID:14 Address:127.0.0.1:10054}]
2021-02-11T12:58:36.639Z [INFO]  raft: Added peer 14, starting replication
2021/02/11 12:58:36.639441 broker_test: leader/13: member joined, marking health alive: TestBroker_RegisterMember-node-14
2021/02/11 12:58:36 [DEBUG] raft-net: 127.0.0.1:10054 accepted connection from: 127.0.0.1:22182
2021-02-11T12:58:36.639Z [WARN]  raft: Failed to get previous log: 4 log not found (last: 0)
2021-02-11T12:58:36.639Z [WARN]  raft: AppendEntries to {Voter 14 127.0.0.1:10054} rejected, sending older logs (next: 1)
2021-02-11T12:58:36.640Z [INFO]  raft: pipelining replication to peer {Voter 14 127.0.0.1:10054}
2021/02/11 12:58:36.640478 broker_test: leader/13: skipping self join since cluster is too small: servers: 2
2021/02/11 12:58:36.640512 broker_test: leader/13: member joined, marking health alive: TestBroker_RegisterMember-node-13
2021/02/11 12:58:36.646671 broker_test: broker/14: shutting down broker
2021/02/11 12:58:36.646693 serf/14: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:36.647044 broker_test: broker/13: shutting down broker
2021/02/11 12:58:36.647066 serf/13: [WARN] serf: Shutdown without a Leave
2021-02-11T12:58:36.647Z [INFO]  raft: aborting pipeline replication to peer {Voter 14 127.0.0.1:10054}
--- PASS: TestBroker_RegisterMember (0.41s)
=== RUN   TestBroker_FailedMember
2021-02-11T12:58:36.649Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:15 Address:127.0.0.1:10058}]
2021-02-11T12:58:36.649Z [INFO]  raft: Node at 127.0.0.1:10058 [Leader] entering Leader state
2021/02/11 12:58:36.649731 serf/15: [INFO] serf: EventMemberJoin: TestBroker_FailedMember-node-15 127.0.0.1
2021/02/11 12:58:36.649798 broker_test: leader/15: cluster leadership acquired
2021/02/11 12:58:36.649817 broker_test: broker/15: adding LAN server: 15
2021/02/11 12:58:36.649840 broker_test: broker/15: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-15115152314/raft
2021/02/11 12:58:36.649895 broker_test: leader/15: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:36.649933 broker_test: leader/15: member joined, marking health alive: TestBroker_FailedMember-node-15
2021/02/11 12:58:36.650128 broker_test: leader/15: skipping self join since cluster is too small: servers: 1
2021-02-11T12:58:36.650Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:36.650Z [INFO]  raft: Node at 127.0.0.1:10062 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:36.650988 serf/16: [INFO] serf: EventMemberJoin: TestBroker_FailedMember-node-16 127.0.0.1
2021/02/11 12:58:36.651034 broker_test: broker/16: adding LAN server: 16
2021/02/11 12:58:36.651126 memberlist/15: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10063
2021/02/11 12:58:36.651163 memberlist/16: [DEBUG] memberlist: Stream connection from=127.0.0.1:22183
2021/02/11 12:58:36.651557 serf/16: [INFO] serf: EventMemberJoin: TestBroker_FailedMember-node-15 127.0.0.1
2021/02/11 12:58:36.651606 broker_test: broker/16: adding LAN server: 15
2021/02/11 12:58:36.651717 serf/15: [INFO] serf: EventMemberJoin: TestBroker_FailedMember-node-16 127.0.0.1
2021/02/11 12:58:36.651758 broker_test: broker/15: adding LAN server: 16
2021/02/11 12:58:36.651775 broker_test: broker/16: shutting down broker
2021/02/11 12:58:36.651787 serf/16: [WARN] serf: Shutdown without a Leave
2021-02-11T12:58:36.651Z [INFO]  raft: Updating configuration with AddNonvoter (16, 127.0.0.1:10062) to [{Suffrage:Voter ID:15 Address:127.0.0.1:10058} {Suffrage:Nonvoter ID:16 Address:127.0.0.1:10062}]
2021-02-11T12:58:36.651Z [INFO]  raft: Added peer 16, starting replication
2021/02/11 12:58:36.651919 broker_test: leader/15: member joined, marking health alive: TestBroker_FailedMember-node-16
2021/02/11 12:58:36.677177 broker_test: broker/15: shutting down broker
2021/02/11 12:58:36.677196 serf/15: [WARN] serf: Shutdown without a Leave
2021-02-11T12:58:46.652Z [ERROR] raft: Failed to AppendEntries to {Nonvoter 16 127.0.0.1:10062}: dial tcp 127.0.0.1:10062: i/o timeout
2021-02-11T12:58:46.673Z [ERROR] raft: Failed to heartbeat to 127.0.0.1:10062: dial tcp 127.0.0.1:10062: i/o timeout
--- PASS: TestBroker_FailedMember (10.03s)
=== RUN   TestBroker_LeftMember
2021-02-11T12:58:46.676Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:17 Address:127.0.0.1:10066}]
2021-02-11T12:58:46.676Z [INFO]  raft: Node at 127.0.0.1:10066 [Leader] entering Leader state
2021/02/11 12:58:46.676858 serf/17: [INFO] serf: EventMemberJoin: TestBroker_LeftMember-node-17 127.0.0.1
2021/02/11 12:58:46.676932 broker_test: leader/17: cluster leadership acquired
2021/02/11 12:58:46.676975 broker_test: broker/17: adding LAN server: 17
2021/02/11 12:58:46.677002 broker_test: broker/17: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-17924346108/raft
2021/02/11 12:58:46.677107 broker_test: leader/17: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:46.677143 broker_test: leader/17: member joined, marking health alive: TestBroker_LeftMember-node-17
2021/02/11 12:58:46.677416 broker_test: leader/17: skipping self join since cluster is too small: servers: 1
2021-02-11T12:58:46.678Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:46.678Z [INFO]  raft: Node at 127.0.0.1:10070 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:46.678576 serf/18: [INFO] serf: EventMemberJoin: TestBroker_LeftMember-node-18 127.0.0.1
2021/02/11 12:58:46.678638 broker_test: broker/18: adding LAN server: 18
2021/02/11 12:58:46.678729 memberlist/17: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10071
2021/02/11 12:58:46.678766 memberlist/18: [DEBUG] memberlist: Stream connection from=127.0.0.1:22272
2021/02/11 12:58:46.679156 serf/18: [INFO] serf: EventMemberJoin: TestBroker_LeftMember-node-17 127.0.0.1
2021/02/11 12:58:46.679201 broker_test: broker/18: adding LAN server: 17
2021/02/11 12:58:46.679247 serf/17: [INFO] serf: EventMemberJoin: TestBroker_LeftMember-node-18 127.0.0.1
2021/02/11 12:58:46.679291 broker_test: broker/17: adding LAN server: 18
2021-02-11T12:58:46.679Z [INFO]  raft: Updating configuration with AddNonvoter (18, 127.0.0.1:10070) to [{Suffrage:Voter ID:17 Address:127.0.0.1:10066} {Suffrage:Nonvoter ID:18 Address:127.0.0.1:10070}]
2021-02-11T12:58:46.679Z [INFO]  raft: Added peer 18, starting replication
2021/02/11 12:58:46.679462 broker_test: leader/17: member joined, marking health alive: TestBroker_LeftMember-node-18
2021/02/11 12:58:46 [DEBUG] raft-net: 127.0.0.1:10070 accepted connection from: 127.0.0.1:22273
2021-02-11T12:58:46.679Z [WARN]  raft: Failed to get previous log: 5 log not found (last: 0)
2021-02-11T12:58:46.679Z [WARN]  raft: AppendEntries to {Nonvoter 18 127.0.0.1:10070} rejected, sending older logs (next: 1)
2021-02-11T12:58:46.680Z [INFO]  raft: pipelining replication to peer {Nonvoter 18 127.0.0.1:10070}
2021/02/11 12:58:46.704442 broker_test: broker/18: starting leave
2021/02/11 12:58:46 [DEBUG] raft-net: 127.0.0.1:10070 accepted connection from: 127.0.0.1:22274
2021/02/11 12:58:46.778088 serf/18: [DEBUG] serf: messageJoinType: TestBroker_LeftMember-node-17
2021/02/11 12:58:46.778564 serf/18: [DEBUG] serf: messageJoinType: TestBroker_LeftMember-node-17
2021/02/11 12:58:46.778684 serf/17: [DEBUG] serf: messageJoinType: TestBroker_LeftMember-node-17
2021/02/11 12:58:46.778718 serf/18: [INFO] serf: EventMemberLeave: TestBroker_LeftMember-node-18 127.0.0.1
2021/02/11 12:58:46.778721 serf/17: [DEBUG] serf: messageLeaveType: TestBroker_LeftMember-node-18
2021/02/11 12:58:46.778755 broker_test: broker/18: removing LAN server: TestBroker_LeftMember-node-18
2021/02/11 12:58:46.779045 serf/17: [DEBUG] serf: messageJoinType: TestBroker_LeftMember-node-17
2021/02/11 12:58:46.779092 serf/18: [DEBUG] serf: messageLeaveType: TestBroker_LeftMember-node-18
2021/02/11 12:58:46.779119 serf/17: [DEBUG] serf: messageLeaveType: TestBroker_LeftMember-node-18
2021/02/11 12:58:46.877073 serf/18: [DEBUG] serf: messageLeaveType: TestBroker_LeftMember-node-18
2021/02/11 12:58:46.878873 serf/17: [INFO] serf: EventMemberLeave: TestBroker_LeftMember-node-18 127.0.0.1
2021/02/11 12:58:46.878902 broker_test: broker/17: removing LAN server: TestBroker_LeftMember-node-18
2021/02/11 12:58:46.878931 broker_test: leader/17: removing server by id: 18
2021-02-11T12:58:46.878Z [INFO]  raft: Updating configuration with RemoveServer (18, ) to [{Suffrage:Voter ID:17 Address:127.0.0.1:10066}]
2021-02-11T12:58:46.879Z [INFO]  raft: Removed peer 18, stopping replication after 7
2021/02/11 12:58:46.879231 broker_test: leader/17: member is deregistering: reason: left; node: 18
2021-02-11T12:58:46.879Z [INFO]  raft: aborting pipeline replication to peer {Nonvoter 18 127.0.0.1:10070}
2021/02/11 12:58:46 [ERR] raft-net: Failed to flush response: write tcp 127.0.0.1:10070->127.0.0.1:22273: write: broken pipe
2021/02/11 12:58:46.927226 broker_test: broker/17: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 17
			status: passing
	topics:
2021/02/11 12:58:46.928842 broker_test: broker/18: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 17
			status: passing
		- 1:
			id: 18
			status: passing
	topics:
2021/02/11 12:58:46.977115 broker_test: leader/17: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:47.177017 broker_test: broker/17: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 17
			status: passing
	topics:
2021/02/11 12:58:47.178704 broker_test: broker/18: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 17
			status: passing
		- 1:
			id: 18
			status: passing
	topics:
2021/02/11 12:58:47.278541 broker_test: leader/17: skipping self join since cluster is too small: servers: 1
2021-02-11T12:58:47.349Z [WARN]  raft: not part of stable configuration, aborting election
2021/02/11 12:58:47.427102 broker_test: broker/17: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 17
			status: passing
	topics:
2021/02/11 12:58:47.428784 broker_test: broker/18: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 17
			status: passing
		- 1:
			id: 18
			status: passing
	topics:
2021/02/11 12:58:47.578682 broker_test: leader/17: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:47.677040 broker_test: broker/17: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 17
			status: passing
	topics:
2021/02/11 12:58:47.678681 broker_test: broker/18: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 17
			status: passing
		- 1:
			id: 18
			status: passing
	topics:
2021/02/11 12:58:47.878796 broker_test: leader/17: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:47.927054 broker_test: broker/17: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 17
			status: passing
	topics:
2021/02/11 12:58:47.928782 broker_test: broker/18: state:
	members:
		- 0:
			name: TestBroker_LeftMember-node-18
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftMember-node-17
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 17
			status: passing
		- 1:
			id: 18
			status: passing
	topics:
2021/02/11 12:58:48.030607 broker_test: broker/18: shutting down broker
2021/02/11 12:58:48.031079 broker_test: broker/17: shutting down broker
2021/02/11 12:58:48.031144 serf/17: [WARN] serf: Shutdown without a Leave
--- PASS: TestBroker_LeftMember (1.36s)
=== RUN   TestBroker_ReapLeader
2021-02-11T12:58:48.033Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:19 Address:127.0.0.1:10074}]
2021-02-11T12:58:48.033Z [INFO]  raft: Node at 127.0.0.1:10074 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.034025 serf/19: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-19 127.0.0.1
2021/02/11 12:58:48.034396 broker_test: broker/19: adding LAN server: 19
2021-02-11T12:58:48.035Z [INFO]  raft: Initial configuration (index=0): []
2021/02/11 12:58:48.036356 serf/20: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-20 127.0.0.1
2021-02-11T12:58:48.036Z [INFO]  raft: Node at 127.0.0.1:10078 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.036560 broker_test: broker/20: adding LAN server: 20
2021-02-11T12:58:48.037Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:48.037Z [INFO]  raft: Node at 127.0.0.1:10082 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.038149 serf/21: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-21 127.0.0.1
2021/02/11 12:58:48.038285 broker_test: broker/21: adding LAN server: 21
2021/02/11 12:58:48.038352 memberlist/20: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10075
2021/02/11 12:58:48.038515 memberlist/19: [DEBUG] memberlist: Stream connection from=127.0.0.1:22283
2021/02/11 12:58:48.038986 serf/19: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-20 127.0.0.1
2021/02/11 12:58:48.039166 serf/20: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-19 127.0.0.1
2021/02/11 12:58:48.039178 broker_test: broker/19: adding LAN server: 20
2021/02/11 12:58:48.039291 broker_test: broker/20: adding LAN server: 19
2021/02/11 12:58:48.039462 memberlist/21: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10075
2021/02/11 12:58:48.039645 memberlist/19: [DEBUG] memberlist: Stream connection from=127.0.0.1:22284
2021/02/11 12:58:48.041132 serf/19: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-21 127.0.0.1
2021/02/11 12:58:48.041320 broker_test: broker/19: adding LAN server: 21
2021/02/11 12:58:48.041673 serf/21: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-20 127.0.0.1
2021/02/11 12:58:48.041842 serf/21: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-19 127.0.0.1
2021/02/11 12:58:48.041862 broker_test: broker/21: adding LAN server: 20
2021/02/11 12:58:48.042174 broker_test: broker/21: adding LAN server: 19
2021/02/11 12:58:48.135683 serf/20: [INFO] serf: EventMemberJoin: TestBroker_ReapLeader-node-21 127.0.0.1
2021/02/11 12:58:48.135892 broker_test: broker/20: adding LAN server: 21
2021/02/11 12:58:48.136122 serf/19: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48.136877 serf/21: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48.137086 serf/20: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48.138664 serf/20: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-21
2021/02/11 12:58:48.138692 serf/20: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48.138940 serf/21: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48.138972 serf/19: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-21
2021/02/11 12:58:48.138991 serf/19: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48.234410 serf/20: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-21
2021/02/11 12:58:48.234594 serf/21: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-21
2021/02/11 12:58:48.236661 serf/19: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-21
2021/02/11 12:58:48.236810 serf/21: [DEBUG] serf: messageJoinType: TestBroker_ReapLeader-node-21
2021-02-11T12:58:48.238Z [WARN]  raft: Heartbeat timeout from "" reached, starting election
2021-02-11T12:58:48.238Z [INFO]  raft: Node at 127.0.0.1:10074 [Candidate] entering Candidate state in term 2
2021-02-11T12:58:48.238Z [DEBUG] raft: Votes needed: 1
2021-02-11T12:58:48.238Z [DEBUG] raft: Vote granted from 19 in term 2. Tally: 1
2021-02-11T12:58:48.238Z [INFO]  raft: Election won. Tally: 1
2021-02-11T12:58:48.238Z [INFO]  raft: Node at 127.0.0.1:10074 [Leader] entering Leader state
2021/02/11 12:58:48.238867 broker_test: leader/19: cluster leadership acquired
2021/02/11 12:58:48.238989 broker_test: leader/19: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:48.239052 broker_test: leader/19: member joined, marking health alive: TestBroker_ReapLeader-node-19
2021/02/11 12:58:48.239385 broker_test: leader/19: join cluster: add voter: 20
2021-02-11T12:58:48.239Z [INFO]  raft: Updating configuration with AddStaging (20, 127.0.0.1:10078) to [{Suffrage:Voter ID:19 Address:127.0.0.1:10074} {Suffrage:Voter ID:20 Address:127.0.0.1:10078}]
2021-02-11T12:58:48.239Z [INFO]  raft: Added peer 20, starting replication
2021/02/11 12:58:48.239561 broker_test: leader/19: member joined, marking health alive: TestBroker_ReapLeader-node-20
2021/02/11 12:58:48 [DEBUG] raft-net: 127.0.0.1:10078 accepted connection from: 127.0.0.1:22285
2021-02-11T12:58:48.239Z [WARN]  raft: Failed to get previous log: 5 log not found (last: 0)
2021-02-11T12:58:48.240Z [WARN]  raft: AppendEntries to {Voter 20 127.0.0.1:10078} rejected, sending older logs (next: 1)
2021-02-11T12:58:48.240Z [INFO]  raft: pipelining replication to peer {Voter 20 127.0.0.1:10078}
2021/02/11 12:58:48.240952 broker_test: leader/19: join cluster: add voter: 21
2021-02-11T12:58:48.241Z [INFO]  raft: Updating configuration with AddStaging (21, 127.0.0.1:10082) to [{Suffrage:Voter ID:19 Address:127.0.0.1:10074} {Suffrage:Voter ID:20 Address:127.0.0.1:10078} {Suffrage:Voter ID:21 Address:127.0.0.1:10082}]
2021-02-11T12:58:48.241Z [INFO]  raft: Added peer 21, starting replication
2021/02/11 12:58:48 [DEBUG] raft-net: 127.0.0.1:10082 accepted connection from: 127.0.0.1:22286
2021-02-11T12:58:48.241Z [WARN]  raft: Failed to get previous log: 7 log not found (last: 0)
2021/02/11 12:58:48.241646 broker_test: leader/19: member joined, marking health alive: TestBroker_ReapLeader-node-21
2021-02-11T12:58:48.241Z [WARN]  raft: AppendEntries to {Voter 21 127.0.0.1:10082} rejected, sending older logs (next: 1)
2021-02-11T12:58:48.242Z [INFO]  raft: pipelining replication to peer {Voter 21 127.0.0.1:10082}
2021/02/11 12:58:48.247242 broker_test: leader/19: removing server by id: 21
2021-02-11T12:58:48.247Z [INFO]  raft: Updating configuration with RemoveServer (21, ) to [{Suffrage:Voter ID:19 Address:127.0.0.1:10074} {Suffrage:Voter ID:20 Address:127.0.0.1:10078}]
2021-02-11T12:58:48.247Z [INFO]  raft: Removed peer 21, stopping replication after 9
2021-02-11T12:58:48.247Z [INFO]  raft: aborting pipeline replication to peer {Voter 21 127.0.0.1:10082}
2021/02/11 12:58:48.247822 broker_test: leader/19: member is deregistering: reason: reaped; node: 21
2021/02/11 12:58:48.248474 broker_test: broker/21: shutting down broker
2021/02/11 12:58:48.248500 serf/21: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:48.248919 broker_test: broker/20: shutting down broker
2021/02/11 12:58:48.248950 serf/20: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:48.249283 broker_test: broker/19: shutting down broker
2021/02/11 12:58:48.249343 serf/19: [WARN] serf: Shutdown without a Leave
2021-02-11T12:58:48.249Z [INFO]  raft: aborting pipeline replication to peer {Voter 20 127.0.0.1:10078}
--- PASS: TestBroker_ReapLeader (0.22s)
=== RUN   TestBroker_ReapMember
2021-02-11T12:58:48.251Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:22 Address:127.0.0.1:10086}]
2021-02-11T12:58:48.251Z [INFO]  raft: Node at 127.0.0.1:10086 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.252253 serf/22: [INFO] serf: EventMemberJoin: TestBroker_ReapMember-node-22 127.0.0.1
2021/02/11 12:58:48.252425 broker_test: broker/22: adding LAN server: 22
2021-02-11T12:58:48.253Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:48.253Z [INFO]  raft: Node at 127.0.0.1:10090 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.254108 serf/23: [INFO] serf: EventMemberJoin: TestBroker_ReapMember-node-23 127.0.0.1
2021/02/11 12:58:48.254233 broker_test: broker/23: adding LAN server: 23
2021/02/11 12:58:48.254322 memberlist/23: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10087
2021/02/11 12:58:48.254345 memberlist/22: [DEBUG] memberlist: Stream connection from=127.0.0.1:22287
2021/02/11 12:58:48.254869 serf/22: [INFO] serf: EventMemberJoin: TestBroker_ReapMember-node-23 127.0.0.1
2021/02/11 12:58:48.254944 broker_test: broker/22: adding LAN server: 23
2021/02/11 12:58:48.255170 serf/23: [INFO] serf: EventMemberJoin: TestBroker_ReapMember-node-22 127.0.0.1
2021/02/11 12:58:48.255223 broker_test: broker/23: adding LAN server: 22
2021/02/11 12:58:48.353174 serf/22: [DEBUG] serf: messageJoinType: TestBroker_ReapMember-node-23
2021/02/11 12:58:48.354419 serf/22: [DEBUG] serf: messageJoinType: TestBroker_ReapMember-node-23
2021/02/11 12:58:48.354549 serf/23: [DEBUG] serf: messageJoinType: TestBroker_ReapMember-node-23
2021/02/11 12:58:48.452518 serf/23: [DEBUG] serf: messageJoinType: TestBroker_ReapMember-node-23
2021-02-11T12:58:48.459Z [WARN]  raft: Heartbeat timeout from "" reached, starting election
2021-02-11T12:58:48.459Z [INFO]  raft: Node at 127.0.0.1:10086 [Candidate] entering Candidate state in term 2
2021-02-11T12:58:48.459Z [DEBUG] raft: Votes needed: 1
2021-02-11T12:58:48.459Z [DEBUG] raft: Vote granted from 22 in term 2. Tally: 1
2021-02-11T12:58:48.459Z [INFO]  raft: Election won. Tally: 1
2021-02-11T12:58:48.459Z [INFO]  raft: Node at 127.0.0.1:10086 [Leader] entering Leader state
2021/02/11 12:58:48.459596 broker_test: leader/22: cluster leadership acquired
2021/02/11 12:58:48.459703 broker_test: leader/22: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:48.459756 broker_test: leader/22: member joined, marking health alive: TestBroker_ReapMember-node-22
2021/02/11 12:58:48.460055 broker_test: leader/22: join cluster: add voter: 23
2021-02-11T12:58:48.460Z [INFO]  raft: Updating configuration with AddStaging (23, 127.0.0.1:10090) to [{Suffrage:Voter ID:22 Address:127.0.0.1:10086} {Suffrage:Voter ID:23 Address:127.0.0.1:10090}]
2021-02-11T12:58:48.460Z [INFO]  raft: Added peer 23, starting replication
2021/02/11 12:58:48.460228 broker_test: leader/22: member joined, marking health alive: TestBroker_ReapMember-node-23
2021/02/11 12:58:48 [DEBUG] raft-net: 127.0.0.1:10090 accepted connection from: 127.0.0.1:22288
2021-02-11T12:58:48.460Z [WARN]  raft: Failed to get previous log: 5 log not found (last: 0)
2021-02-11T12:58:48.460Z [WARN]  raft: AppendEntries to {Voter 23 127.0.0.1:10090} rejected, sending older logs (next: 1)
2021-02-11T12:58:48.461Z [INFO]  raft: pipelining replication to peer {Voter 23 127.0.0.1:10090}
2021/02/11 12:58:48.482137 broker_test: leader/22: removing server by id: 23
2021-02-11T12:58:48.482Z [INFO]  raft: Updating configuration with RemoveServer (23, ) to [{Suffrage:Voter ID:22 Address:127.0.0.1:10086}]
2021-02-11T12:58:48.482Z [INFO]  raft: Removed peer 23, stopping replication after 7
2021/02/11 12:58:48.482405 broker_test: leader/22: member is deregistering: reason: reaped; node: 23
2021-02-11T12:58:48.482Z [INFO]  raft: aborting pipeline replication to peer {Voter 23 127.0.0.1:10090}
2021/02/11 12:58:48.482634 broker_test: broker/23: shutting down broker
2021/02/11 12:58:48.482664 serf/23: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:48 [ERR] raft-net: Failed to flush response: write tcp 127.0.0.1:10090->127.0.0.1:22288: write: broken pipe
2021/02/11 12:58:48.483153 broker_test: broker/22: shutting down broker
2021/02/11 12:58:48.483179 serf/22: [WARN] serf: Shutdown without a Leave
--- PASS: TestBroker_ReapMember (0.23s)
=== RUN   TestBroker_LeftLeader
2021-02-11T12:58:48.485Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:24 Address:127.0.0.1:10094}]
2021-02-11T12:58:48.485Z [INFO]  raft: Node at 127.0.0.1:10094 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.485659 serf/24: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-24 127.0.0.1
2021/02/11 12:58:48.485784 broker_test: broker/24: adding LAN server: 24
2021/02/11 12:58:48.485845 broker_test: broker/24: raft data found, disabling bootstrap mode: index: 1, path: /tmp/jocko-test-server-24293516697/raft
2021-02-11T12:58:48.486Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:48.486Z [INFO]  raft: Node at 127.0.0.1:10098 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.487334 serf/25: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-25 127.0.0.1
2021/02/11 12:58:48.487438 broker_test: broker/25: adding LAN server: 25
2021/02/11 12:58:48.487462 broker_test: broker/25: maybe bootstrap: need more brokers: got: 1: expect: 3
2021-02-11T12:58:48.488Z [INFO]  raft: Initial configuration (index=0): []
2021-02-11T12:58:48.488Z [INFO]  raft: Node at 127.0.0.1:10102 [Follower] entering Follower state (Leader: "")
2021/02/11 12:58:48.488986 serf/26: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-26 127.0.0.1
2021/02/11 12:58:48.489062 broker_test: broker/26: adding LAN server: 26
2021/02/11 12:58:48.489078 broker_test: broker/26: maybe bootstrap: need more brokers: got: 1: expect: 3
2021/02/11 12:58:48.489168 memberlist/24: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10099
2021/02/11 12:58:48.489210 memberlist/25: [DEBUG] memberlist: Stream connection from=127.0.0.1:22289
2021/02/11 12:58:48.489649 serf/25: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-24 127.0.0.1
2021/02/11 12:58:48.489718 broker_test: broker/25: adding LAN server: 24
2021/02/11 12:58:48.489746 broker_test: broker/25; member TestBroker_LeftLeader-node-24 has bootstrap mode. expect disabled
2021/02/11 12:58:48.489861 serf/24: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-25 127.0.0.1
2021/02/11 12:58:48.489917 broker_test: broker/24: adding LAN server: 25
2021/02/11 12:58:48.490118 memberlist/24: [DEBUG] memberlist: Initiating push/pull sync with: 127.0.0.1:10103
2021/02/11 12:58:48.490161 memberlist/26: [DEBUG] memberlist: Stream connection from=127.0.0.1:22290
2021/02/11 12:58:48.491058 serf/26: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-25 127.0.0.1
2021/02/11 12:58:48.491166 serf/26: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-24 127.0.0.1
2021/02/11 12:58:48.491343 broker_test: broker/26: adding LAN server: 25
2021/02/11 12:58:48.491412 serf/24: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-26 127.0.0.1
2021/02/11 12:58:48.491434 broker_test: broker/26; member TestBroker_LeftLeader-node-24 has bootstrap mode. expect disabled
2021/02/11 12:58:48.491561 broker_test: broker/26: adding LAN server: 24
2021/02/11 12:58:48.491606 broker_test: broker/26; member TestBroker_LeftLeader-node-24 has bootstrap mode. expect disabled
2021/02/11 12:58:48.491687 broker_test: broker/24: adding LAN server: 26
2021/02/11 12:58:48.586551 serf/25: [INFO] serf: EventMemberJoin: TestBroker_LeftLeader-node-26 127.0.0.1
2021/02/11 12:58:48.586597 serf/25: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.586620 serf/25: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.586650 broker_test: broker/25: adding LAN server: 26
2021/02/11 12:58:48.586704 broker_test: broker/25; member TestBroker_LeftLeader-node-24 has bootstrap mode. expect disabled
2021/02/11 12:58:48.586828 serf/26: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.586851 serf/26: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.587807 serf/24: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.587827 serf/24: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.587988 serf/24: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.588011 serf/24: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.589492 serf/25: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.589556 serf/24: [DEBUG] serf: messageJoinType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.735927 broker_test: broker/24: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
	topics:
2021/02/11 12:58:48.737564 broker_test: broker/25: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
	topics:
2021/02/11 12:58:48.739225 broker_test: broker/26: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: alive
	nodes:
	topics:
2021-02-11T12:58:48.800Z [WARN]  raft: no known peers, aborting election
2021-02-11T12:58:48.841Z [WARN]  raft: Heartbeat timeout from "" reached, starting election
2021-02-11T12:58:48.841Z [INFO]  raft: Node at 127.0.0.1:10094 [Candidate] entering Candidate state in term 2
2021-02-11T12:58:48.841Z [DEBUG] raft: Votes needed: 1
2021-02-11T12:58:48.841Z [DEBUG] raft: Vote granted from 24 in term 2. Tally: 1
2021-02-11T12:58:48.841Z [INFO]  raft: Election won. Tally: 1
2021-02-11T12:58:48.841Z [INFO]  raft: Node at 127.0.0.1:10094 [Leader] entering Leader state
2021/02/11 12:58:48.841667 broker_test: leader/24: cluster leadership acquired
2021/02/11 12:58:48.841831 broker_test: leader/24: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:48.841903 broker_test: leader/24: member joined, marking health alive: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.842205 broker_test: leader/24: join cluster: add voter: 25
2021-02-11T12:58:48.842Z [INFO]  raft: Updating configuration with AddStaging (25, 127.0.0.1:10098) to [{Suffrage:Voter ID:24 Address:127.0.0.1:10094} {Suffrage:Voter ID:25 Address:127.0.0.1:10098}]
2021-02-11T12:58:48.842Z [INFO]  raft: Added peer 25, starting replication
2021/02/11 12:58:48.842381 broker_test: leader/24: member joined, marking health alive: TestBroker_LeftLeader-node-25
2021/02/11 12:58:48 [DEBUG] raft-net: 127.0.0.1:10098 accepted connection from: 127.0.0.1:22300
2021-02-11T12:58:48.842Z [WARN]  raft: Failed to get previous log: 5 log not found (last: 0)
2021-02-11T12:58:48.842Z [WARN]  raft: AppendEntries to {Voter 25 127.0.0.1:10098} rejected, sending older logs (next: 1)
2021-02-11T12:58:48.843Z [INFO]  raft: pipelining replication to peer {Voter 25 127.0.0.1:10098}
2021/02/11 12:58:48.843565 broker_test: leader/24: join cluster: add voter: 26
2021-02-11T12:58:48.843Z [INFO]  raft: Updating configuration with AddStaging (26, 127.0.0.1:10102) to [{Suffrage:Voter ID:24 Address:127.0.0.1:10094} {Suffrage:Voter ID:25 Address:127.0.0.1:10098} {Suffrage:Voter ID:26 Address:127.0.0.1:10102}]
2021-02-11T12:58:48.843Z [INFO]  raft: Added peer 26, starting replication
2021/02/11 12:58:48 [DEBUG] raft-net: 127.0.0.1:10102 accepted connection from: 127.0.0.1:22301
2021-02-11T12:58:48.844Z [WARN]  raft: Failed to get previous log: 7 log not found (last: 0)
2021-02-11T12:58:48.844Z [WARN]  raft: AppendEntries to {Voter 26 127.0.0.1:10102} rejected, sending older logs (next: 1)
2021/02/11 12:58:48.844236 broker_test: leader/24: member joined, marking health alive: TestBroker_LeftLeader-node-26
2021/02/11 12:58:48.851151 broker_test: broker/24: starting leave
2021-02-11T12:58:48.851Z [INFO]  raft: Updating configuration with RemoveServer (24, ) to [{Suffrage:Voter ID:25 Address:127.0.0.1:10098} {Suffrage:Voter ID:26 Address:127.0.0.1:10102}]
2021-02-11T12:58:48.851Z [INFO]  raft: pipelining replication to peer {Voter 26 127.0.0.1:10102}
2021-02-11T12:58:48.852Z [INFO]  raft: Removed ourself, shutting down
2021/02/11 12:58:48.852260 broker_test: leader/24: shutting down leader loop
2021-02-11T12:58:48.852Z [INFO]  raft: aborting pipeline replication to peer {Voter 25 127.0.0.1:10098}
2021-02-11T12:58:48.852Z [INFO]  raft: aborting pipeline replication to peer {Voter 26 127.0.0.1:10102}
2021/02/11 12:58:48.852273 broker_test: leader/24: cluster leadership lost
2021/02/11 12:58:48.885970 serf/26: [DEBUG] serf: messageLeaveType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.885986 serf/24: [INFO] serf: EventMemberLeave: TestBroker_LeftLeader-node-24 127.0.0.1
2021/02/11 12:58:48.886022 broker_test: broker/24: removing LAN server: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.886202 serf/25: [DEBUG] serf: messageLeaveType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.887550 serf/24: [DEBUG] serf: messageLeaveType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.887764 serf/24: [DEBUG] serf: messageLeaveType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.887842 serf/25: [INFO] serf: EventMemberLeave: TestBroker_LeftLeader-node-24 127.0.0.1
2021/02/11 12:58:48.887873 broker_test: broker/25: removing LAN server: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.889278 serf/24: [DEBUG] serf: messageLeaveType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.889408 serf/24: [DEBUG] serf: messageLeaveType: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.889476 serf/26: [INFO] serf: EventMemberLeave: TestBroker_LeftLeader-node-24 127.0.0.1
2021/02/11 12:58:48.889501 broker_test: broker/26: removing LAN server: TestBroker_LeftLeader-node-24
2021/02/11 12:58:48.985843 broker_test: broker/24: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:48.987458 broker_test: broker/25: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:48.989141 broker_test: broker/26: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021-02-11T12:58:49.181Z [WARN]  raft: Heartbeat timeout from "127.0.0.1:10094" reached, starting election
2021-02-11T12:58:49.181Z [INFO]  raft: Node at 127.0.0.1:10098 [Candidate] entering Candidate state in term 3
2021-02-11T12:58:49.181Z [DEBUG] raft: Votes needed: 2
2021-02-11T12:58:49.181Z [DEBUG] raft: Vote granted from 25 in term 3. Tally: 1
2021/02/11 12:58:49 [DEBUG] raft-net: 127.0.0.1:10102 accepted connection from: 127.0.0.1:22302
2021-02-11T12:58:49.181Z [WARN]  raft: Rejecting vote request from 127.0.0.1:10098 since we have a leader: 127.0.0.1:10094
2021-02-11T12:58:49.218Z [WARN]  raft: Heartbeat timeout from "127.0.0.1:10094" reached, starting election
2021-02-11T12:58:49.218Z [INFO]  raft: Node at 127.0.0.1:10102 [Candidate] entering Candidate state in term 3
2021-02-11T12:58:49.218Z [DEBUG] raft: Votes needed: 2
2021-02-11T12:58:49.218Z [DEBUG] raft: Vote granted from 26 in term 3. Tally: 1
2021/02/11 12:58:49 [DEBUG] raft-net: 127.0.0.1:10098 accepted connection from: 127.0.0.1:22303
2021-02-11T12:58:49.218Z [INFO]  raft: Duplicate RequestVote for same term: 3
2021/02/11 12:58:49.235858 broker_test: broker/24: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.237512 broker_test: broker/25: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.239146 broker_test: broker/26: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021-02-11T12:58:49.403Z [WARN]  raft: Election timeout reached, restarting election
2021-02-11T12:58:49.403Z [INFO]  raft: Node at 127.0.0.1:10098 [Candidate] entering Candidate state in term 4
2021-02-11T12:58:49.403Z [DEBUG] raft: Votes needed: 2
2021-02-11T12:58:49.403Z [DEBUG] raft: Vote granted from 25 in term 4. Tally: 1
2021-02-11T12:58:49.403Z [DEBUG] raft: lost leadership because received a requestvote with newer term
2021-02-11T12:58:49.403Z [INFO]  raft: Node at 127.0.0.1:10102 [Follower] entering Follower state (Leader: "")
2021-02-11T12:58:49.403Z [DEBUG] raft: Vote granted from 26 in term 4. Tally: 2
2021-02-11T12:58:49.403Z [INFO]  raft: Election won. Tally: 2
2021-02-11T12:58:49.403Z [INFO]  raft: Node at 127.0.0.1:10098 [Leader] entering Leader state
2021-02-11T12:58:49.403Z [INFO]  raft: Added peer 26, starting replication
2021/02/11 12:58:49.403863 broker_test: leader/25: cluster leadership acquired
2021-02-11T12:58:49.404Z [INFO]  raft: pipelining replication to peer {Voter 26 127.0.0.1:10102}
2021/02/11 12:58:49.404401 broker_test: leader/25: member is deregistering: reason: left; node: 24
2021/02/11 12:58:49.404885 broker_test: leader/25: skipping self join since cluster is too small: servers: 2
2021/02/11 12:58:49 [DEBUG] raft-net: 127.0.0.1:10102 accepted connection from: 127.0.0.1:22304
2021/02/11 12:58:49.485822 broker_test: broker/24: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.487478 broker_test: broker/25: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 25
			status: passing
		- 1:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.489119 broker_test: broker/26: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.704396 broker_test: leader/25: skipping self join since cluster is too small: servers: 2
2021/02/11 12:58:49.735886 broker_test: broker/24: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
	nodes:
		- 0:
			id: 24
			status: passing
		- 1:
			id: 25
			status: passing
		- 2:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.737495 broker_test: broker/25: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 25
			status: passing
		- 1:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.739200 broker_test: broker/26: state:
	members:
		- 0:
			name: TestBroker_LeftLeader-node-26
			addr: 127.0.0.1
			status: alive
		- 1:
			name: TestBroker_LeftLeader-node-25
			addr: 127.0.0.1
			status: alive
		- 2:
			name: TestBroker_LeftLeader-node-24
			addr: 127.0.0.1
			status: left
	nodes:
		- 0:
			id: 25
			status: passing
		- 1:
			id: 26
			status: passing
	topics:
2021/02/11 12:58:49.890382 broker_test: broker/24: shutting down broker
2021/02/11 12:58:49.890615 broker_test: broker/26: shutting down broker
2021/02/11 12:58:49.890639 serf/26: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:49.890936 broker_test: broker/25: shutting down broker
2021/02/11 12:58:49.890958 serf/25: [WARN] serf: Shutdown without a Leave
2021-02-11T12:58:49.891Z [INFO]  raft: aborting pipeline replication to peer {Voter 26 127.0.0.1:10102}
2021/02/11 12:58:49.891326 broker_test: broker/24: shutting down broker
--- PASS: TestBroker_LeftLeader (1.41s)
=== RUN   TestConn
2021-02-11T12:58:49.892Z [INFO]  raft: Initial configuration (index=1): [{Suffrage:Voter ID:27 Address:127.0.0.1:10106}]
2021-02-11T12:58:49.892Z [INFO]  raft: Node at 127.0.0.1:10106 [Leader] entering Leader state
2021/02/11 12:58:49.893226 serf/27: [INFO] serf: EventMemberJoin: TestConn-node-27 127.0.0.1
2021/02/11 12:58:49.893290 broker_test: server/27: run handler
=== RUN   TestConn/close_immediately
2021/02/11 12:58:49.893303 broker_test: leader/27: cluster leadership acquired
2021/02/11 12:58:49.893312 broker_test: broker/27: adding LAN server: 27
2021/02/11 12:58:49.893334 broker_test: broker/27: raft data found, disabling bootstrap mode: index: 2, path: /tmp/jocko-test-server-27020760310/raft
2021/02/11 12:58:49.893463 broker_test: leader/27: skipping self join since cluster is too small: servers: 1
=== RUN   TestConn/create_topice
2021/02/11 12:58:49.893505 broker_test: leader/27: member joined, marking health alive: TestConn-node-27
2021/02/11 12:58:49.893709 broker_test: server/27: handle request: ctx: correlation id: 1, api key: 19, client: , size: 44
2021/02/11 12:58:49.893739 broker_test: broker/27: request: ctx: correlation id: 1, api key: 19, client: , size: 44
2021/02/11 12:58:49.893778 broker_test: leader/27: skipping self join since cluster is too small: servers: 1
2021/02/11 12:58:49.893789 broker_test: server/27: handle response: ctx: correlation id: 1, api key: 19, client: , size: 44
=== RUN   TestConn/leader_and_isr
2021/02/11 12:58:49.894100 broker_test: server/27: handle request: ctx: correlation id: 1, api key: 4, client: , size: 74
2021/02/11 12:58:49.894116 broker_test: broker/27: request: ctx: correlation id: 1, api key: 4, client: , size: 74
2021/02/11 12:58:49.894142 broker_test: server/27: handle response: ctx: correlation id: 1, api key: 4, client: , size: 74
=== RUN   TestConn/fetch
2021/02/11 12:58:49.894440 broker_test: server/27: handle request: ctx: correlation id: 1, api key: 1, client: , size: 58
2021/02/11 12:58:49.894464 broker_test: broker/27: request: ctx: correlation id: 1, api key: 1, client: , size: 58
2021/02/11 12:58:49.894494 broker_test: server/27: handle response: ctx: correlation id: 1, api key: 1, client: , size: 58
=== RUN   TestConn/alter_configs
    conn_test.go:187: 
2021/02/11 12:58:49.894750 broker_test: broker/27: shutting down broker
2021/02/11 12:58:49.894766 serf/27: [WARN] serf: Shutdown without a Leave
2021/02/11 12:58:49.894906 broker_test: server/27: listener accept error: accept tcp 127.0.0.1:10105: use of closed network connection
--- PASS: TestConn (0.00s)
    --- PASS: TestConn/close_immediately (0.00s)
    --- PASS: TestConn/create_topice (0.00s)
    --- PASS: TestConn/leader_and_isr (0.00s)
    --- PASS: TestConn/fetch (0.00s)
    --- SKIP: TestConn/alter_configs (0.00s)
=== RUN   TestNewReplicaLookup
--- PASS: TestNewReplicaLookup (0.00s)
=== RUN   TestBroker_Replicate
--- PASS: TestBroker_Replicate (0.01s)
=== RUN   TestProduceConsume
    server_test.go:30: 
--- SKIP: TestProduceConsume (0.00s)
=== RUN   TestConsumerGroup
    server_test.go:176: 
--- SKIP: TestConsumerGroup (0.00s)
PASS
ok  	github.com/travisjeffery/jocko/jocko	13.944s
?   	github.com/travisjeffery/jocko/jocko/config	[no test files]
=== RUN   TestRegisterNode
--- PASS: TestRegisterNode (0.00s)
=== RUN   TestRegisterTopic
--- PASS: TestRegisterTopic (0.00s)
=== RUN   TestRegisterPartition
--- PASS: TestRegisterPartition (0.00s)
=== RUN   TestRegisterGroup
--- PASS: TestRegisterGroup (0.00s)
=== RUN   TestStore_maxIndex
--- PASS: TestStore_maxIndex (0.00s)
=== RUN   TestStore_Abandon
--- PASS: TestStore_Abandon (0.00s)
=== RUN   TestStore_DeleteNode
--- PASS: TestStore_DeleteNode (0.00s)
=== RUN   TestStore_RegisterTopic
--- PASS: TestStore_RegisterTopic (0.00s)
=== RUN   TestStore_RegisterPartition
--- PASS: TestStore_RegisterPartition (0.00s)
=== RUN   TestStore_RegisterGroup
--- PASS: TestStore_RegisterGroup (0.00s)
PASS
ok  	github.com/travisjeffery/jocko/jocko/fsm	(cached)
=== RUN   TestIsBroker
=== RUN   TestIsBroker/minumum_config
--- PASS: TestIsBroker (0.00s)
    --- PASS: TestIsBroker/minumum_config (0.00s)
PASS
ok  	github.com/travisjeffery/jocko/jocko/metadata	(cached)
?   	github.com/travisjeffery/jocko/jocko/state	[no test files]
=== RUN   TestEncodeDecode
=== RUN   TestEncodeDecode/partition
=== RUN   TestEncodeDecode/group
--- PASS: TestEncodeDecode (0.00s)
    --- PASS: TestEncodeDecode/partition (0.00s)
    --- PASS: TestEncodeDecode/group (0.00s)
PASS
ok  	github.com/travisjeffery/jocko/jocko/structs	(cached)
?   	github.com/travisjeffery/jocko/jocko/util	[no test files]
?   	github.com/travisjeffery/jocko/log	[no test files]
?   	github.com/travisjeffery/jocko/mock	[no test files]
=== RUN   TestAlterConfigsRequest
--- PASS: TestAlterConfigsRequest (0.00s)
=== RUN   TestAlterConfigsResponse
--- PASS: TestAlterConfigsResponse (0.00s)
=== RUN   TestCreateTopicRequests
--- PASS: TestCreateTopicRequests (0.00s)
=== RUN   TestCreateTopicResponse
--- PASS: TestCreateTopicResponse (0.00s)
=== RUN   TestDescribeConfigsRequest
--- PASS: TestDescribeConfigsRequest (0.00s)
=== RUN   TestDescribeConfigsResponse
--- PASS: TestDescribeConfigsResponse (0.00s)
=== RUN   TestFetchRequest
--- PASS: TestFetchRequest (0.00s)
=== RUN   TestFetchResponse
--- PASS: TestFetchResponse (0.00s)
=== RUN   TestFindCoordinatorRequest
--- PASS: TestFindCoordinatorRequest (0.00s)
=== RUN   TestFindCoordinatorResponse
--- PASS: TestFindCoordinatorResponse (0.00s)
=== RUN   TestHeartbeatRequest
--- PASS: TestHeartbeatRequest (0.00s)
=== RUN   TestLeaderAndISRRequest
--- PASS: TestLeaderAndISRRequest (0.00s)
=== RUN   TestLeaderAndISRResponse
--- PASS: TestLeaderAndISRResponse (0.00s)
PASS
ok  	github.com/travisjeffery/jocko/protocol	(cached)
?   	github.com/travisjeffery/jocko/testutil	[no test files]
    opened by dch 0
Releases(0.0.1)

  • 0.0.1(Nov 15, 2017)

    Changelog

    c4bdec8 add goreleaser a4a403e Merge pull request #77 from travisjeffery/tests 37c8608 rm comment 89f2b7c broker: test add leader and isr new partition 61bbdbf broker: test shutdown be0ab37 fix sarama example to continue if topic already exists e9ca7e6 rm line leftover (fix #75) 98c4593 server: test join bad request 74e1fe2 add docs 7c5ba89 split out prometheus pkg dd1803f server: test join e498a89 broker: test fetch 6045c3a broker: test metadata f929635 broker: test offsets 790c1e9 broker: test deleting topic c66e9ed broker: test creating topic with invalid replication factor 6bdf05a broker: test produce unknown topic/partition 32b1cf1 broker: test produce 0ae4b66 fix readme 9925fa8 readme: add project layout (#71) 82db271 docs and codecov (#70) 6553b9a fixes for #68 29e8f02 broker tests (#68) 5463e2a broker/test: put request/response in args f2cd1fc broker: rm handle tests - should be tested via Run() tests 93bed6a broker: fix test log dir 98f0107 broker: test api versions 71d0b58 broker: test Join() b52d7dd broker: test New() b89b3d6 Merge pull request #65 from edkellena/master cf500bd fix the docker-compose file to spin up an example cluster. closes #64 8b4c40a cmd: fix parsing --debug flag 7f71492 rename create partition cmd b68cd2b rename pointer 6e78979 broker: test create replica 7a01fa4 broker: test cluster member 83a67cb broker: test create partition 993ed5f broker: test partition 916dbfb broker: test cluster members b3ee3c6 rename mock pkg 67d809b broker: test is controller da7782b add tests stubs, test topics() and topicPartitions() 0ef268d add mocks for raft and serf 84d27a9 fix msg a1f6c88 add jocko create topic cmd 36f488f privatize most the broker apis, follow up to the refactor 02a1063 cleanup e738825 broker: simplify run eab1e08 Merge pull request #62 from travisjeffery/fix-listeners c1603ab serf: fix pointer names 4e49c78 docker: update compose image 1bd95e9 add listener for http (fix #60) d116f9b refactor write ade5ff4 Update readme and Dockerfile with Prometheus address (#59) 83c6e61 Merge pull request #61 from travisjeffery/server-broker-refactor e0363fe Refactor server and broker 94c4f20 add err map 2cbf11c update readme 28c5a83 fix docker-compose 3fe5a80 Merge pull request #57 from travisjeffery/docker 86e3094 example: update args b89f90d docker/compose: give cluster 3 nodes, update args 6af135d docker: update for dep and use makefile 960d403 add Dockerfile, docker-compose.yml. updated examples/cluster documentation closes #49 565363f Fix travis-ci f964a59 govendor -> dep f1ceba1 always expose metrics 5aa4d9b Added Prometheus client and exposed number of requests handled (#55) 36414d5 cli: improve logs 7689523 Refactor cli (#54) 8562803 make broker use protocol.error consistently 5b076d5 use protocol.error 1ad1648 fix lock use 49dd62f add protocol.Error 42513a8 add some more docs 19e71e4 fix example baf2c05 create topic: support setting replication factor 636eecc cmd/createtopic: better logging 4d12fcd Fix metadata response (fix #47) 91740c3 add all error codes 8f45c6d broker: update test to compare contents rather than pointer 4276643 server: mv file 1fe8536 broker: better func name 027b58e rename proxy as client 2b34f68 server/test: close conn to create topic after creating topic 4828840 make: add separate test target w/o race f48e12f server: better req handling e7c94f6 server: better logs a2fd23a lowercase logs 6343bdf broker: add docs for opts e543f9a protocol: replace api iota with values 34c3fde Added APIVersions handler Solved bug in SetupIndex function that prevents the index to be set up 54ea894 fix style/api names 92020b0 Fix peer logging format in serf (#31) 06afc24 add more docs, clean up fn names 075f73b remove extraneous methods 8394c6b Move leadership from broker to raft (#36) 03a7547 rewrite serf tests without broker 7d43f27 stop accepting connections after server shutdown 0ceb188 expose replicator, rewrite replicator test w/ mocks 55bd7bb reduce commitlog interface f80578b commitlog: add benchmark test 6474f62 createtopic: default number of partitions to 1 537fe9e vendor: bump sarama for tests 4e81b6d add benchmark test cf9551c rename clients 25f1554 change partition conn when leader changes 824b34c added proxy for sending requests to server f4c1a63 rm docker-compose.yml 6f4a7f6 add docker-compose for benchmarking against kafka 8a703c9 add lock around creating commitlog/partition e585d27 make: add -race to test target (#27) 54f6616 raft: rename var 90cbc99 separate raft fsm from broker 4dd4d70 seprate serf, make raft interface using broker a61b698 add cmd/createtopic 03079b3 server: just use sarama for the tests rather than directly using the commit log/connections 8ac530a protocol: add doc, fix name of api versions key a6d6e4a broker: clean up serf tests 06a1a9c mv jocko to be top level pkg 9ffedd8 rename brokerconn to clustermember 27056b7 use serf const reap status 65595df hide serf and raft from broker 8145ad3 extract serf and raft pkgs from broker 2ce0047 add var for leader id fee05bf fix replica assignment 4c84f56 fix sarama example (close #26) fa68ab6 server: rm the server test's data dir only! (close #24) b2996a6 Merge branch 'cmdargs' 3a9bad8 update example e67a336 fix brokeraddr arg 0d9e74b make args consistent (fix #21) 25a0bb7 server: add sarama test 3b65289 server: put tests in own pkg 92c6260 Merge branch 'klavs-master' 49a12da keep protocol and commitlog separate 74e190e remove commitlog dependecy on protocol 5d5abbe rm binary (fix #18) 2d43357 use /... to prevent go get from complaining cdcf1cb Fix cmd/jocko/main.go 0bd2f7f Fix Cluster creation 67f2dce Fix setting Port 1847dc4 Standardize self-referrer in broker.go 25b3cdf Make Travis CI badge clickable 3bf02b3 clean up examples 5a63de5 check errs on serf bootstrap e4d0269 use kingpin's Strings flag (#10) cd63854 actually use the serf members list 4b82fae intial commit 55577b7 Config bind addresses from command line, and allow serf memberlist to be configured 3dbfe17 leader and isr: handle when partition doesn't exist yet b1fe2ec rename fn 42ca154 make replicator private and args consistent d638c4a clean up option apis 4916a05 readme: add reading 2f25f4f separate raft and serf some more 38da9d1 index: remove existing test since indexes are rebuilt 4ed2c4a add license 6523f47 Update README.md 9e9dd23 add makefile c9f8a40 update readme 70d6475 add rest of serf implementation e21072b travis: run tests serially 3002c82 add serf event handling to add/remove brokers 48fb383 add initial serf calls 180154f update readme f02f278 Add Gitter badge 1408532 add replication manager (#3) ac25989 add err checks 6ebb5ce protocol: fix isr leader request 6c88901 add replicator options 5c4a38b add broker options bd29860 refactor d0a4ecf protocol: fix replicas on leader and isr request 7cb979b remove naked returns (#7) 5bb50a1 protocol: add bool encoder/decoder (#7) b4176fe 1. add stop replica protocol 2. add const api key variable 6ff11be replication: truncate to hw when restarting to replicate 9dccfcf protocol: add leader and isr request/response c00ca30 add var for corrupt index err 36a5977 add sanity check for indexes (close #6) fc5b1ae rename index.offset -> position since its the byte position/offset to not confuse with a message offset b9ddd2c update contributing 6a5d11f rename var 50e7538 update contributing 8e45211 add CONTRIBUTING.md 44f3120 process errors for create data directory (#4) 01ccf60 be able to pass in replica id for fetch requests a98f5b2 add log cleaner 154ba08 commitlog: fix position of 1st entry 56a588b commitlog: add .TruncateTo(offset) (#3) 128fab4 clean up call f58630a add doc f742237 broker: check err 222cc90 replicator: fail fast when something goes wrong during replication 22b8a88 replicator: make client id descriptive 18c4e94 update readme bea38ac test: run server on another port so tests ran in parallel don't conflict 44c0edf update vendor 6c206d0 git: ignore vendor 8959492 tests: specify dir for logs 0e5ab15 fix test faa99c4 start on partition replicator 53ac606 hold references to brokers for host addrs, not just ids 1f03ae5 server: don't panic when serve fails 1ac1e3f example: separate client and jocko setup b21d0a6 add message, message set protocols cf5df62 add vendor 9a8906d add rest of group protocols 6266e97 add join group request/response 555414c check err f201be8 fix typo (close #2) 7fd8038 update readme 40df661 add godeps ee78e52 add ci 6923c1e change test addr 3255abb add err for topic created already aa2b36d add index rebuilding 0083695 add note about creating topic 70b7628 example: use broker api to create topic 031de7d remove some unused apis 25120ea add docs to example effb7b8 add example ca770ad fix some concurrency issues c0719ab add flag to enable debug logs 39bf869 update readme 837c9eb add topic deletes 555c0e4 implement baseoffset for produce response 7604435 remove logs 6c3d374 use better var names and add err checks 3871074 produce request: check all errors 6469b55 add highwatermark to fetch response 24abb98 make newest offset public and add oldest offset for offsets api 70f7d43 segment message set offset on write 309c097 clean up server logging 050dd3f add acks to produce request 46c56ea make segment writes threadsafe d7876bb brokers need both tcp and raft addrs 1c4fd92 support version 1 fetch request 55a533e add throttle time to produce response c630499 add offsets api 4d8cf0a produce: support acks 59ab629 update for api change 7b1daac use ints for ids 6c3066c fix server test 51c436d improve consensus test 54fbf5f add protocol to readme f6a86bf remove unused http stuff e3539fc add fetch request/response bbbed4a add produce request/response 651b725 add support for metadata request/response 21cceee update readme 3e1c83d implement protocol for create topics request/response b193f7e remove client cb07073 move protocol into its own pkg 08da003 start on decoding protocol d45bcd6 add client metadata api aedbf89 rename store to broker 39a5cb5 use bytes buffer/copy d7526c8 add max wait time support to fetch api a698dee update readme ea2e890 add fetch api 4d30e5c fix read to handle byte slices with various sizes 0423e40 update commitlog to read and write messagesets ee2f1ad add api to get each msg from msg set aa49997 add message, message set 84fe73b check that broker is leader on produce cf4038f add api to append messages b7d1367 add api to create topics 89196da use raft for partition consensus af87a57 add types for metadata 9f0cb7f add .gitignore 4b0aa1d add store f0c4e52 add server tests 7173665 remove main ca8623c add cluster 33dc011 add server 5109016 Update README.md 659c223 update readme 85692c8 update readme 3fd013b add partition 95c9c4d fix test to fail if write isn't ok 95b359f fix indexes to store relative offsets 7c3b7c7 move find entry to segment 93eca0a add clients to todos 433fad4 rename fba2b8a update readme e00d34b fix reading non-zero offsets of different segments 4885e5b add working index b106408 rename cfc462a able to read and write across segments 37146b3 add readme feeb19d add commit log 9f5b28e Just allow 1 log dir for now cf17240 Initial commit

    Automated with GoReleaser Built with go version go1.9.2 darwin/amd64

    Source code(tar.gz)
    Source code(zip)
    jocko_0.0.1_checksums.txt(195 bytes)
    jocko_0.0.1_darwin_amd64.tar.gz(3.75 MB)
    jocko_0.0.1_linux_amd64.tar.gz(3.57 MB)
Owner
Travis Jeffery
Worked at Basecamp, Segment, Confluent on Kafka/Cloud. Author of Distributed Services with Go.
Travis Jeffery
GitHub https://twitter.com/travisjeffery
Micro is a platform for cloud native development

Micro Overview Micro addresses the key requirements for building services in the cloud. It leverages the microservices architecture pattern and provid

Micro 11.5k Dec 31, 2022
High-Performance server for NATS, the cloud native messaging system.

NATS is a simple, secure and performant communications system for digital systems, services and devices. NATS is part of the Cloud Native Computing Fo

NATS - The Cloud Native Messaging System 12k Jan 8, 2023
CockroachDB - the open source, cloud-native distributed SQL database.

CockroachDB is a cloud-native distributed SQL database designed to build, scale, and manage modern, data-intensive applications. What is CockroachDB?

CockroachDB 26.3k Dec 29, 2022
A distributed systems library for Kubernetes deployments built on top of spindle and Cloud Spanner.

hedge A library built on top of spindle and Cloud Spanner that provides rudimentary distributed computing facilities to Kubernetes deployments. Featur

null 21 Nov 9, 2022
A distributed locking library built on top of Cloud Spanner and TrueTime.

A distributed locking library built on top of Cloud Spanner and TrueTime.

null 47 Sep 13, 2022
A library built to provide support for defining service health for golang services. It allows you to register async health checks for your dependencies and the service itself, provides a health endpoint that exposes their status, and health metrics.

go-sundheit A library built to provide support for defining service health for golang services. It allows you to register async health checks for your

AppsFlyer 505 Dec 27, 2022
A distributed MySQL binlog storage system built on Raft

What is kingbus? 中文 Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the sl

Fei Chen 856 Dec 31, 2022
The lightweight, distributed relational database built on SQLite

rqlite is a lightweight, distributed relational database, which uses SQLite as its storage engine. Forming a cluster is very straightforward, it grace

rqlite 12.9k Jan 2, 2023
JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native

Juicedata, Inc 7.2k Jan 4, 2023
Dapr is a portable, event-driven, runtime for building distributed applications across cloud and edge.

Dapr is a portable, serverless, event-driven runtime that makes it easy for developers to build resilient, stateless and stateful microservices that run on the cloud and edge and embraces the diversity of languages and developer frameworks.

Dapr 20.2k Jan 5, 2023
SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, to store and serve billions of files fast! Blob store has O(1) disk seek, local tiering, cloud tiering. Filer supports cross-cluster active-active replication, Kubernetes, POSIX, S3 API, encryption, Erasure Coding for warm storage, FUSE mount, Hadoop, WebDAV.

SeaweedFS Sponsor SeaweedFS via Patreon SeaweedFS is an independent Apache-licensed open source project with its ongoing development made possible ent

Chris Lu 16.3k Jan 5, 2023
Golang client library for adding support for interacting and monitoring Celery workers, tasks and events.

Celeriac Golang client library for adding support for interacting and monitoring Celery workers and tasks. It provides functionality to place tasks on

Stefan von Cavallar 73 Oct 28, 2022
Consistent hashing with bounded loads in Golang

consistent This library provides a consistent hashing function which simultaneously achieves both uniformity and consistency. For detailed information

Burak Sezer 551 Dec 29, 2022
Compute cluster (HPC) job submission library for Go (#golang) based on the open DRMAA standard.

go-drmaa This is a job submission library for Go (#golang) which is compatible to the DRMAA standard. The Go library is a wrapper around the DRMAA C l

Daniel Gruber 43 Nov 17, 2022
Simple, fast and scalable golang rpc library for high load

gorpc Simple, fast and scalable golang RPC library for high load and microservices. Gorpc provides the following features useful for highly loaded pro

Aliaksandr Valialkin 667 Dec 19, 2022
Hprose is a cross-language RPC. This project is Hprose for Golang.

Hprose 3.0 for Golang Introduction Hprose is a High Performance Remote Object Service Engine. It is a modern, lightweight, cross-language, cross-platf

Hprose 1.2k Dec 26, 2022
Golang implementation of the Raft consensus protocol

raft raft is a Go library that manages a replicated log and can be used with an FSM to manage replicated state machines. It is a library for providing

HashiCorp 6.7k Jan 9, 2023
The pure golang implementation of nanomsg (version 1, frozen)

mangos NOTE: This is the legacy version of mangos (v1). Users are encouraged to use mangos v2 instead if possible. No further development is taking pl

nanomsg 1.5k Dec 7, 2022
A Golang implementation of the Umee network, a decentralized universal capital facility in the Cosmos ecosystem.

Umee A Golang implementation of the Umee network, a decentralized universal capital facility in the Cosmos ecosystem. Umee is a Universal Capital Faci

null 169 Jan 3, 2023