A distributed MySQL binlog storage system built on Raft

Overview

Go Report Card Build Status

What is kingbus? 中文

Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the slaves in the same way as an intermediate MySQL master does. Kingbus has the following key features:

  • MySQL replication protocol compatibility, pull the binlog files from the master through gtid mode, and push the binlog file to slave through gtid mode in the same way.

  • Geo-Replication, kingbus uses Raft to support Geo-Replication. The binlog data written to the cluster is guaranteed to be consistent between multiple nodes, and the order of binlog event is exactly the same as that on the master.

  • High availability, your mysql binlog replication is always on and continuously available with kingbus.

Why need kingbus?

In a traditional MySQL replication setup a single master server is created and a set of slaves of MySQL servers are configured to pull the binlog files from the master, putting a lot of load on the master.

  • Introducing a layer between the master server and the slave servers can reduce the load on the master by only serving kingbus instead of all the slaves.

  • The slaves will only need to be aware of kingbus and not the real master server. Removing the requirement for the slaves to have knowledge of the master also simplifies the process of replacing a failed master within a replication environment.

  • Kingbus allows us to horizontally scale our slaves without fear of overloading the network interface of the master

  • Kingbus can also be used to avoide deep nested replication on remote sites, with kingbus you don't need e deeply nested replication.

  • The size of the binlog storage space on the Master can be reduced, and store the binlog in kingbus.

  • Support MYSQL database heterogeneous log based replication. Other heterogeneous replication components can be connected to the kingbus, such as canal.

For more usage scenarios of binlog server, please refer:

Quick start

Read the Quick Start

Documentation

1.Kingbus management API introduction

2.Start a kingbus cluster with docker-compose

License

Kingbus is under the Apache 2.0 license. See the LICENSES file for details.

Acknowledgments

  • Thanks etcd for providing raft library.

  • Thanks go-mysql for providing mysql replication protocol parser.

Issues
  • [Question] Howto change kingbus setting when real master mysql is down

    [Question] Howto change kingbus setting when real master mysql is down

    Hi there,

    Sorry, I don't know where I should ask the question.

    I had 3 nodes kingbus and 10 slaves node. Kingbus was binlog server real master A. Master B was a slave master A. Now, we have some issues with master A and we want to change the real master for kingbus cluster. Should we do now? What is the script in the situation?

    +--------+                                                         +----------+
    |master A| ------------replicated----------------------------------| master B |
    +--------+                                                         +----------+
      ^
      |
      |
    
    [ kingbus 1-3] <---------------------replicate [slave 1-10]
    

    Thank you!

    opened by bienkma 2
  • 同步延迟不会被放大吗?

    同步延迟不会被放大吗?

    看了代码跟设计架构,作为一个中间件直接介入到数据库集群中,感觉存在这样几个问题:

    1. 引入运维复杂度跟风险点,维护数据库集群的时候需要再关注kingbus集群的状态。
    2. 同步复杂度增加,同样由于一个raft集群的引入,binlog的传输从之前的点对点变为了点到集群再到点,中间还夹杂着raft一致性保证带来的写放大的问题。即使是内网环境,随着kingbus节点和业务的增长,同步性能与数据安全性的平衡怎么保证?写入量大的场景备份延迟如何保证不能被放大?
    3. kingbus集群leader挂掉,等同于所有的从库需要做一次主从切换,并没有减少切换成本。

    感觉这个东西做个binlog server来备份、分发数据什么的还可以,至少比canel少了一堆组件要部署,但是作为中间件连接master和slave,感觉稍微有些鸡肋啊。

    opened by liipx 1
  • 是否支持增强半同步复制

    是否支持增强半同步复制

    在异步复制架构中,master宕机时,如果部分binlog event未及时发送,会导致事务丢失。同样的,在使用mysqlbinlog工具做binlog server时,也会存在这种问题,并且由于操作系统机制,可能会有更多的情况产生。 请问,该工具能否支持半同步复制,或者是否有额外的措施用以弥补上述缺陷,谢谢!

    opened by dbadylan 0
  • 从库无法获取serverid  导致无法同步

    从库无法获取serverid 导致无法同步

    slave 报错 Last_IO_Errno: 1105 Last_IO_Error: The slave I/O thread stops because a fatal error is encountered when it try to get the value of SERVER_ID variable from master. Error: mysql:sql not support

    kingbus报错

    2019-07-08T15:50:00.397+0800 info mysql/command.go:119 handleQuery sql:SELECT UNIX_TIMESTAMP() 2019-07-08T15:50:00.398+0800 info mysql/command.go:119 handleQuery sql:SHOW VARIABLES LIKE 'SERVER_ID' 2019-07-08T15:50:00.399+0800 debug mysql/command.go:283 show query showVariable: 2019-07-08T15:50:00.399+0800 error mysql/conn.go:151 dispatch error,err:mysql:sql not support,data:[3 83 72 79 87 32 86 65 82 73 65 66 76 69 83 32 76 73 75 69 32 39 83 69 82 86 69 82 95 73 68 39] 2019-07-08T15:50:00.399+0800 debug mysql/conn.go:135 close client connection

    如何解决?

    opened by kamactor 0
  • 关于mysql配置的疑问

    关于mysql配置的疑问

    通过代码发现:syncer获取mysql master的连接配置,是通过raft协议传递的,而不是通过配置的方式,通过代码暴露出的http接口,也没有找到能通过http设置mysql的连接配置的地方,请问该连接配置,怎么通知集群的?

    BinlogServer的启动时通过http接口,用户主动开启,然后等待mysql slave的tcp 连接。

    opened by xkeyideal 0
Releases(0.8.0)
Owner
Fei Chen
Familiar with distributed system architecture,MySQL,Redis,Rocksdb,Kafka,Raft/Paxos etc
Fei Chen
null 1 Feb 3, 2022
Distributed disk storage database based on Raft and Redis protocol.

IceFireDB Distributed disk storage system based on Raft and RESP protocol. High performance Distributed consistency Reliable LSM disk storage Cold and

IceFireDB 898 Aug 7, 2022
The TinyKV course builds a key-value storage system with the Raft consensus algorithm.

The TinyKV Course The TinyKV course builds a key-value storage system with the Raft consensus algorithm. It is inspired by MIT 6.824 and TiKV Project.

jaegerwang 1 Nov 19, 2021
A linearizability distributed database by raft and wisckey.

AlfheimDB A linearizability distributed database by raft and wisckey, which supports redis client. Build This project build by mage, you will need ins

chunming.dong 73 Jul 18, 2022
An implementation of a distributed KV store backed by Raft tolerant of node failures and network partitions 🚣

barge A simple implementation of a consistent, distributed Key:Value store which uses the Raft Concensus Algorithm. This project launches a cluster of

Shehjad Khan 0 Nov 24, 2021
Easy to use Raft library to make your app distributed, highly available and fault-tolerant

An easy to use customizable library to make your Go application Distributed, Highly available, Fault Tolerant etc... using Hashicorp's Raft library wh

Richard Bertok 58 May 29, 2022
Distributed-Services - Distributed Systems with Golang to consequently build a fully-fletched distributed service

Distributed-Services This project is essentially a result of my attempt to under

Hamza Yusuff 6 Jun 1, 2022
A distributed key-value storage system developed by Alibaba Group

Product Overview Tair is fast-access memory (MDB)/persistent (LDB) storage service. Using a high-performance and high-availability distributed cluster

Alibaba 1.9k Aug 3, 2022
BlobStore is a highly reliable,highly available and ultra-large scale distributed storage system

BlobStore Overview Documents Build BlobStore Deploy BlobStore Manage BlobStore License Overview BlobStore is a highly reliable,highly available and ul

CubeFS 15 Jun 30, 2022
JuiceFS is a distributed POSIX file system built on top of Redis and S3.

JuiceFS is a high-performance POSIX file system released under GNU Affero General Public License v3.0. It is specially optimized for the cloud-native

Juicedata, Inc 5.7k Aug 4, 2022
Distributed reliable key-value store for the most critical data of a distributed system

etcd Note: The main branch may be in an unstable or even broken state during development. For stable versions, see releases. etcd is a distributed rel

etcd-io 40.8k Aug 13, 2022
A feature complete and high performance multi-group Raft library in Go.

Dragonboat - A Multi-Group Raft library in Go / 中文版 News 2021-01-20 Dragonboat v3.3 has been released, please check CHANGELOG for all changes. 2020-03

lni 4.3k Aug 9, 2022
Golang implementation of the Raft consensus protocol

raft raft is a Go library that manages a replicated log and can be used with an FSM to manage replicated state machines. It is a library for providing

HashiCorp 6.3k Aug 13, 2022
A naive implementation of Raft consensus algorithm.

This implementation is used to learn/understand the Raft consensus algorithm. The code implements the behaviors shown in Figure 2 of the Raft paper wi

Martin 0 Dec 3, 2021
Raft: a consensus algorithm for managing a replicated log

Raft Consensus Algorithm Raft is a consensus algorithm for managing a replicated

null 0 Dec 20, 2021
This is my implementation of Raft consensus algorithm that I did for own learning.

This is my implementation of Raft consensus algorithm that I did for own learning. Please follow the link to learn more about raft consensus algorithm https://raft.github.io. And Soon, I will be developing same algorithm in Java as well

Umar Tahir 1 Jan 12, 2022
The lightweight, distributed relational database built on SQLite

rqlite is a lightweight, distributed relational database, which uses SQLite as its storage engine. Forming a cluster is very straightforward, it grace

rqlite 10.8k Aug 5, 2022