Bifrost ---- 面向生产环境的 MySQL 同步到Redis,MongoDB,ClickHouse,MySQL等服务的异构中间件

Overview

Bifrost ---- 面向生产环境的 MySQL 同步到Redis,ClickHouse等服务的异构中间件

Gitter Build Status License

English

漫威里的彩虹桥可以将 雷神 送到 阿斯加德 和 地球

而这个 Bifrost 可以将 你 MySQL 里的数据 全量 , 实时的同步到 :


WIKI : https://wiki.xbifrost.com


Bifrost 特点

  • 支持 MySQL 所有存储类型

  • 界面化动态灵活配置不同的数据表及目标库

  • 多数据源,多种目标库支持

  • 同时支持 增量全量 数据同步

  • 一个Binlog解析线程,多个目标库并行同步

  • 宕机后重启自动恢复配置,位点精确控制

  • 支持 Email 及 微信企业号等监控报警功能

  • 提供第三方监控平台访问接口(Http Basic验证)https://wiki.xbifrost.com/api/http_api/

  • MySQL,ClickHouse 插件支持 DDL 同步

  • 简单即可实现插件开发

源码下载

Github , Gitee

你的点击 star 就是对 Bifrost 最大的支持!!!

Bifrost在线界面体验

https://www.xbifrost.com/demo/goto.html

用户名: Bifrost

密码 : Bifrost123

image

Bifrsot Diagram

image

MySQL 版本(经过测试)

  • 5.1.73

  • 5.5.42

  • 5.6.36

  • 5.7.18

  • 8.0.21 (用户密码验证需要采用 mysql_native_password 方式)

安装

编译
git clone https://github.com/brokercap/Bifrost.git

make install prefix=./target
二进制文件安装
wget https://github.com/brokercap/Bifrost/releases/download/v1.7.4-release/bifrost_v1.7.4-release_Linux-amd64-bin.tar.gz

tar -zxvf bifrost_v1.7.4-release_Linux-amd64-bin.tar.gz

cd bifrost_v1.7.4-release_Linux-amd64-bin/bin && chmod a+x ./Bifrost*
启动
./Bifrost-server start
停止
./Bifrost-server stop
界面管理

https://127.0.0.1:21036

用户名和密码是在 etc/Bifrost.ini 配置文件中配置的

默认用户名和密码

用户名:Bifrost

密码:Bifrost123

Docker启动
docker pull jc3wish/bifrost

mkdir -p /data/BifrostData

docker run -d -p21036:21036 -v /etc/localtime:/etc/localtime -v /data/BifrostData:/linux/data jc3wish/bifrost
配置
vim Bifrost.ini

[user]
#登入用户名和密码
#用户名在前,= 号后面是密码
Bifrost=Bifrost123
BifrostMonitor=Bifrost123

[groups]
#administrator 管理人员可以添加删除等操作
#monitor 只能查看数据
Bifrost=administrator
BifrostMonitor=monitor

[Bifrostd]
#log写到入的文件夹,默认为启动文件所在文件夹下的logs目录
#log_dir=/data/log/

#监听端口,默认 0.0.0.0:21036
listen=0.0.0.0:21036

#通过界面配置的信息保存文件夹,默认为启动文件所在文件夹下的data目录
data_dir=./

#设置同步给第三方服务的队列大小,默认5000
toserver_queue_size=10000

#通道队列的大小,默认1000
channel_queue_size=1000

#是否支持动态加plugin so 插件,默认为false
#这个参数只在 linux 下有效,其他平台全部为false
dynamic_plugin=false

#是否支持https
tls=true

#server.key 及 server.crt 相对于 Bifrost 的路径,也可以填绝对路径
tls_key_file=./etc/server.key
tls_crt_file=./etc/server.crt

#是否开启文件队列功能 true|false
file_queue_usable=true

#统计是否启文件队列的时间,单位毫秒
file_queue_usable_count_time_diff=5000

#file_queue_usable_count_time_diff 时间内内存队列被挤满的次数
file_queue_usable_count=10

#采用什么方式存储,默认文件方式
meta_storage_type=redis

#存储介质的连接地址
meta_storage_path=127.0.0.1:6379

#用于区别实例的名字
cluster_name=bifrostTestClusterName

#是否开启文件队列功能 true|false
file_queue_usable=true

#统计是否启文件队列的时间,单位毫秒
file_queue_usable_count_time_diff=5000

#file_queue_usable_count_time_diff 时间内内存队列被挤满的次数
file_queue_usable_count=10

#在没有数据的情况下,间隔多久提交一次插件,单位 秒
plugin_commit_timeout=5

#在同步出错的情况下,每2次重试之后 间隔多久再重试 ,单位 秒
plugin_sync_retry_time=5
生成https证书

参考证书生成

感谢相关依赖包提供者

  • github.com/gmallard/stompngo
  • github.com/hprose/hprose-golang
  • github.com/Shopify/sarama
  • github.com/bradfitz/gomemcache/memcache
  • gopkg.in/mgo.v2
  • github.com/streadway/amqp
  • github.com/go-redis/redis
  • github.com/syndtr/goleveldb/leveldb
  • github.com/ClickHouse/clickhouse-go

QQ群号: 70331601

qq群

捐赠

Issues
  • 请教一下Biforst同步到ClickHouse的问题

    请教一下Biforst同步到ClickHouse的问题

    1.请问对于update delete

    是使用这种方式处理的吗吗 https://mp.weixin.qq.com/s/a8OfsBn9VFnj7oxp0IIVGg ReplacingMergeTree + 视图(argMax)? + is_delete列?

    2.全量数据拉取完成后, 咱们这边是如何知道从哪个点消费binlog做增量呢? 是类似逻辑备份的原理吗

    FLUSH /*!40101 LOCAL */ TABLES ' # 对整个实例加全局读锁,如果存在表锁将阻塞加全局读锁语句'

    FLUSH TABLES WITH READ LOCK ' #在session级别修改隔离级别为RR'

    SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ '# 开启一个一致性快照事务,必须在隔离级别RR下才能开启一个快照事务'

    START TRANSACTION /*!40100 WITH CONSISTENT SNAPSHOT */ '#查看是否开启GTID'

    SHOW VARIABLES LIKE 'gtid_mode' '#如果开启GTID则查看当前的事务GTID集合'

    SELECT @@GLOBAL.GTID_EXECUTED '#查看当前数据的binlog pos'

    SHOW MASTER STATUS
    '#释放全局读锁'

    UNLOCK TABLES

    开始对指定表执行SELECT获取全量数据 FTWRL 目的获取binlog file pos 或 GTID. 开启RR级别保证后续所有表的查询结果都是在一个一致性的视图中获取 增量同步时是根据binlog file pos 还是 GTID呢?

    3.写入clickhouse是否使用batch insert

    对于OLTP数据库 DML可能是频繁地小事务, clickhouse建议batch insert, 避免too many part合并不过来, 咱们这边对源端产生的语句转换为INSERT后有做缓存然后再Batch insert写入clickhouse吗?还是就是源端来1条clickhouse就插一次 比如mysql执行1000个insert , clickhouse是执行1000个insert还是1个insert values1000个?

    4.MySQL源端DDL

    • DDL如何处理? 尤其是使用PT-OST GH-OST等类似工具时

    工具原理是创建影子表, DDL应用在影子表上, 全量增量数据同步后再和源表做切换, 那么消费binlog就不能只是消费源表的binlog还要消费影子表的binlog(至少要消费影子表的ALTER语句)

    • 加索引. 加索引clickhouse会自行忽略吗
    opened by Fanduzi 4
  • add es plugin

    add es plugin

    插件文档: image

    参数配置: image

    checkurl: image

    同步结果: image

    插件自动创建如下mapping (自动发现/转化日期时间,支持毫秒)

    { "mappings": { "properties": {}, "date_detection": true, "dynamic_date_formats": [ "yyyy-MM-dd", "yyyy-MM-dd HH:mm:ss", "yyyy-MM-dd HH:mm:ss.S", "yyyy-MM-dd HH:mm:ss.SS", "yyyy-MM-dd HH:mm:ss.SSS", "yyyy-MM-dd HH:mm:ss.SSSS", "yyyy-MM-dd HH:mm:ss.SSSSS", "yyyy-MM-dd HH:mm:ss.SSSSSS" ] } }

    opened by zhwei820 4
  • 能否只允许insert操作

    能否只允许insert操作

    设想是用mysql做中间库,通过bifrost入库到clickhouse 从mysql同步到clickhouse的时候,需要定时删除mysql的数据防止数据源过大。 这样就需要bifrost忽略掉delete的操作,请问可以设置吗?

    opened by aloasut 2
  • 增加clickhouse 本地表、分布式表,视图等。支持集群模式分布式写入.。同步DDl等功能

    增加clickhouse 本地表、分布式表,视图等。支持集群模式分布式写入.。同步DDl等功能

    1. 增加clickhouse 本地表、分布式表,视图等。支持集群模式分布式写入
    2. 支持在线DDL同步。包含:增加字段,删除字段,修改字段类型,修改字段名称,修改表名称等。
    3. 同步界面增加集群,单机模式可选 ddl同步程度可选
    4. 兼容原单机模式 同时增加单机模式下的 同步DDL功能
    5. 由于涉及到final 效率问题,推荐ck版本为 v20.5.2.7-stable以上(包含)。
    opened by zygitup 2
  • redis 插件

    redis 插件

    redis 同步插件 类型完善

    支持的类型

    String : SET 命令写入数据

    List : LPUSH 命令写入数据

    Hash : HSET 命令写入数据

    Zset : ZSET 命令写入数据

    SET : SADD 命令写入数据

    opened by Ryan-gsq 2
  • nginx 配置外部域名,无法访问

    nginx 配置外部域名,无法访问

    您好,请教个问题。 docker 启动之后,证书也有做相应替换。 IP+端口 是可以访问的。 配置域名之后,https访问一直失败,400错误。 请问下这是什么问题。

    opened by kwz1993 2
  • mysql to clickhouse 中文乱码

    mysql to clickhouse 中文乱码

    mysql库的字符集是gbk,源库支持设置字符集?

    opened by GiftLee 2
  • SUPER

    SUPER

    为什么mysql数据库需要super权限 这样很多云数据库拿不到

    opened by junsonw 2
  • 同步到目标库的队列满了情况下,也不阻塞其他数据表同步

    同步到目标库的队列满了情况下,也不阻塞其他数据表同步

    Bifrost 每个表的每一个目标库同步都是相互独立的,都各自有一个队列,这个队列大小可以通过 toserver_queue_size 来配置

    但是每个队列 是保存在内存中的,假如一个表设置了同时配置到3个目标库

    在3个目标库中同步有,一个目标库出现异常了, 一直同步失败,那队列满了的情况下,会影响其他目标库也同步不了

    这里需要在这种队列满了的情况下,将新数据到磁盘队列中去,做到真正不阻塞的方式同步

    这里要考滤到解析很快,所有的同步都 慢的情况下,这里就没必要刷到磁盘了,还不如阻塞 在那里,所以这里,应该提供在线设置,某一个同步,是否启用磁盘队列

    enhancement 
    opened by jc3wish 1
  • binlog 不消费,web 后台显示running , log 最后一行显示 已经关闭 mysql 连接

    binlog 不消费,web 后台显示running , log 最后一行显示 已经关闭 mysql 连接

    日志文件

    | 2020/10/10 02:00:56 jw monitor: parseEvent err:runtime error: index out of range [-1] sync_1 | 2020/10/10 02:00:56 jw monitor: running [email protected]:~/docker/bifrost# docker-compose logs --tail=100 sync Attaching to bifrost_sync_1 sync_1 | created by github.com/brokercap/Bifrost/server.(*db).Start sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/server/db.go:467 +0x56e sync_1 | sync_1 | 2020/10/10 02:16:32 jw monitor: parseEvent err:runtime error: index out of range [-1] sync_1 | 2020/10/10 02:16:32 jw monitor: running sync_1 | 2020/10/10 02:16:37 jw monitor: starting sync_1 | 2020/10/10 02:16:37 jw monitor: running sync_1 | 2020/10/10 02:16:37 binlog_checksum: true sync_1 | 2020/10/10 02:16:37 root:[email protected](mysql:3306)/new_julietswhisper ROTATE_EVENT mysql-bin.000005 0 sync_1 | 2020/10/10 02:16:37 goroutine 2443 [running]: sync_1 | runtime/debug.Stack(0xc00036d110, 0x26, 0x0) sync_1 | /usr/local/Cellar/[email protected]/1.14.9/libexec/src/runtime/debug/stack.go:24 +0x9d sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlog.func2.1(0xc00014fd48) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:578 +0xcb sync_1 | panic(0xc617e0, 0xc000553ac0) sync_1 | /usr/local/Cellar/[email protected]/1.14.9/libexec/src/runtime/panic.go:969 +0x166 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*eventParser).parseEventRow(0xc0000796c0, 0xc0004f5d10, 0xc0003e11e0, 0xc00034cf40, 0x7, 0x8, 0x0, 0x0, 0x0) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/event_row.go:379 +0x526b sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*eventParser).parseRowsEvent(0xc0000796c0, 0xc0004f5d10, 0xe5, 0x10000c0001800f8, 0x0) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/event_row.go:65 +0x381 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*eventParser).parseEvent(0xc0000796c0, 0xc0001aaa01, 0x41, 0x41, 0x0, 0x42, 0x0, 0xc0001aaa00, 0x0) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:181 +0x935 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlog.func2(0xc00014fd48, 0xc0000796c0, 0xc0001aaa00, 0x42, 0x42, 0xc00014fd38) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:581 +0xa0 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlog(0xc000d157a0, 0xc0001495c0, 0x10, 0x8ec3b3, 0xc0000796c0, 0xc000259330, 0xc00034a300, 0x0, 0x0, 0x0, ...) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:582 +0x3fa sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*BinlogDump).startConnAndDumpBinlog(0xc0001ae420, 0xc00034a300) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:863 +0x3d5 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*BinlogDump).StartDumpBinlog(0xc0001ae420, 0xc0003575a0, 0x10, 0xca008e4084, 0xc00034a300, 0x0, 0x0, 0x0) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:741 +0x464 sync_1 | created by github.com/brokercap/Bifrost/server.(*db).Start sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/server/db.go:467 +0x56e sync_1 | sync_1 | 2020/10/10 02:16:37 jw monitor: parseEvent err:runtime error: index out of range [-1] sync_1 | 2020/10/10 02:16:37 jw monitor: running sync_1 | parseEvent err:runtime error: index out of range [-1] sync_1 | 2020/10/10 02:16:41 This.mysqlConn close ,connectionId: 329032 sync_1 | 2020/10/10 02:16:42 jw monitor: starting sync_1 | 2020/10/10 02:16:42 jw monitor: running sync_1 | 2020/10/10 02:16:42 binlog_checksum: true sync_1 | 2020/10/10 02:16:42 root:[email protected](mysql:3306)/new_julietswhisper ROTATE_EVENT mysql-bin.000005 0 sync_1 | 2020/10/10 02:16:42 goroutine 2443 [running]: sync_1 | runtime/debug.Stack(0xc00030e6f0, 0x26, 0x0) sync_1 | /usr/local/Cellar/[email protected]/1.14.9/libexec/src/runtime/debug/stack.go:24 +0x9d sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlog.func2.1(0xc00014fd48) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:578 +0xcb sync_1 | panic(0xc617e0, 0xc000ac6660) sync_1 | /usr/local/Cellar/[email protected]/1.14.9/libexec/src/runtime/panic.go:969 +0x166 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*eventParser).parseEventRow(0xc0000796c0, 0xc000411170, 0xc0000cc9a0, 0xc000a3b080, 0x7, 0x8, 0x0, 0x0, 0x0) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/event_row.go:379 +0x526b sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*eventParser).parseRowsEvent(0xc0000796c0, 0xc000411170, 0xe5, 0x10000c0001800f8, 0x0) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/event_row.go:65 +0x381 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*eventParser).parseEvent(0xc0000796c0, 0xc0002246e1, 0x41, 0x41, 0x0, 0x42, 0x0, 0xc0002246e0, 0x0) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:181 +0x935 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlog.func2(0xc00014fd48, 0xc0000796c0, 0xc0002246e0, 0x42, 0x42, 0xc00014fd38) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:581 +0xa0 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlog(0xc000d15e60, 0xc0001495c0, 0x10, 0x8ec3b3, 0xc0000796c0, 0xc000259330, 0xc00034a300, 0x0, 0x0, 0x0, ...) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:582 +0x3fa sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*BinlogDump).startConnAndDumpBinlog(0xc0001ae420, 0xc00034a300) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:863 +0x3d5 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*BinlogDump).StartDumpBinlog(0xc0001ae420, 0xc0003575a0, 0x10, 0xca008e4084, 0xc00034a300, 0x0, 0x0, 0x0) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:741 +0x464 sync_1 | created by github.com/brokercap/Bifrost/server.(*db).Start sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/server/db.go:467 +0x56e sync_1 | sync_1 | 2020/10/10 02:16:42 jw monitor: parseEvent err:runtime error: index out of range [-1] sync_1 | 2020/10/10 02:16:42 jw monitor: running sync_1 | parseEvent err:runtime error: index out of range [-1] sync_1 | 2020/10/10 02:16:47 jw monitor: starting sync_1 | 2020/10/10 02:16:47 jw monitor: running sync_1 | 2020/10/10 02:16:47 binlog_checksum: true sync_1 | 2020/10/10 02:16:47 root:[email protected](mysql:3306)/new_julietswhisper ROTATE_EVENT mysql-bin.000005 0 sync_1 | parseEvent err:runtime error: index out of range [-1] sync_1 | 2020/10/10 02:16:47 goroutine 2443 [running]: sync_1 | runtime/debug.Stack(0xc00036dcb0, 0x26, 0x0) sync_1 | /usr/local/Cellar/[email protected]/1.14.9/libexec/src/runtime/debug/stack.go:24 +0x9d sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlog.func2.1(0xc00014fd48) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:578 +0xcb sync_1 | panic(0xc617e0, 0xc000037fa0) sync_1 | /usr/local/Cellar/[email protected]/1.14.9/libexec/src/runtime/panic.go:969 +0x166 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*eventParser).parseEventRow(0xc0000796c0, 0xc000469920, 0xc0003e1b80, 0xc00034d280, 0x7, 0x8, 0x0, 0x0, 0x0) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/event_row.go:379 +0x526b sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*eventParser).parseRowsEvent(0xc0000796c0, 0xc000469920, 0xe5, 0x10000c0001800f8, 0x0) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/event_row.go:65 +0x381 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*eventParser).parseEvent(0xc0000796c0, 0xc0002ff451, 0x41, 0x41, 0x0, 0x42, 0x0, 0xc0002ff450, 0x0) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:181 +0x935 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlog.func2(0xc00014fd48, 0xc0000796c0, 0xc0002ff450, 0x42, 0x42, 0xc00014fd38) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:581 +0xa0 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*mysqlConn).DumpBinlog(0xc000c1a3c0, 0xc0001495c0, 0x10, 0x8ec3b3, 0xc0000796c0, 0xc000259330, 0xc00034a300, 0x0, 0x0, 0x0, ...) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:582 +0x3fa sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*BinlogDump).startConnAndDumpBinlog(0xc0001ae420, 0xc00034a300) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:863 +0x3d5 sync_1 | github.com/brokercap/Bifrost/Bristol/mysql.(*BinlogDump).StartDumpBinlog(0xc0001ae420, 0xc0003575a0, 0x10, 0xca008e4084, 0xc00034a300, 0x0, 0x0, 0x0) sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/Bristol/mysql/binlog.go:741 +0x464 sync_1 | created by github.com/brokercap/Bifrost/server.(*db).Start sync_1 | /Users/apple/Desktop/WorkSpae/go/Bifrost/server/db.go:467 +0x56e sync_1 | sync_1 | 2020/10/10 02:16:47 jw monitor: parseEvent err:runtime error: index out of range [-1] sync_1 | 2020/10/10 02:16:47 jw monitor: running sync_1 | 2020/10/10 02:16:51 This.mysqlConn close ,connectionId: 329034

    后台状态

    image

    补充说明

    1. 后台日志显示mysql 断开连接后没有其他日志产生了, 且web后台显示的还是running 状态 (会重新连接, 日志没打印全)
    2. binlog 已经不消费了, 一直重复这个错误 大概15s 出现一次 重启也没用, 除非我跳过这个位点
    3. Bristol/mysql/event_row.go:379 +0x526b 该错误产生频率较高,不太了解mysql binlog 的数据包结构, 这个是属于正常情况吗
    4. go version go1.15.2 linux/amd64
    opened by Ryan-gsq 1
  • json字段为null 同步会有问题

    json字段为null 同步会有问题

    提示 parseEvent err:Json length: 0 is larger than packet length -1

    opened by junsonw 1
  • 同步报错

    同步报错

    Version:

    • Bifrost Version: 1.8.0
    • Os Version: centos7

    Not enough Arguments to call STMT_EXEC (Got: 22 Has: 48 | {1631870965 update [map[answer:{"2":"3","1":"1","0":"0"} id:219268530 itemId:0 pId:0 questionId:37564229 score:0 status:none teacherSay: testId:715568 testPaperResultId:5679658 userId:453457] map[answer:{"2":"3","1":"1","0":"0"} id:219268530 itemId:0 pId:0 questionId:37564229 score:2 status:right teacherSay: testId:715568 testPaperResultId:5679658 userId:453457]] edusoho testpaper_item_result 3658 458852069 [id] 416285 map[answer:Nullable(text) id:uint32 itemId:uint32 pId:uint32 questionId:uint32 score:float(10,1) status:enum('none','right','partRight','wrong','noAnswer') teacherSay:Nullable(text) testId:uint32 testPaperResultId:uint32 userId:uint32]} -- | --

    opened by sunsunzhe 0
  • 同步数据报错

    同步数据报错

    同步mysql的数据到Clickhouse 报错: This.err: driver: bad connection This.err: driver: bad connection This.err: driver: bad connection This.err: driver: bad connection

    大佬们如何处理?

    opened by wpepec 0
  • 更新频繁的表,同步时间太长

    更新频繁的表,同步时间太长

    Version:

    • Bifrost Version: 1.7.4
    • Os Version: CentOS 8

    部署了一套mysql 到clickhouse的同步, 有一个更新很较为频繁的表, 大概一秒钟5次左右的update或者insert 从文档上来看, 应该是每5s会同步一次, 但是这个表每次要3分钟左右才会同步(batchSize设置是1000, 估计是要满足1000条才会更新)

    现在只能是设置batchSize为50勉强使用... 请问有解决办法吗?

    opened by ppyupp 0
  • 2进制部署后,登录提示user error无法登录

    2进制部署后,登录提示user error无法登录

    [user] #管理界面的用户名及密码 Bifrost=Bifrost123

    [groups] #administrator 管理人员可以添加删除等操作 #monitor 只能查看数据 Bifrost=administrator BifrostMonitor=monitor

    opened by cloudcache 2
  • parseEvent err:parseEvent err recover err:runtime error: index out of range [7] with length 7 ;lastMapEvent:*mysql.TableMapEvent ;binlogFileName:mysql-bin.000741 ;binlogPosition:320281610

    parseEvent err:parseEvent err recover err:runtime error: index out of range [7] with length 7 ;lastMapEvent:*mysql.TableMapEvent ;binlogFileName:mysql-bin.000741 ;binlogPosition:320281610

    parseEvent err:parseEvent err recover err:runtime error: index out of range [7] with length 7 ;lastMapEvent:*mysql.TableMapEvent ;binlogFileName:mysql-bin.000741 ;binlogPosition:320281610

    opened by xieyuhua 0
  • 按GTID实现同表数据并行同步

    按GTID实现同表数据并行同步

    背景:

    1. 当前是按配置的一个表多个目标库, 每个目标库各一个协程然后抢占目标库的连接池进行同步的, 在表的数据都比较平均的时候是没有问题的, 同时也取决于目标库的性能
    2. 当整个库就那么1,2个表是热点数据,并且是绝对高并发更新,理论就存在数据同步的延迟

    需求:

    1. 按GTID的算法 单一个表的数据,也允许并行同步
    enhancement 
    opened by jc3wish 0
  • 单测(集成测试)的环境自动部署

    单测(集成测试)的环境自动部署

    背景:

    1. 当前好多单测写成了集成测试代码, 依懒了mysql, clickhouse 等环境

    2. 小伙伴提交合并的代码,并不能快速的知道逻辑有没有大问题,可不可用

    需求:

    1. 希望能docker-compose 一键所有依懒的环境搭建起来, 所有希望环境连接docker 运行

    2. 希望 travis 里, 也能将这种依懒环境的单测跑过

    enhancement 
    opened by jc3wish 0
  • 后台反渲染式插件开发

    后台反渲染式插件开发

    背景:

    1. 当前目标库是插件式开发, 不同目标库 在界面上所需要用户填写的 input 内容 是不一样的, 当前每个目标库插件开发的时候 ,都得写一个html文件及js文件,由 Bifrost 界面去动态加载, 这样对 html 存放是有要求的,并且假如 Bifrost 界面如果做了大改动,那插件的 html 及 js 文件都得做相应的修改,

    2. 假如提供数据源及告警等插件式开发,那界面这块,其实界面这块管理起来,可能随着插件的种类的增多,会没那么管理

    当前思路:

    寻找一套标准,或者自定义一套标准格式,,给每个 输入框 定义一套标准 , 插件开发的时候, 把只要把需要的input , select ,textarea, radio, checkbox 等定义好在一个 json 变量中, Bifrost server 层直接调用插件层的 go 代码即可,,不需要再单独写html

    但有几个问题

    1). 前端的选择不同, 插件要展示的 input 也不一样,比如 clickhouse插件,选择了 追加模式 , 下面的其他的input 可配置的输入框就不一样, 怎么解决?

    希望有相关经验的小伙伴 提提建议!!!

    enhancement 
    opened by jc3wish 0
  • 是否支持特定表特定字段的同步?

    是否支持特定表特定字段的同步?

    是否支持特定表特定字段的同步?

    opened by cloudcache 1
Releases(v1.8.0-beta.01)
ClickHouse http proxy and load balancer

chproxy English | 简体中文 Chproxy, is an http proxy and load balancer for ClickHouse database. It provides the following features: May proxy requests to

Vertamedia 725 Oct 12, 2021
Collects many small inserts to ClickHouse and send in big inserts

ClickHouse-Bulk Simple Yandex ClickHouse insert collector. It collect requests and send to ClickHouse servers. Installation Download binary for you pl

Nikolay Pavlovich 308 Oct 12, 2021
go mysql driver, support distributed transaction

Go-MySQL-Driver A MySQL-Driver for Go's database/sql package Features Requirements Installation Usage DSN (Data Source Name) Password Protocol Address

Open Transaction 26 Sep 22, 2021
MySQL Binlog 增量实时同步工具

简介 go-mysql-transfer是一款MySQL数据库实时增量同步工具。 能够监听MySQL二进制日志(Binlog)的变动,将变更内容形成指定格式的消息,实时发送到接收端。从而在数据库和接收端之间形成一个高性能、低延迟的增量数据同步更新管道。 特性 1、简单,不依赖其它组件,一键部署 2、

wangjie 470 Oct 16, 2021
Jaeger ClickHouse storage plugin implementation

Jaeger ClickHouse Jaeger ClickHouse gRPC storage plugin. This is WIP and it is based on https://github.com/bobrik/jaeger/tree/ivan/clickhouse/plugin/s

Pavol Loffay 0 Aug 11, 2021
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

go-mysql-server go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice.

DoltHub 634 Oct 23, 2021
a powerful mysql toolset with Go

go-mysql A pure go library to handle MySQL network protocol and replication. Call for Committer/Maintainer Sorry that I have no enough time to maintai

siddontang 3.3k Oct 13, 2021
Universal command-line interface for SQL databases

usql A universal command-line interface for PostgreSQL, MySQL, Oracle Database, SQLite3, Microsoft SQL Server, and many other databases including NoSQ

XO 6.8k Oct 16, 2021
Clickhouse support for GORM

clickhouse Clickhouse support for GORM Quick Start package main import ( "fmt" "github.com/sweetpotato0/clickhouse" "gorm.io/gorm" ) // User

null 1 Oct 24, 2021
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

siddontang 3.6k Oct 14, 2021
support clickhouse

Remote storage adapter This is a write adapter that receives samples via Prometheus's remote write protocol and stores them in Graphite, InfluxDB, cli

weetime 8 Aug 30, 2021
MySQL replication topology manager - agent (daemon)

orchestrator-agent MySQL topology agent (daemon) orchestrator-agent is a sub-project of orchestrator. It is a service that runs on MySQL hosts and com

GitHub 51 Oct 23, 2021
write APIs using direct SQL queries with no hassle, let's rethink about SQL

SQLer SQL-er is a tiny portable server enables you to write APIs using SQL query to be executed when anyone hits it, also it enables you to define val

Mohammed Al Ashaal 1.9k Oct 18, 2021
Zero boilerplate database operations for Go

(Now compatible with MySQL and PostgreSQL!) Everyone knows that performing simple DATABASE queries in Go takes numerous lines of code that is often re

null 318 Oct 16, 2021
The MongoDB driver for Go

The MongoDB driver for Go This fork has had a few improvements by ourselves as well as several PR's merged from the original mgo repo that are current

GlobalSign 1.9k Oct 15, 2021
A Go rest API project that is following solid and common principles and is connected to local MySQL database.

This is an intermediate-level go project that running with a project structure optimized RESTful API service in Go. API's of that project is designed based on solid and common principles and connected to the local MySQL database.

Kıvanç Aydoğmuş 12 Sep 23, 2021
Golang MySql binary log replication listener

Go MySql binary log replication listener Pure Go Implementation of MySQL replication protocol. This allow you to receive event like insert, update, de

Pavel <Ven> Gulbin 176 Oct 11, 2021
Interactive terminal user interface and CLI for database connections. MySQL, PostgreSQL. More to come.

?? dbui dbui is the terminal user interface and CLI for database connections. It provides features like, Connect to multiple data sources and instance

Kanan Rahimov 57 Oct 19, 2021