Gaea is a mysql proxy, it's developed by xiaomi b2c-dev team.

Overview

LICENSE Build Status Go Report Card

简介

Gaea是小米中国区电商研发部研发的基于mysql协议的数据库中间件,目前在小米商城大陆和海外得到广泛使用,包括订单、社区、活动等多个业务。Gaea支持分库分表、sql路由、读写分离等基本特性,更多详细功能可以参照下面的功能列表。其中分库分表方案兼容了mycat和kingshard两个项目的路由方式。Gaea在设计、实现阶段参照了mycat、kingshard和vitess,并使用tidb parser作为内置的sql parser,在此表达诚挚感谢。为了方便使用和学习Gaea,我们也提供了详细的使用和设计文档,也欢迎大家多多参与。

功能列表

基础功能

  • 多集群
  • 多租户
  • SQL透明转发
  • 慢SQL指纹
  • 错误SQL指纹
  • 注解路由
  • 慢日志
  • 读写分离,从库负载均衡
  • 自定义SQL拦截与过滤
  • 连接池
  • 配置热加载
  • IP/IP段白名单
  • 全局序列号

分库、分表功能

  • 分库: 支持mycat分库方式
  • 分表: 支持kingshard分表方式
  • 聚合函数: 支持max、min、sum、count、group by、order by等
  • join: 支持分片表和全局表的join、支持多个分片表但是路由规则相同的join

架构图

gaea架构图

集群部署图

gaea集群部署图

如上图所示, 部署一套gaea-cc和etcd可用来管理多套gaea集群, 负责集群内namespace配置的增删改查. gaea-cc的HTTP接口文档

安装使用

设计与实现

Roadmap

  • 支持配置加密存储,开关
  • 支持执行计划缓存
  • 支持事务追踪
  • 支持二级索引
  • 支持分布式事务
  • 支持平滑的扩容、缩容
  • 后端连接池优化 (按照请求时间排队)

自有开发模块

  • backend
  • cmd
  • log
  • models
  • proxy/plan
  • proxy/router(kingshard路由方式源自kingshard项目本身)
  • proxy/sequence
  • server

外部模块

  • mysql(google vitess、tidb、kingshard都有引入)
  • parser(tidb)
  • stats(google vitess,打点统计)
  • util(混合)

社区

gitter

Gitter

钉钉

钉钉

Issues
  • test: 补齐 dc 直连的单元测试

    test: 补齐 dc 直连的单元测试

    补齐 dc 直连的单元测试,包含 一开始的 initial handshake packet from MariaDB 和之后 Gaea 的回应

    主要变更的文档有三份,如下

    Gaea/backend/direct_connection_test.go Gaea/backend/direct_connection_cn.md Gaea/backend/direct_connection_en.md

    之前 pipeTest 包也进行修改,包含新增函数和部份函数改名

    opened by panhongrainbow 15
  • make时报错

    make时报错

    go: finding gopkg.in/resty.v1 v1.12.0 go: google.golang.org/[email protected]: unrecognized import path "google.golang.org/genproto" (https fetch: Get https://google.golang.org/genproto?go-get=1: dial tcp 216.239.37.1:443: i/o timeout) go: error loading module requirements Makefile:12: recipe for target 'gaea' failed make: *** [gaea] Error 1

    question 
    opened by xzy1586 7
  • feature: 把 Etcd V3 API 整合到 Gaea

    feature: 把 Etcd V3 API 整合到 Gaea

    目前把 Etcd V3 API 整合到 Gaea

    1 保留原有的 Etcd V2 Api,在设定文档 Gaea/etc/gaea.ini 的设定值不变,为 config_type=etcd,这样大家就不用去修改设定值 2 现在设定文档 Gaea/etc/gaea.ini,支援Etcd V3 API 的设定值为 config type=etcdv3

    接下来会处理 gaea cc 也会内置一个可视化的配置管理功能

    opened by panhongrainbow 5
  • gaea-cc在配置两阶段提交时,没考虑proxy返回的错误信息,可能会导致proxy进程崩溃。

    gaea-cc在配置两阶段提交时,没考虑proxy返回的错误信息,可能会导致proxy进程崩溃。

    gaea-cc在配置两阶段提交时,请求 /api/proxy/config/prepare/:namespace 接口后,没有判断接口返回的内容,只判断了请求是否成功,接着直接去请求了/api/proxy/config/commit/:namespace接口,在以下情况会导致proxy进程崩溃。

    如提交一个不存在的分片类型,prepare接口会报 “err: unknown rule type”,但是cc忽略了这个错误直接去请求 commit接口后,proxy通过ReloadNamespaceCommit方法把 Manager.switchIndex置为1,此时 Manager.namespaces下标1为nil(因为prepare接口并没有解析成功刚刚提交的错误分片类型namespace配置),导致Manager.startConnectPoolMetricsTask()方法中的这行代码for nameSpaceName, _ := range m.namespaces[current].namespaces引发错误"invalid memory address or nil pointer dereference"。

    investigate 
    opened by minxinqing 5
  • 请问如何将namespace的json配置数据导入到etcd呢?

    请问如何将namespace的json配置数据导入到etcd呢?

    对etcd不是很熟悉,用这样的命令将etc/namespaces下的json串导入到etcd后

    ./etcdctl put /gaea/file/namespace/test_namespace_1.json "{"name":"test_namespace_1","online":true,"read_only":false,"allowed_dbs":{"test_db":true},"slow_sql_time":"1000","black_sql":[""],"allowed_ip":null,"slices":[{"name":"slice-0","user_name":"root","password":null,"master":"127.0.0.1:3306","slaves":null,"statistic_slaves":null,"capacity":12,"max_capacity":24,"idle_timeout":60}],"shard_rules":[{"db":"test_db","table":"log","type":"date_month","key":"created_at","slices":["slice-0"],"date_range":["201804-201910"]}],"users":[{"user_name":"root","password":"root","namespace":"test_namespace_1","rw_flag":2,"rw_split":1,"other_property":0}],"default_slice":"slice-0","global_sequences":null}"
    

    启动Gaea服务

    bogon:Gaea user$ ./bin/gaea -config etc/gaea.ini
    Build Version Information:Version: 666d59e9f7ea563733299d97260ee43307881d70
    GitRevision: 666d59e9f7ea563733299d97260ee43307881d70
    User: [email protected]
    GolangVersion: go1.13
    BuildStatus: Clean
    BuildTime: 2019-09-16--14:08:20
    

    服务无法启动,也没有相关的报错信息,请问应该如何正确的把文本模式下正常工具的namespace配置导入etcd呢?

    opened by flappyink 5
  • SAVEPOINT 问题

    SAVEPOINT 问题

    'SQLSTATE[HY000]: General error: 1105 unknown error: get plan error, db: test_db, sql: SAVEPOINT LEVEL1, err: parse sql error, sql: SAVEPOINT LEVEL1, err: line 1 column 9 near "SAVEPOINT LEVEL1" The SQL being executed was: SAVEPOINT LEVEL1'

    image

    opened by xieyuhua 3
  • 单机数据库,事务嵌套执行失败

    单机数据库,事务嵌套执行失败

    session error SQL, namespace: tp5_center_namespace_1, sql: SAVEPOINT LEVEL1, cost: 0 ms, err: get plan error, db: test_azmbk_db, sql: SAVEPOINT LEVEL1, err: parse sql error, sql: SAVEPOINT LEVEL1, err: line 1 column 9 near "SAVEPOINT LEVEL1"

    opened by xieyuhua 3
  • 客户端重连后出现连接断开Broken pipe的异常

    客户端重连后出现连接断开Broken pipe的异常

    复现

    启动Gaea,使用MySQL客户端连接,然后重启Gaea,MySQL客户端执行show tables,MySQL客户端出现报错:

    mysql> show tables;
    ERROR 2006 (HY000): MySQL server has gone away
    No connection. Trying to reconnect...
    Connection id:    10007
    Current database: biz
    
    ERROR 2006 (HY000): MySQL server has gone away
    No connection. Trying to reconnect...
    Connection id:    10008
    Current database: biz
    
    ERROR 2006 (HY000): MySQL server has gone away
    mysql>
    
    

    查看Gaea日志,有broken pipe 的错误。

    [2021-05-20 18:58:08] [gaea_proxy] [mi-OptiPlex-7060] [WARN] [800000001] [github.com/XiaoMi/Gaea/proxy/server.(*Manager).RecordBackendSQLMetrics:manager.go:363] backend slow SQL, namespace: test_namespace_1, addr: 127.0.0.1:3308, sql: show tables, cost: 0 ms
    [2021-05-20 18:58:08] [gaea_proxy] [mi-OptiPlex-7060] [NOTICE] [800000001] Ignore broken pipe signal
    [2021-05-20 18:58:08] [gaea_proxy] [mi-OptiPlex-7060] [WARN] [800000001] [github.com/XiaoMi/Gaea/proxy/server.(*Session).Run:session.go:240] Session write response error, connId: 10008, err: Conn 10008: Write(header) failed: write tcp4 127.0.0.1:13306->127.0.0.1:41478: write: broken pipe
    [2021-05-20 18:58:08] [gaea_proxy] [mi-OptiPlex-7060] [DEBUG] [800000001] [github.com/XiaoMi/Gaea/proxy/server.(*Session).Close:session.go:195] client closed, 10008
    

    且出错后,其他查询都无法执行,比如select 1,报错同上。 退出MySQL客户端,重新登录,再次执行show tables,没有报错。

    原因

    抓包看可能是处理ComFieldList的返回包有问题,导致客户端自动断开。在将writeFieldList方法修改,不使用currentEphemeralBuffer后,问题修复。

    opened by igoso 3
  • ./bin/gaea 无法启动

    ./bin/gaea 无法启动

    执行./bin/gaea时候只显示info信息然后退出

    [[email protected] Gaea]# ./bin/gaea -config=etc/gaea.ini
    Build Version Information:Version: be1b26bff56a8dd7ce95a2bd0ee6c3c6b1496ce8-dirty
    GitRevision: be1b26bff56a8dd7ce95a2bd0ee6c3c6b1496ce8-dirty
    User: [email protected]
    GolangVersion: go1.14.12
    BuildStatus: Modified
    BuildTime: 2021-02-24--11:32:51
    
    opened by MEclipse1 3
  • 应用报大量 “resource pool timed out”,大佬帮分析下原因。

    应用报大量 “resource pool timed out”,大佬帮分析下原因。

    昨天应用报大量“SQLSTATE[HY000]: General error: 1105 unknown error: resource pool timed out (SQL: xxxxxxxx”错误,我查看Gaea源码,发现 resource pool timed out 错误在 util\resource_pool.go的第36行定义。

    在这之前的一小段时间应用大量报" SQLSTATE[HY000]: General error: 1105 unknown error: Conn 0: Write(header) failed: write tcp 10.5.1.128:53996->10.0.8.48:3306: write: connection reset by peer (SQL:xxxxxxxxx"错误,在mysql/conn.go的402行发现 Write(header) failed:定义;与此同时Gaea日志中出现几十条"[shazam_proxy] [nicetuan-middle-payment-prod-002] [NOTICE] [900000001] [main.main.func1:main.go:90] Ignore broken pipe signal"。

    重启proxy后,应用恢复正常,而且这些日志只出现在一台应用服务器上(我把应用和proxy部署在同一台机器上),其它7台机器没事。 没弄明白这个问题出现的原因,麻烦大佬帮忙分析下。

    opened by minxinqing 3
  • jetbrain DataGrip connect error

    jetbrain DataGrip connect error

    Connection to @xxx.xxx.xx.xx failed. [08001] Could not create connection to database server. Attempted reconnect 3 times. Giving up. 使用mysql自带客户端和Navicat连接没问题,经过测试发现使用jdbc mysql-connector 5.1的驱动可以,最新的8.0不行

    investigate 
    opened by dwdcth 3
  • prepare类型语句出错

    prepare类型语句出错

    用mysql官网的prepare语句测试 PREPARE stmt1 FROM 'SELECT SQRT(POW(?,2) + POW(?,2)) AS hypotenuse'; SET @a = 3; SET @b = 4; EXECUTE stmt1 USING @a, @b; 报错如下: ERROR 1243 (HY000): Unknown prepared statement handler (stmt1) given to EXECUTE

    opened by smilethensee 1
  • 分库分表场景下,不支持跨分片insert,支持跨分片update和delete,这样是否合理?辛苦指导

    分库分表场景下,不支持跨分片insert,支持跨分片update和delete,这样是否合理?辛苦指导

    //跨分片insert报错 mysql> insert into sbtest(id,k,c,pad) values(101,101,'aaa','aaa'),(102,102,'aaa','aaa'),(103,103,'aaa','aaa'),(104,104,'aaa','aaa'),(105,105,'aaa','aaa'); ERROR 1105 (HY000): unknown error: get plan error, db: dba_shard_test, sql: insert into sbtest(id,k,c,pad) values(101,101,'aaa','aaa'),(102,102,'aaa','aaa'),(103,103,'aaa','aaa'),(104,104,'aaa','aaa'),(105,105,'aaa','aaa'), err: create select plan error: handleInsertValues error: batch insert has cross slice values or no route found

    // 但是支持跨分片update和delete mysql> update sbtest set c='ccc' where k in (101,102,103,104,105); Query OK, 5 rows affected (0.00 sec)

    mysql> explain update sbtest set c='ddd' where k in (101,102,103,104,105); +-------+---------+------------------+------------------------------------------------------+ | type | slice | db | sql | +-------+---------+------------------+------------------------------------------------------+ | shard | slice-0 | dba_shard_test_0 | UPDATE sbtest SET c='ddd' WHERE k IN (101,104) | | shard | slice-0 | dba_shard_test_1 | UPDATE sbtest SET c='ddd' WHERE k IN (102,103) | | shard | slice-1 | dba_shard_test_3 | UPDATE sbtest SET c='ddd' WHERE k IN (105) | +-------+---------+------------------+------------------------------------------------------+ 3 rows in set (0.00 sec)

    mysql> delete from sbtest where k in (101,102,103,104,105); Query OK, 5 rows affected (0.00 sec)

    mysql> explain delete from sbtest where k in (101,102,103,104,105); +-------+---------+------------------+---------------------------------------------+ | type | slice | db | sql | +-------+---------+------------------+---------------------------------------------+ | shard | slice-1 | dba_shard_test_3 | DELETE FROM sbtest WHERE k IN (105) | | shard | slice-0 | dba_shard_test_1 | DELETE FROM sbtest WHERE k IN (102,103) | | shard | slice-0 | dba_shard_test_0 | DELETE FROM sbtest WHERE k IN (101,104) | +-------+---------+------------------+---------------------------------------------+ 3 rows in set (0.00 sec)

    opened by ximenluffy 0
  • 分片和非分片均不支持 with rollup 语句

    分片和非分片均不支持 with rollup 语句

    mysql> select id from t1 group by k with rollup ; ERROR 1105 (HY000): unknown error: get plan error, db: dba_shard_test, sql: select id from t1 group by k with rollup, err: parse sql error, sql: select id from t1 group by k with rollup, err: line 1 column 43 near "with rollup"

    opened by ximenluffy 0
  • 考虑使用 podman 进行单元测试 (consider using Podman for unit test.)

    考虑使用 podman 进行单元测试 (consider using Podman for unit test.)

    小米您好

    经您建议,去參考 soar 计划的单元测试,在 soar 会真的启动一个容器去进行测试,每个测试都有不同的优点 (You suggest that I should refer soar project when I make unit test. I think many unit tests have many different advantages)

    后来我想想,不如利用这次机会,把容器测试做到最好 (I consider making a more efficient unit test by this chance.)

    我去请教资深的 SRE ,未来容器的走向,他是回说未来会使用 containerd (I consult for senior SRE about the future of containers. He suggests using containerd.)

    但是后来接收到更多资讯,经过思考,现在以我的立场,最适合使用 podman,因为以我的立场,要追求的是最佳性能,我根本不需要去关心云生态目前的走向 (I think about it over and over again. Podman is more suitable for my situation. It is because I want to test fast. I don't need to mind the current container ecosystem.)

    以 SRE 的立场来说,SRE 追求的是和目前的云生态相容,所以他会建议使用 containerd,其实建议也是对的,就怪我没有想清楚立场 (SRE goals for compatible with the current container ecosystem, so he suggests using containerd. It is correct. My fault is not appropriately considering it on my side.)

    再来,我最新的 PR ,封装也封装的不好,这些问题都要改进 (In addition to my newest PR, the encapsulation in my code is not good quality. It is a problem I need to deal with.)

    如果我使用最小的容器,加上 Padman,也许会有很好的测试体验,我再努力看看,谢谢 (What if I use smaller consider and Podman? It will have more practical experience. Let us try it. Thank you.)

    opened by panhongrainbow 2
  • 使用java 8.0版本的driver连接gaea,后端mysql版本低于5.7.20时,报Unknown system variable 'transaction_isolation'

    使用java 8.0版本的driver连接gaea,后端mysql版本低于5.7.20时,报Unknown system variable 'transaction_isolation'

    java驱动版本:mysql-connector-java-8.0.11.jar mysql版本:5.7.11 报错:java.sql.SQLException: Unknown system variable 'transaction_isolation'

    初步判断是mysql版本问题,隔离级别参数说明:

    Note transaction_isolation was added in MySQL 5.7.20 as a synonym for tx_isolation, which is now deprecated and is removed in MySQL 8.0. Applications should be adjusted to use transaction_isolation in preference to tx_isolation.

    MySQL 5.7.11中的确不支持这个参数:

    mysql> select version();
    +--------------+
    | version()    |
    +--------------+
    | 5.7.11-4-log |
    +--------------+
    1 row in set (0.00 sec)
    
    mysql> set transaction_isolation='READ-COMMITTED';
    ERROR 1193 (HY000): Unknown system variable 'transaction_isolation'
    

    但是同样的driver直连mysql不报错,即使mysql版本不支持transaction_isolation。

    另外,测试其他版本的driver,则不报错,如:mysql-connector-java-6.0.5.jar

    综上:触发报错的条件是:

    1. driver 版本是8.0
    2. mysql 版本低于5.7.20
    opened by wangxj1986 0
  • 将试着做出最小的 docker 容器给 Gaea 进行测试 (Provide a smaller size of container for Gaea usage)

    将试着做出最小的 docker 容器给 Gaea 进行测试 (Provide a smaller size of container for Gaea usage)

    小米您好

    我试着做出最小的数据库容器给 Gaea 进行测试,想辨法把容器的大小压到比官方容器还要小,目前找到关键所在 (I am struggling with creating a smaller database container for Gaea usage. There is a critical point.)

    关键在官方把 mariadb 的 lib 全部清除,也是这个原因,我之前无法做出比官方容器更小的镜像 (Official MariaDB docker removes all files inside /var/lib/mysql. It was the main reason I had not made the image smaller than the Official before.)

    大家会使用官方镜像去做为基底,所以多多少少,为了大家方便,官方镜像无法做到太小 (People use the official docker as a base image. It will avoid the official docker being too tiny to have trouble being the base image.)

    我是觉得为了测试方便和效率,有必要把容器做到尽量小 (It is necessary to get a smaller image size to get more efficiency.)

    圖片

    opened by panhongrainbow 1
Releases(v1.2.5)
Owner
Xiaomi
Xiaomi
mysql to mysql 轻量级多线程的库表数据同步

goMysqlSync golang mysql to mysql 轻量级多线程库表级数据同步 测试运行 设置当前binlog位置并且开始运行 go run main.go -position mysql-bin.000001 1 1619431429 查询当前binlog位置,参数n为秒数,查询结

null 13 Jun 14, 2022
A high-performance MySQL proxy

kingshard 中文主页 Overview kingshard is a high-performance proxy for MySQL powered by Go. Just like other mysql proxies, you can use it to split the read

Fei Chen 6k Jun 28, 2022
MySQL proxy backups check recovery

一 、前言 感谢kingshard明星开源项目,本服务正是基于kingshard 开发而来。 本服务适用于相对封闭且经常断电的环境 针对此场景建议使用 MyISAM引擎。 在生产环境中我们数据库可能会遭遇各种各样的不测从而导致数据丢失,大概分为以下几种: 硬件故障 软件故障(目前生产环境经常发生的)

null 4 Oct 29, 2021
Goproxy4mysql - Kingshard- a high-performance proxy for MySQL powered by Go

kingshard 中文主页 Fork from github.com/flike/kingshard Overview kingshard is a high

null 1 Jan 7, 2022
a powerful mysql toolset with Go

go-mysql A pure go library to handle MySQL network protocol and replication. Call for Committer/Maintainer Sorry that I have no enough time to maintai

siddontang 3.7k Jun 24, 2022
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

siddontang 3.8k Jun 25, 2022
Golang MySql binary log replication listener

Go MySql binary log replication listener Pure Go Implementation of MySQL replication protocol. This allow you to receive event like insert, update, de

Pavel <Ven> Gulbin 185 Apr 13, 2022
MySQL replication topology management and HA

orchestrator [Documentation] orchestrator is a MySQL high availability and replication management tool, runs as a service and provides command line ac

null 4.6k Jun 28, 2022
Vitess is a database clustering system for horizontal scaling of MySQL.

Vitess Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vite

Vitess 14.3k Jun 26, 2022
db-recovery is a tool for recovering MySQL data.

db-recovery is a tool for recovering MySQL data. It is used in scenarios where the database has no backup or binlog. It can parse data files and redo/undo logs to recover data.

null 22 Jun 13, 2022
一个使 mysql,pgsql 数据库表自动生成 go struct 的工具

db2go 一个使 mysql、pgsql 数据库表自动生成 go struct 的工具 快速使用 将项目放入到GOPATH/src目录下

易水韩 18 Jun 1, 2022
🐳 A most popular sql audit platform for mysql

?? A most popular sql audit platform for mysql

Henry Yee 6.5k Jun 21, 2022
Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

?? Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

PingCAP 260 Jun 10, 2022
Vitess is a database clustering system for horizontal scaling of MySQL.

Vitess Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vite

Vitess 14.3k Jun 29, 2022
GitHub's Online Schema Migrations for MySQL

gh-ost GitHub's online schema migration for MySQL gh-ost is a triggerless online schema migration solution for MySQL. It is testable and provides paus

GitHub 10.1k Jun 22, 2022
Bifrost ---- 面向生产环境的 MySQL 同步到Redis,MongoDB,ClickHouse,MySQL等服务的异构中间件

Bifrost ---- 面向生产环境的 MySQL 同步到Redis,ClickHouse等服务的异构中间件 English 漫威里的彩虹桥可以将 雷神 送到 阿斯加德 和 地球 而这个 Bifrost 可以将 你 MySQL 里的数据 全量 , 实时的同步到 : Redis MongoDB Cl

brokerCAP 1.2k Jun 19, 2022
go mysql driver, support distributed transaction

Go-MySQL-Driver A MySQL-Driver for Go's database/sql package Features Requirements Installation Usage DSN (Data Source Name) Password Protocol Address

Open Transaction 33 Apr 3, 2022
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

go-mysql-server go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice.

DoltHub 817 Jun 28, 2022