GitHub's Online Schema Migrations for MySQL

Related tags

Database tools mysql schema-migrations
Overview

gh-ost

build status downloads release

GitHub's online schema migration for MySQL

gh-ost is a triggerless online schema migration solution for MySQL. It is testable and provides pausability, dynamic control/reconfiguration, auditing, and many operational perks.

gh-ost produces a light workload on the master throughout the migration, decoupled from the existing workload on the migrated table.

It has been designed based on years of experience with existing solutions, and changes the paradigm of table migrations.

How?

All existing online-schema-change tools operate in similar manner: they create a ghost table in the likeness of your original table, migrate that table while empty, slowly and incrementally copy data from your original table to the ghost table, meanwhile propagating ongoing changes (any INSERT, DELETE, UPDATE applied to your table) to the ghost table. Finally, at the right time, they replace your original table with the ghost table.

gh-ost uses the same pattern. However it differs from all existing tools by not using triggers. We have recognized the triggers to be the source of many limitations and risks.

Instead, gh-ost uses the binary log stream to capture table changes, and asynchronously applies them onto the ghost table. gh-ost takes upon itself some tasks that other tools leave for the database to perform. As result, gh-ost has greater control over the migration process; can truly suspend it; can truly decouple the migration's write load from the master's workload.

In addition, it offers many operational perks that make it safer, trustworthy and fun to use.

gh-ost general flow

Highlights

  • Build your trust in gh-ost by testing it on replicas. gh-ost will issue same flow as it would have on the master, to migrate a table on a replica, without actually replacing the original table, leaving the replica with two tables you can then compare and satisfy yourself that the tool operates correctly. This is how we continuously test gh-ost in production.
  • True pause: when gh-ost throttles, it truly ceases writes on master: no row copies and no ongoing events processing. By throttling, you return your master to its original workload
  • Dynamic control: you can interactively reconfigure gh-ost, even as migration still runs. You may forcibly initiate throttling.
  • Auditing: you may query gh-ost for status. gh-ost listens on unix socket or TCP.
  • Control over cut-over phase: gh-ost can be instructed to postpone what is probably the most critical step: the swap of tables, until such time that you're comfortably available. No need to worry about ETA being outside office hours.
  • External hooks can couple gh-ost with your particular environment.

Please refer to the docs for more information. No, really, read the docs.

Usage

The cheatsheet has it all. You may be interested in invoking gh-ost in various modes:

  • a noop migration (merely testing that the migration is valid and good to go)
  • a real migration, utilizing a replica (the migration runs on the master; gh-ost figures out identities of servers involved. Required mode if your master uses Statement Based Replication)
  • a real migration, run directly on the master (but gh-ost prefers the former)
  • a real migration on a replica (master untouched)
  • a test migration on a replica, the way for you to build trust with gh-ost's operation.

Our tips:

  • Testing above all, try out --test-on-replica first few times. Better yet, make it continuous. We have multiple replicas where we iterate our entire fleet of production tables, migrating them one by one, checksumming the results, verifying migration is good.
  • For each master migration, first issue a noop
  • Then issue the real thing via --execute.

More tips:

  • Use --exact-rowcount for accurate progress indication
  • Use --postpone-cut-over-flag-file to gain control over cut-over timing
  • Get familiar with the interactive commands

Also see:

What's in a name?

Originally this was named gh-osc: GitHub Online Schema Change, in the likes of Facebook online schema change and pt-online-schema-change.

But then a rare genetic mutation happened, and the c transformed into t. And that sent us down the path of trying to figure out a new acronym. gh-ost (pronounce: Ghost), stands for GitHub's Online Schema Transmogrifier/Translator/Transformer/Transfigurator

License

gh-ost is licensed under the MIT license

gh-ost uses 3rd party libraries, each with their own license. These are found here.

Community

gh-ost is released at a stable state, but with mileage to go. We are open to pull requests. Please first discuss your intentions via Issues.

We develop gh-ost at GitHub and for the community. We may have different priorities than others. From time to time we may suggest a contribution that is not on our immediate roadmap but which may appeal to others.

Please see Coding gh-ost for a guide to getting started developing with gh-ost.

Download/binaries/source

gh-ost is now GA and stable.

gh-ost is available in binary format for Linux and Mac OS/X

Download latest release here

gh-ost is a Go project; it is built with Go 1.14 and above. To build on your own, use either:

  • script/build - this is the same build script used by CI hence the authoritative; artifact is ./bin/gh-ost binary.
  • build.sh for building tar.gz artifacts in /tmp/gh-ost

Generally speaking, master branch is stable, but only releases are to be used in production.

Authors

gh-ost is designed, authored, reviewed and tested by the database infrastructure team at GitHub:

Comments
  • Force index in unique key range queries

    Force index in unique key range queries

    Description

    This PR forces the unique-key index to be used when gathering the min/max range of the unique key

    This is to avoid possibilities of the optimizer picking a different index than we planned, discussed in more detail here. cc @morgo / @cyrinux

    In case this PR introduced Go code changes:

    • [x] contributed code is using same conventions as original code
    • [x] script/cibuild returns with no formatting errors, build errors or unit test errors.
    enhancement 
    opened by timvaillancourt 2
  • feat(#1235): add an optimizer switch flag to migration context

    feat(#1235): add an optimizer switch flag to migration context

    Description

    This add a optimizer_switch string flag.

    --optimizer-switch="prefer_ordering_index=on"

    I don't check the value, easier to maintains, this will just fail if its a unknown value, depending the mysql/maria/percona version.

    Related issue: https://github.com/github/gh-ost/issues/1235

    opened by cyrinux 1
  • add the possibility to toggle the optimizer switch during the migration session

    add the possibility to toggle the optimizer switch during the migration session

    Hi,

    We realized we have for some databases the need to be able to set the migration session optimizer switch, for example set prefer_ordering_index=on

    https://dev.mysql.com/doc/refman/8.0/en/switchable-optimizations.html

    I will do a PR for this.

    opened by cyrinux 0
  • The table cut failed and caused the slave library to block

    The table cut failed and caused the slave library to block

    Excuse me, I would like to ask a question. When using gh-ost to execute ddl yesterday, an error occurred in the main library, which caused the table to fail to switch successfully. But there are two errors, and I am not sure which error caused the table to fail to switch successfully.

    2022/12/19 16:33:11 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 root false false } 2022/12/19 16:33:11 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000196, 1004574218) 2022/12/19 16:33:11 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:33:11 binlogsyncer.go:573: [info] rotate to (mysql-bin.000196, 1004574218) Migrating customer3.customer_company; Ghost table is customer3._customer_company_gho Migrating rdsdbn3-sirius-customer-online:3306; inspecting rdsdbn3-sirius-customer-online:3306; executing on rdsdbn3-sirius-customer-online Migration started at Mon Dec 19 16:33:11 +0800 2022 chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: Threads_running=100; critical-load: Threads_running=500; nice-ratio: 0.000000 throttle-additional-flag-file: /tmp/gh-ost.throttle Serving on unix socket: /tmp/gh-ost.customer3.customer_company.sock Serving on TCP port: 13316 Copy: 0/821093 0.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: mysql-bin.000196:1004589539; State: migrating; ETA: N/A Copy: 0/821093 0.0%; Applied: 0; Backlog: 0/1000; Time: 1s(total), 1s(copy); streamer: mysql-bin.000196:1004965536; State: migrating; ETA: N/A Copy: 9000/821093 1.1%; Applied: 0; Backlog: 0/1000; Time: 2s(total), 2s(copy); streamer: mysql-bin.000196:1007404907; State: migrating; ETA: 3m0s Copy: 18000/821093 2.2%; Applied: 0; Backlog: 0/1000; Time: 3s(total), 3s(copy); streamer: mysql-bin.000196:1010115047; State: migrating; ETA: 2m13s ... ... ... 2022/12/19 16:35:14 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m5s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:35:17 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:17 ERROR 2022-12-19 16:35:17 ERROR Timeout while waiting for events up to lock 2022/12/19 16:35:19 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:35:19 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:35:19 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:35:19 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:35:21 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m10s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:35:21 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:21 ERROR 2022-12-19 16:35:21 ERROR Timeout while waiting for events up to lock 2022/12/19 16:35:26 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:35:26 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:35:26 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:35:26 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m15s(total), 2m0s(copy); streamer: mysql-bin.000197:3096692; State: migrating; ETA: due 2022-12-19 16:35:27 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:27 ERROR 2022-12-19 16:35:27 ERROR Timeout while waiting for events up to lock 2022/12/19 16:35:28 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m20s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:35:31 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:31 ERROR 2022-12-19 16:35:31 ERROR Timeout while waiting for events up to lock 2022/12/19 16:35:33 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:35:33 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:35:33 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:35:33 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:35:35 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m25s(total), 2m0s(copy); streamer: mysql-bin.000197:21536965; State: migrating; ETA: due 2022-12-19 16:35:37 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:37 ERROR 2022-12-19 16:35:37 ERROR Timeout while waiting for events up to lock 2022/12/19 16:35:40 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:35:40 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:35:40 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:35:40 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m30s(total), 2m0s(copy); streamer: mysql-bin.000197:10420122; State: migrating; ETA: due 2022/12/19 16:35:43 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:35:43 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:43 ERROR 2022-12-19 16:35:43 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m35s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:35:48 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:35:48 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:35:48 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:35:48 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:35:50 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:50 ERROR 2022-12-19 16:35:50 ERROR Timeout while waiting for events up to lock 2022/12/19 16:35:50 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m40s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:35:55 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:35:55 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:35:55 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:35:55 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:35:56 ERROR Timeout while waiting for events up to lock 2022-12-19 16:35:56 ERROR 2022-12-19 16:35:56 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m45s(total), 2m0s(copy); streamer: mysql-bin.000197:9353085; State: migrating; ETA: due 2022/12/19 16:35:58 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m50s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:36:02 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:02 ERROR 2022-12-19 16:36:02 ERROR Timeout while waiting for events up to lock 2022/12/19 16:36:03 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:03 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:03 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:03 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:36:05 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 2m55s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:36:08 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:08 ERROR 2022-12-19 16:36:08 ERROR Timeout while waiting for events up to lock 2022/12/19 16:36:10 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:10 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:10 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:10 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m0s(total), 2m0s(copy); streamer: mysql-bin.000197:17516759; State: migrating; ETA: due 2022/12/19 16:36:12 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:36:12 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:12 ERROR 2022-12-19 16:36:12 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:16 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:16 ERROR 2022-12-19 16:36:16 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m5s(total), 2m0s(copy); streamer: mysql-bin.000197:21608867; State: migrating; ETA: due 2022/12/19 16:36:17 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:17 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:17 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:17 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:36:18 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m10s(total), 2m0s(copy); streamer: mysql-bin.000197:21584851; State: migrating; ETA: due 2022-12-19 16:36:22 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:22 ERROR 2022-12-19 16:36:22 ERROR Timeout while waiting for events up to lock 2022/12/19 16:36:23 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:23 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:23 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:23 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:36:25 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:36:26 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:26 ERROR 2022-12-19 16:36:26 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m15s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:36:30 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:30 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:30 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:30 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m20s(total), 2m0s(copy); streamer: mysql-bin.000197:12833146; State: migrating; ETA: due 2022/12/19 16:36:32 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:36:32 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:32 ERROR 2022-12-19 16:36:32 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m25s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:36:37 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:37 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:37 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:37 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:36:38 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:38 ERROR 2022-12-19 16:36:38 ERROR Timeout while waiting for events up to lock 2022/12/19 16:36:38 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m30s(total), 2m0s(copy); streamer: mysql-bin.000197:21569140; State: migrating; ETA: due 2022/12/19 16:36:43 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:43 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:43 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:43 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:36:44 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:44 ERROR 2022-12-19 16:36:44 ERROR Timeout while waiting for events up to lock 2022/12/19 16:36:45 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m35s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:36:50 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:50 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:50 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:50 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:36:51 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:51 ERROR 2022-12-19 16:36:51 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m40s(total), 2m0s(copy); streamer: mysql-bin.000197:18875201; State: migrating; ETA: due 2022/12/19 16:36:51 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m45s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:36:56 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:36:56 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:36:56 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:36:56 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:36:57 ERROR Timeout while waiting for events up to lock 2022-12-19 16:36:57 ERROR 2022-12-19 16:36:57 ERROR Timeout while waiting for events up to lock 2022/12/19 16:36:58 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m50s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:37:03 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:03 ERROR 2022-12-19 16:37:03 ERROR Timeout while waiting for events up to lock 2022/12/19 16:37:03 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:03 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:03 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:03 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:37:05 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 3m55s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:37:09 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:09 ERROR 2022-12-19 16:37:09 ERROR Timeout while waiting for events up to lock 2022/12/19 16:37:10 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:10 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:10 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:10 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m0s(total), 2m0s(copy); streamer: mysql-bin.000197:15550736; State: migrating; ETA: due 2022/12/19 16:37:12 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:37:16 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:16 ERROR 2022-12-19 16:37:16 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m5s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:37:17 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:17 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:17 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:17 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:37:18 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m10s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:37:22 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:22 ERROR 2022-12-19 16:37:22 ERROR Timeout while waiting for events up to lock 2022/12/19 16:37:23 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:23 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:23 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:23 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:37:25 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m15s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:37:28 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:28 ERROR 2022-12-19 16:37:28 ERROR Timeout while waiting for events up to lock 2022/12/19 16:37:30 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:30 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:30 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:30 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m20s(total), 2m0s(copy); streamer: mysql-bin.000197:8319580; State: migrating; ETA: due 2022/12/19 16:37:32 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:37:34 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:34 ERROR 2022-12-19 16:37:34 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m25s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:37:37 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:37 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:37 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:37 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:37:39 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:37:40 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:40 ERROR 2022-12-19 16:37:40 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m30s(total), 2m0s(copy); streamer: mysql-bin.000197:21624860; State: migrating; ETA: due 2022/12/19 16:37:44 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:44 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:44 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:44 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:37:46 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m35s(total), 2m0s(copy); streamer: mysql-bin.000197:21584851; State: migrating; ETA: due 2022-12-19 16:37:47 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:47 ERROR 2022-12-19 16:37:47 ERROR Timeout while waiting for events up to lock 2022/12/19 16:37:51 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:51 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:51 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:51 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m40s(total), 2m0s(copy); streamer: mysql-bin.000197:3346793; State: migrating; ETA: due 2022-12-19 16:37:53 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:53 ERROR 2022-12-19 16:37:53 ERROR Timeout while waiting for events up to lock 2022/12/19 16:37:53 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m45s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:37:58 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:37:58 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:37:58 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:37:58 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:37:59 ERROR Timeout while waiting for events up to lock 2022-12-19 16:37:59 ERROR 2022-12-19 16:37:59 ERROR Timeout while waiting for events up to lock 2022/12/19 16:38:00 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m50s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:38:05 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:05 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:05 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:05 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:38:05 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:05 ERROR 2022-12-19 16:38:05 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 4m55s(total), 2m0s(copy); streamer: mysql-bin.000197:8519807; State: migrating; ETA: due 2022/12/19 16:38:07 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m0s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:38:12 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:12 ERROR 2022-12-19 16:38:12 ERROR Timeout while waiting for events up to lock 2022/12/19 16:38:12 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:12 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:12 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:12 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:38:16 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m5s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:38:18 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:18 ERROR 2022-12-19 16:38:18 ERROR Timeout while waiting for events up to lock 2022/12/19 16:38:21 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:21 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:21 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:21 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m10s(total), 2m0s(copy); streamer: mysql-bin.000197:2841618; State: migrating; ETA: due 2022/12/19 16:38:24 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:38:25 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:25 ERROR 2022-12-19 16:38:25 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m15s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:38:29 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:29 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:29 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:29 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:38:31 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:38:31 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:31 ERROR 2022-12-19 16:38:31 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m20s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:38:36 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:36 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:36 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:36 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m25s(total), 2m0s(copy); streamer: mysql-bin.000197:4548675; State: migrating; ETA: due 2022-12-19 16:38:37 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:37 ERROR 2022-12-19 16:38:37 ERROR Timeout while waiting for events up to lock 2022/12/19 16:38:37 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m30s(total), 2m0s(copy); streamer: mysql-bin.000197:21569140; State: migrating; ETA: due 2022/12/19 16:38:42 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:42 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:42 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:42 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:38:43 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:43 ERROR 2022-12-19 16:38:43 ERROR Timeout while waiting for events up to lock 2022/12/19 16:38:44 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m35s(total), 2m0s(copy); streamer: mysql-bin.000197:21561394; State: migrating; ETA: due 2022/12/19 16:38:49 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:49 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:49 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:49 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:38:49 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:49 ERROR 2022-12-19 16:38:49 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m40s(total), 2m0s(copy); streamer: mysql-bin.000197:20796315; State: migrating; ETA: due 2022/12/19 16:38:51 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:38:55 ERROR Timeout while waiting for events up to lock 2022-12-19 16:38:55 ERROR 2022-12-19 16:38:55 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m45s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:38:56 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:38:56 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:38:56 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:38:56 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:38:58 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m50s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:39:02 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:02 ERROR 2022-12-19 16:39:02 ERROR Timeout while waiting for events up to lock 2022/12/19 16:39:03 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:03 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:03 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:03 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:39:05 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 5m55s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:39:08 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:08 ERROR 2022-12-19 16:39:08 ERROR Timeout while waiting for events up to lock 2022/12/19 16:39:10 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:10 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:10 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:10 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m0s(total), 2m0s(copy); streamer: mysql-bin.000197:15982497; State: migrating; ETA: due 2022/12/19 16:39:12 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:39:14 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:14 ERROR 2022-12-19 16:39:14 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m5s(total), 2m0s(copy); streamer: mysql-bin.000197:21616782; State: migrating; ETA: due 2022/12/19 16:39:17 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:17 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:17 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:17 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:39:19 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:39:20 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:20 ERROR 2022-12-19 16:39:20 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m10s(total), 2m0s(copy); streamer: mysql-bin.000197:21569140; State: migrating; ETA: due 2022/12/19 16:39:24 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:24 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:24 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:24 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:39:25 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:39:26 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:26 ERROR 2022-12-19 16:39:26 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m15s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:39:30 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:30 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:30 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:30 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m20s(total), 2m0s(copy); streamer: mysql-bin.000197:15536690; State: migrating; ETA: due 2022/12/19 16:39:32 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:39:32 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:32 ERROR 2022-12-19 16:39:32 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m25s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:39:37 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:37 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:37 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:37 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:39:38 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:38 ERROR 2022-12-19 16:39:38 ERROR Timeout while waiting for events up to lock 2022/12/19 16:39:38 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m30s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:39:43 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:43 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:43 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:43 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:39:45 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction 2022-12-19 16:39:45 ERROR Error 1205: Lock wait timeout exceeded; try restarting transaction 2022/12/19 16:39:45 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m35s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:39:50 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:50 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:50 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:50 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m40s(total), 2m0s(copy); streamer: mysql-bin.000197:7727416; State: migrating; ETA: due 2022/12/19 16:39:52 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:39:53 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:53 ERROR 2022-12-19 16:39:53 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m45s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:39:57 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:39:57 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:39:57 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:39:57 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:39:59 ERROR Timeout while waiting for events up to lock 2022-12-19 16:39:59 ERROR 2022-12-19 16:39:59 ERROR Timeout while waiting for events up to lock 2022/12/19 16:39:59 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m50s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:40:04 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:04 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:04 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:04 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:40:05 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:05 ERROR 2022-12-19 16:40:05 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 6m55s(total), 2m0s(copy); streamer: mysql-bin.000197:19093954; State: migrating; ETA: due 2022/12/19 16:40:07 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:40:11 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:11 ERROR 2022-12-19 16:40:11 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m0s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:40:12 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:12 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:12 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:12 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:40:13 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m5s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:40:17 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:17 ERROR 2022-12-19 16:40:17 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:18 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:18 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:18 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:18 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022/12/19 16:40:20 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m10s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:40:23 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:23 ERROR 2022-12-19 16:40:23 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:25 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:25 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:25 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:25 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m15s(total), 2m0s(copy); streamer: mysql-bin.000197:6506231; State: migrating; ETA: due 2022-12-19 16:40:27 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:27 ERROR 2022-12-19 16:40:27 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:29 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m20s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:40:31 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:31 ERROR 2022-12-19 16:40:31 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:34 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:34 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:34 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:34 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:40:35 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:35 ERROR 2022-12-19 16:40:35 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:36 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m25s(total), 2m0s(copy); streamer: mysql-bin.000197:21553308; State: migrating; ETA: due 2022-12-19 16:40:39 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:39 ERROR 2022-12-19 16:40:39 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:41 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:41 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:41 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:41 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m30s(total), 2m0s(copy); streamer: mysql-bin.000197:8687216; State: migrating; ETA: due 2022/12/19 16:40:42 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 2022-12-19 16:40:43 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:43 ERROR 2022-12-19 16:40:43 ERROR Timeout while waiting for events up to lock Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m35s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022/12/19 16:40:47 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:47 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:47 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:47 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:40:47 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:47 ERROR 2022-12-19 16:40:47 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:49 binlogstreamer.go:47: [error] close sync with err: data len 310 < expected 2179 Copy: 769329/769329 100.0%; Applied: 0; Backlog: 0/1000; Time: 7m40s(total), 2m0s(copy); streamer: mysql-bin.000197:21632791; State: migrating; ETA: due 2022-12-19 16:40:51 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:51 ERROR 2022-12-19 16:40:51 ERROR Timeout while waiting for events up to lock 2022/12/19 16:40:54 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql localhost 3306 rdsadmin false false } 2022/12/19 16:40:54 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.000197, 4) 2022/12/19 16:40:54 binlogsyncer.go:139: [info] register slave for master server localhost:3306 2022/12/19 16:40:54 binlogsyncer.go:573: [info] rotate to (mysql-bin.000197, 4) 2022-12-19 16:40:55 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:55 ERROR 2022-12-19 16:40:55 ERROR Timeout while waiting for events up to lock 2022-12-19 16:40:56 FATAL 2022-12-19 16:40:55 ERROR Timeout while waiting for events up to lock

    There are two errors here: 1、 [error] close sync with err: data len 310 < expected 2179 The trigger reason for this error is not detailed，i don't know what caused it.

    2、 ERROR Timeout while waiting for events up to lock Here I think that due to the existence of large transactions, multiple acquisitions of metadata locks failed, resulting in timeouts for acquiring metadata locks.

    In addition, the execution failure on the main library will also cause the replication of the slave library to be stuck, and the drop table and multi-threaded replication will cause lock conflicts. image

    opened by boomballa 1
  • feat(binlogsyncer): support set binlogsyncer maxreconnectattempts

    feat(binlogsyncer): support set binlogsyncer maxreconnectattempts

    A Pull Request should be associated with an Issue.

    We wish to have discussions in Issues. A single issue may be targeted by multiple PRs. If you're offering a new feature or fixing anything, we'd like to know beforehand in Issues, and potentially we'll be able to point development in a particular direction.

    Related issue: https://github.com/github/gh-ost/issues/0123456789

    Further notes in https://github.com/github/gh-ost/blob/master/.github/CONTRIBUTING.md Thank you! We are open to PRs, but please understand if for technical reasons we are unable to accept each and any PR

    Description

    This PR [briefly explain what it does]

    In case this PR introduced Go code changes:

    • [ ] contributed code is using same conventions as original code
    • [ ] script/cibuild returns with no formatting errors, build errors or unit test errors.
    opened by shaohk 0
  • use last second copy rate to project ETA

    use last second copy rate to project ETA

    A Pull Request should be associated with an Issue.

    We wish to have discussions in Issues. A single issue may be targeted by multiple PRs. If you're offering a new feature or fixing anything, we'd like to know beforehand in Issues, and potentially we'll be able to point development in a particular direction.

    Related issue: https://github.com/github/gh-ost/issues/1204

    Further notes in https://github.com/github/gh-ost/blob/master/.github/CONTRIBUTING.md Thank you! We are open to PRs, but please understand if for technical reasons we are unable to accept each and any PR

    Description

    Currently, the ETA that gh-ost uses is based on a projection that averages over the whole run time. Because (with a b+tree) the insert rate will slow down as the table gets larger, this can lead to inaccurate projections.

    This uses a different approach, which is to project from the number of rows copied in the last second. It has the downside that the estimate can tend to "jump around" a bit, but sometimes this is helpful. I played around with changing the sample average to 10s etc, but it looks like 1s seems to be long enough to balance estimate stability.

    Here is a sample output showing the time bouncing around a little:

    ~/go/src/github.com/morgo/gh-ost/go/cmd/gh-ost more-accurate-eta go run . \
--max-load=Threads_running=25 \
--critical-load=Threads_running=1000 \
--chunk-size=1000 \
--user="msandbox" \
--password="msandbox" \
--host="127.0.0.1" \
--port=8031 \
--allow-on-master \
--database="test" \
--table="customer" \
--initially-drop-old-table \
--ok-to-drop-table \
--execute  --initially-drop-ghost-table --alter="engine=innodb"
[2022/12/14 14:28:21] [info] binlogsyncer.go:148 create BinlogSyncer with config {99999 mysql 127.0.0.1 8031 msandbox    false false <nil> false UTC true 0 0s 0s 0 false false 0 <nil>}
[2022/12/14 14:28:21] [info] binlogsyncer.go:374 begin to sync binlog from position (binlog.000199, 157)
[2022/12/14 14:28:21] [info] binlogsyncer.go:791 rotate to (binlog.000199, 157)
# Migrating `test`.`customer`; Ghost table is `test`.`_customer_gho`
# Migrating mtocker-macbookpro.local:8031; inspecting mtocker-macbookpro.local:8031; executing on mtocker-macbookpro.local
# Migration started at Wed Dec 14 14:28:21 -0700 2022
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: Threads_running=25; critical-load: Threads_running=1000; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# Serving on unix socket: /tmp/gh-ost.test.customer.sock
Copy: 0/2881784 0.0%; Applied: 0; Backlog: 0/1000; Time: 0s(total), 0s(copy); streamer: binlog.000199:3566; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: N/A
Copy: 0/2881784 0.0%; Applied: 0; Backlog: 0/1000; Time: 1s(total), 1s(copy); streamer: binlog.000199:8040; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: N/A
Copy: 84000/2881784 2.9%; Applied: 0; Backlog: 0/1000; Time: 2s(total), 2s(copy); streamer: binlog.000199:47281787; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 1m6s
Copy: 168000/2881784 5.8%; Applied: 0; Backlog: 0/1000; Time: 3s(total), 3s(copy); streamer: binlog.000199:94566987; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 32s
Copy: 245000/2881784 8.5%; Applied: 0; Backlog: 0/1000; Time: 4s(total), 4s(copy); streamer: binlog.000199:137949978; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 34s
Copy: 323000/2881784 11.2%; Applied: 0; Backlog: 0/1000; Time: 5s(total), 5s(copy); streamer: binlog.000199:181313206; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 32s
Copy: 402000/2881784 13.9%; Applied: 0; Backlog: 0/1000; Time: 6s(total), 6s(copy); streamer: binlog.000199:226333018; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 31s
Copy: 476000/2881784 16.5%; Applied: 0; Backlog: 0/1000; Time: 7s(total), 7s(copy); streamer: binlog.000199:268011561; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 32s
Copy: 555000/2881784 19.3%; Applied: 0; Backlog: 0/1000; Time: 8s(total), 8s(copy); streamer: binlog.000199:312501660; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 29s
Copy: 615000/2881784 21.3%; Applied: 0; Backlog: 0/1000; Time: 9s(total), 9s(copy); streamer: binlog.000199:346275029; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 37s
Copy: 669000/2881784 23.2%; Applied: 0; Backlog: 0/1000; Time: 10s(total), 10s(copy); streamer: binlog.000199:376674938; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 40s
Copy: 726000/2881784 25.2%; Applied: 0; Backlog: 0/1000; Time: 11s(total), 11s(copy); streamer: binlog.000199:408728243; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 37s
Copy: 783000/2881784 27.2%; Applied: 0; Backlog: 0/1000; Time: 12s(total), 12s(copy); streamer: binlog.000199:440842291; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 36s
Copy: 846000/2881784 29.4%; Applied: 0; Backlog: 0/1000; Time: 13s(total), 13s(copy); streamer: binlog.000199:476296347; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 32s
Copy: 910000/2881784 31.6%; Applied: 0; Backlog: 0/1000; Time: 14s(total), 14s(copy); streamer: binlog.000199:512338908; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 30s
Copy: 969000/2881784 33.6%; Applied: 0; Backlog: 0/1000; Time: 15s(total), 15s(copy); streamer: binlog.000199:545572110; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 32s
Copy: 1026000/2881784 35.6%; Applied: 0; Backlog: 0/1000; Time: 16s(total), 16s(copy); streamer: binlog.000199:577675650; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 32s
Copy: 1087000/2881784 37.7%; Applied: 0; Backlog: 0/1000; Time: 17s(total), 17s(copy); streamer: binlog.000199:612023226; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 29s
Copy: 1148000/2881784 39.8%; Applied: 0; Backlog: 0/1000; Time: 18s(total), 18s(copy); streamer: binlog.000199:646342977; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 28s
Copy: 1204000/2881784 41.8%; Applied: 0; Backlog: 0/1000; Time: 19s(total), 19s(copy); streamer: binlog.000199:677881385; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 29s
Copy: 1265000/2881784 43.9%; Applied: 0; Backlog: 0/1000; Time: 20s(total), 20s(copy); streamer: binlog.000199:712235510; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 26s
Copy: 1335000/2881784 46.3%; Applied: 0; Backlog: 0/1000; Time: 21s(total), 21s(copy); streamer: binlog.000199:751664902; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 22s
Copy: 1403000/2881784 48.7%; Applied: 0; Backlog: 0/1000; Time: 22s(total), 22s(copy); streamer: binlog.000199:789461333; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 21s
Copy: 1464000/2881784 50.8%; Applied: 0; Backlog: 0/1000; Time: 23s(total), 23s(copy); streamer: binlog.000199:824289714; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 23s
Copy: 1527000/2881784 53.0%; Applied: 0; Backlog: 0/1000; Time: 24s(total), 24s(copy); streamer: binlog.000199:859468766; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 21s
Copy: 1592000/2881784 55.2%; Applied: 0; Backlog: 0/1000; Time: 25s(total), 25s(copy); streamer: binlog.000199:896372430; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 19s
Copy: 1662000/2881784 57.7%; Applied: 0; Backlog: 0/1000; Time: 26s(total), 26s(copy); streamer: binlog.000199:935798318; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 17s
Copy: 1725000/2881784 59.9%; Applied: 0; Backlog: 0/1000; Time: 27s(total), 27s(copy); streamer: binlog.000199:971271951; Lag: 0.11s, HeartbeatLag: 0.11s, State: migrating; ETA: 18s
Copy: 1793000/2881784 62.2%; Applied: 0; Backlog: 0/1000; Time: 28s(total), 28s(copy); streamer: binlog.000199:1009545478; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 16s
Copy: 1856000/2881784 64.4%; Applied: 0; Backlog: 0/1000; Time: 29s(total), 29s(copy); streamer: binlog.000199:1044945554; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 16s
[2022/12/14 14:28:51] [info] binlogsyncer.go:791 rotate to (binlog.000200, 4)
[2022/12/14 14:28:51] [info] binlogsyncer.go:791 rotate to (binlog.000200, 4)
Copy: 1920000/2881784 66.6%; Applied: 0; Backlog: 0/1000; Time: 30s(total), 30s(copy); streamer: binlog.000200:6753737; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 15s
Copy: 1977000/2881784 68.6%; Applied: 0; Backlog: 0/1000; Time: 31s(total), 31s(copy); streamer: binlog.000200:38877488; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 15s
Copy: 2032000/2881784 70.5%; Applied: 0; Backlog: 0/1000; Time: 32s(total), 32s(copy); streamer: binlog.000200:69863585; Lag: 0.11s, HeartbeatLag: 0.11s, State: migrating; ETA: 15s
Copy: 2093000/2881784 72.6%; Applied: 0; Backlog: 0/1000; Time: 33s(total), 33s(copy); streamer: binlog.000200:104128743; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 12s
Copy: 2158000/2881784 74.9%; Applied: 0; Backlog: 0/1000; Time: 34s(total), 34s(copy); streamer: binlog.000200:140822753; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 11s
Copy: 2220000/2881784 77.0%; Applied: 0; Backlog: 0/1000; Time: 35s(total), 35s(copy); streamer: binlog.000200:175718354; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 10s
Copy: 2280000/2881784 79.1%; Applied: 0; Backlog: 0/1000; Time: 36s(total), 36s(copy); streamer: binlog.000200:209458784; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 10s
Copy: 2342000/2881784 81.3%; Applied: 0; Backlog: 0/1000; Time: 37s(total), 37s(copy); streamer: binlog.000200:244384714; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 8s
Copy: 2405000/2881784 83.5%; Applied: 0; Backlog: 0/1000; Time: 38s(total), 38s(copy); streamer: binlog.000200:279865282; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 7s
Copy: 2469000/2881784 85.7%; Applied: 0; Backlog: 0/1000; Time: 39s(total), 39s(copy); streamer: binlog.000200:315425854; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 6s
Copy: 2529000/2881784 87.8%; Applied: 0; Backlog: 0/1000; Time: 40s(total), 40s(copy); streamer: binlog.000200:349673270; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 5s
Copy: 2598000/2881784 90.2%; Applied: 0; Backlog: 0/1000; Time: 41s(total), 41s(copy); streamer: binlog.000200:388534212; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 4s
Copy: 2661000/2881784 92.3%; Applied: 0; Backlog: 0/1000; Time: 42s(total), 42s(copy); streamer: binlog.000200:423824648; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 3s
Copy: 2718000/2881784 94.3%; Applied: 0; Backlog: 0/1000; Time: 43s(total), 43s(copy); streamer: binlog.000200:456090786; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 2s
Copy: 2775000/2881784 96.3%; Applied: 0; Backlog: 0/1000; Time: 44s(total), 44s(copy); streamer: binlog.000200:488195145; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: 1s
Copy: 2833000/2881784 98.3%; Applied: 0; Backlog: 0/1000; Time: 45s(total), 45s(copy); streamer: binlog.000200:520841157; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: due
Copy: 2894000/2881784 100.4%; Applied: 0; Backlog: 0/1000; Time: 46s(total), 46s(copy); streamer: binlog.000200:555202612; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: due
Copy: 2952000/2881784 102.4%; Applied: 0; Backlog: 0/1000; Time: 47s(total), 47s(copy); streamer: binlog.000200:587420664; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: due
Copy: 3000000/3000000 100.0%; Applied: 0; Backlog: 0/1000; Time: 47s(total), 47s(copy); streamer: binlog.000200:614436979; Lag: 0.01s, HeartbeatLag: 0.03s, State: migrating; ETA: due
Copy: 3000000/3000000 100.0%; Applied: 0; Backlog: 1/1000; Time: 48s(total), 47s(copy); streamer: binlog.000200:614902121; Lag: 0.01s, HeartbeatLag: 0.01s, State: migrating; ETA: due
# Migrating `test`.`customer`; Ghost table is `test`.`_customer_gho`
# Migrating mtocker-macbookpro.local:8031; inspecting mtocker-macbookpro.local:8031; executing on mtocker-macbookpro.local
# Migration started at Wed Dec 14 14:28:21 -0700 2022
# chunk-size: 1000; max-lag-millis: 1500ms; dml-batch-size: 10; max-load: Threads_running=25; critical-load: Threads_running=1000; nice-ratio: 0.000000
# throttle-additional-flag-file: /tmp/gh-ost.throttle
# Serving on unix socket: /tmp/gh-ost.test.customer.sock
Copy: 3000000/3000000 100.0%; Applied: 0; Backlog: 0/1000; Time: 48s(total), 47s(copy); streamer: binlog.000200:614905425; Lag: 0.01s, HeartbeatLag: 0.03s, State: migrating; ETA: due
[2022/12/14 14:29:10] [info] binlogsyncer.go:180 syncer is closing...
[2022/12/14 14:29:10] [info] binlogsyncer.go:864 kill last connection id 11
[2022/12/14 14:29:10] [info] binlogsyncer.go:210 syncer is closed
# Done

    In case this PR introduced Go code changes:

    • [x] contributed code is using same conventions as original code
    • [x] script/cibuild returns with no formatting errors, build errors or unit test errors.
    opened by morgo 0
Releases(v1.1.5)

  • v1.1.5(Jul 9, 2022)

    Changes since 1.1.4: https://github.com/github/gh-ost/compare/v1.1.4...v1.1.5

    gh-ost v1.1.5 is based on v1.1.4, plus commits cherry-picked from master branch.

    Notable:

    • Cancel any row count queries before attempting to cut over, #846, thanks @ajm188
    • Add context/timeout to HTTP throttle check, #1131, thanks @timvaillancourt
      • Throttler will now throttle migration on HTTP timeouts to throttler source. This may cause a change in behaviour in rare cases (more throttling)
      • New command line flags:
        1. --throttle-http-interval-millis - Number of milliseconds to wait between HTTP throttle checks (default 100)
        2. --throttle-http-timeout-millis - Number of milliseconds to use as an HTTP throttle check timeout (default 1000)
      • HTTP throttler calls now use gh-ost/VERSION-HERE as a User-Agent header
    • Use switch statements for readability, simplify .NewGoMySQLReader(), #1135, thanks @timvaillancourt
    • Ensure mysql rows responses are closed, #1132, thanks @timvaillancourt
    • compound pk tests, #387, thanks @shlomi-noach
    • Fix integer divide by zero panic in migrator, #1140, thanks @timvaillancourt for the PR and @ethankhall for reporting
    • fix: lost data in mysql two-phase commit, #1141, thanks @shaohk
    • Add golangci-lint CI action, fix gosimple, govet + unused lint errors, #1127, thanks @timvaillancourt
    • Check RowsAffected when applying DML events to get more accurate statistics, #844, thanks @ajm188

    Deprecation Notice:

    • --replication-lag-query command line flag will be deprecated in gh-ost release 1.1.7

    This release is not based on the master branch as it contains a known issue that is currently being investigated - please note that the master branch of gh-ost is not considered stable; only releases are considered production-ready.

    Thank you to all contributors!

    Updated 20th July 2022: RPM and deb packages have been updated to correctly include x86-64 binaries instead of arm64 binaries (thanks @Cartman75 for the issue report!)

    $ sha256sum *
6a9dff3594b60a26422f7192ff7a79a85684b477d453431bedf025bc4023c389  gh-ost
236a1b9fb8fbb38bb6e27159377bd5af276ba09c3382724b5d88e9da6424dcbd  gh-ost-1.1.5-1.x86_64.rpm
791ab7fe0e8fe9e754480f4decdba8380c87d5ce70648f8a549060d81f889116  gh-ost-binary-linux-amd64-20220707162303.tar.gz
48f53683e3c6bf52957ff985c8ccc2c09f8f2484b44da4346f659e019ef69be9  gh-ost-binary-linux-arm64-20220707162303.tar.gz
1b0d992d03eeba7aa919ca9b40019b92af319dcb3668e0a447db21b35016c5ee  gh-ost-binary-osx-amd64-20220707162303.tar.gz
16fa1e09d2981e8a9312961337e41693a2e1396fe11abfd5b6a60781f3433795  gh-ost-binary-osx-arm64-20220707162303.tar.gz
2574da3ffe60f9a662366ca4f6a46edd90b8885acee8047ba0b11544561eb28d  gh-ost_1.1.5_amd64.deb

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost(8.71 MB)
    gh-ost-1.1.5-1.x86_64.rpm(4.65 MB)
    gh-ost-binary-linux-amd64-20220707162303.tar.gz(4.66 MB)
    gh-ost-binary-linux-arm64-20220707162303.tar.gz(4.36 MB)
    gh-ost-binary-osx-amd64-20220707162303.tar.gz(4.60 MB)
    gh-ost-binary-osx-arm64-20220707162303.tar.gz(4.46 MB)
    gh-ost_1.1.5_amd64.deb(4.66 MB)

  • v1.1.4(Feb 9, 2022)

    Changes since 1.1.3: https://github.com/github/gh-ost/compare/v1.1.3...v1.1.4

    gh-ost v1.1.4 is based on v1.1.3, plus commits cherry-picked from master branch.

    Notable:

    • Add flag to customize the interval which the onStatus hook is called, #1083, thanks @tknodell-recurly
    • Add docs for hooks-status-interval, #1090
    • Reduce the minimal chunk size from 100 to 10, #1087, thanks @arthurschreiber

    This release is not based on the master branch as it contains a known issue that is currently being investigated - please note that the master branch of gh-ost is not considered stable; only releases are considered production-ready.

    Thank you to all contributors!

    Updated 25th Feb 2022: Binaries have been re-uploaded to fix an issue where RPMs were being generated with an OS for darwin instead of linux. (thanks @kedarvj for the report!)

    Updated 11th Mar 2022: Added macOS arm64 binary to support M1 Macs (thanks @jecepeda for adding support in #1108!)

    $ sha256sum *
da2b2a943aaf18fb849b52e19b0828c9878b180546d1adb31ad03b3cf7c50d53  gh-ost
d626f6ae582ca15c58a387c819c410788a78f64dd7ac7ec946df07f0bf66f7a0  gh-ost-1.1.4-1.x86_64.rpm
ccc055a9fa2ca080b99aeb58a6aacf557582c317b8f36ee13c551d87cd5eeb30  gh-ost-binary-linux-20220225143506.tar.gz
9feebe056914335d23127db10de8f20631e0df448b1382f30366a462ab5d5473  gh-ost-binary-osx-20220225143506.tar.gz
66619ad49fb7a2a13edfe08aee110a554f7174b44453451fee2e532661f1f254  gh-ost-binary-osx-arm64-20220311164158.tar.gz
1dead2c07b73a8730ec4b923433be143dcb4dc18719e2ad640962e389168a22d  gh-ost_1.1.4_amd64.deb

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost(8.68 MB)
    gh-ost-1.1.4-1.x86_64.rpm(4.66 MB)
    gh-ost-binary-linux-20220225143506.tar.gz(4.67 MB)
    gh-ost-binary-osx-amd64-20220225143506.tar.gz(4.61 MB)
    gh-ost-binary-osx-arm64-20220311164158.tar.gz(4.40 MB)
    gh-ost_1.1.4_amd64.deb(4.67 MB)

  • v1.1.3(Jan 31, 2022)

    Changes since 1.1.2: https://github.com/github/gh-ost/compare/v1.1.2...v1.1.3

    gh-ost v1.1.3 is comprised of v1.1.2 with a fix for security vulnerability GHSA-rrp4-2xx3-mv29, plus low-risk documentation updates cherry-picked from master branch.

    This release is not based on the master branch as it contains a known issue that is currently being investigated - please note that the master branch of gh-ost is not considered stable; only releases are considered production-ready.

    Notable:

    • Security vulnerability: Improper Input Validation of -database Parameter, GHSA-rrp4-2xx3-mv29, thanks @dwisiswant0
    • doc/triggerless-design.md: typo, #1011, thanks @eddroid
    • Document the flag critical-load-hibernate-seconds, improve grammar of flags documentation, #1066, thanks @EagleEyeJohn
    • Update hook environment variable docs, #1076
    • Update build script to generate releases, #1078
    • Remove CGO dependency in go-mysql in Gh-ost v1.1.3, #1079

    Thank you to all contributors!

    Updated 8th Feb 2022: Binaries have been re-uploaded with the correct version number; previously they were incorrectly labelled as v1.1.2 instead of v1.1.3. (thanks @dharnett for notifying us!)

    Updated 25th Feb 2022: Binaries have been re-uploaded to fix an issue where RPMs were being generated with an OS for darwin instead of linux. (thanks @kedarvj for the report!)

    $ sha256sum *
10255fa8690001d7eb37f51c6d41aba0e55ee9ef4c2b949ca4c0604011b74d71  gh-ost
496ac03a36738e752ad98668a3bb97eca913039431fdff80732fbb4acfac8991  gh-ost-1.1.3-1.x86_64.rpm
c432e7d711268117bc8c13cebcf3e3cf3c986557a205cc041e0ea63937700ef5  gh-ost-binary-linux-20220225143057.tar.gz
e55b3c28c655eab28617a47eefa88fb6a508af506652bafdd226293968bebd8e  gh-ost-binary-osx-20220225143057.tar.gz
116db1f8f5e4bff6a8903f057dad07e14ad36eea0f3b462a16342b6d3b447d1a  gh-ost_1.1.3_amd64.deb

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost(8.68 MB)
    gh-ost-1.1.3-1.x86_64.rpm(4.66 MB)
    gh-ost-binary-linux-20220225143057.tar.gz(4.67 MB)
    gh-ost-binary-osx-20220225143057.tar.gz(4.61 MB)
    gh-ost_1.1.3_amd64.deb(4.67 MB)

  • v1.1.2(Jun 17, 2021)

    Changes since 1.1.1: https://github.com/github/gh-ost/compare/v1.1.1...v1.1.2

    Notable:

    • Copy auto increment, #911, thanks @shlomi-noach
    • Cut-over should wait for heartbeat lag to be low enough to succeed, #921, thanks @ccoffey
    • All MySQL DBs limited to max 3 concurrent/idle connections, #931, thanks @shlomi-noach
    • Report GH_OST_ETA_SECONDS ETA as part of migration context, #936, thanks @shlomi-noach
    • Remove build_id files from rpm, #954, thanks @yakirgb
    • Fix bug with -exact-rowcount on MySQL 8.0.2+, #962, thanks @Fanduzi
    • Support enum to varchar, #963, thanks @shlomi-noach
    • Set the ServerName for TLS configuration, #988, thanks @dbussink
    • Use Golang 1.16 and temporarily disable go modules in builds, #966
    • Use matrix build for replica test CI, #968

    Thank you to all contributors!

    $ sha256sum *
342a7f4760d51dc05154dd1e6663f701bb024868a7f9b0804e8bf38afe67e0ff  gh-ost
255dbaed5de7c20b1f11c82ed0275e0ecfaa8f8afa4f38fd36705cfac4b6207f  gh-ost-1.1.2-1.x86_64.rpm
98ebff834bf930a198d3506cda0ee1a6a3ea67f09e9112ca8a2534ef4f74288c  gh-ost-binary-linux-20210617134741.tar.gz
0b000805894ae5cc6b5ff0aacb67e4273328354f5e559a9ad39b1e7cc617b316  gh-ost-binary-osx-20210617155835.tar.gz
afaf95ea239b5e473080d0b58592e6449cfc48d936396a8540722c4772741893  gh-ost_1.1.2_amd64.deb

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost(9.40 MB)
    gh-ost-1.1.2-1.x86_64.rpm(4.46 MB)
    gh-ost-binary-linux-20210617134741.tar.gz(4.49 MB)
    gh-ost-binary-osx-20210617155835.tar.gz(4.96 MB)
    gh-ost_1.1.2_amd64.deb(4.49 MB)

  • v1.1.1(May 3, 2021)

    Changes since 1.1.0: https://github.com/github/gh-ost/compare/v1.1.0...v1.1.1

    Notable:

    • Use sync.Once to avoid dropping cutover sentry table twice, #755, thanks @MOON-CLJ
    • Parsing IPV6 addresses, #640, thanks @shlomi-noach
    • Support Azure Database for MySQL, #807, thanks @elemount
    • Use Golang 1.15 and remove Debian Jessie (end-of-life) tarball build, #906
    • Improved connection type logging, #515, thanks @akshaychhajed
    • Fix: Data loss on tables with binary primary key, #915, thanks @cenkore
    • Add applier and inspector interactive commands to socket/TCP server, #951

    Thank you to all contributors!

    $ sha256sum *
8e802e8685d4b7d757ae47513b918d6de1d0402511cc3fadb2d961713beb6d89  gh-ost
6b49bbcf1a985d96fa98954d6aba5b2e8fa9f8b0ac9aee2c10bd47755f986254  gh-ost-1.1.1-1.x86_64.rpm
1188bb2d4539d25fc8a4b8a96cccb9b56a217124a9d4c8b9faab4534acd9258b  gh-ost-binary-linux-20210503150942.tar.gz
1ef5585f926af0d93af53007b563c86a34d52bd4c05e1be5c6afaf8064199d3e  gh-ost-binary-osx-20210503171035.tar.gz
02d39afd96e0b5d69baf6a7259871b664445eaa811ccb6c3928376793b56f997  gh-ost_1.1.1_amd64.deb

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost(9.11 MB)
    gh-ost-1.1.1-1.x86_64.rpm(4.57 MB)
    gh-ost-binary-linux-20210503150942.tar.gz(4.60 MB)
    gh-ost-binary-osx-20210503171035.tar.gz(4.95 MB)
    gh-ost_1.1.1_amd64.deb(4.60 MB)

  • v1.1.0(Aug 28, 2020)

    Changes since 1.0.49: https://github.com/github/gh-ost/compare/v1.0.49...v1.1.0

    Notable:

    • Support a complete ALTER TABLE statement in --alter, #865 / #878, thanks @shlomi-noach
    • Support --mysql-timeout flag, #824, thanks @shlomi-noach
    • Add a check to rows.Err after processing all rows, #835, thanks @ajm188
    • Update go-sql-driver to latest, #823, thanks @shlomi-noach
    • Update build to Golang 1.14, #861 / #876, thanks @shlomi-noach
    • Add error checking for an err variable that was left unchecked, #810, thanks @yaserazfar
    • Implement a logging interface, #789 / #864, thanks @abeyum and @jfudally
    • Throttle on HTTP error to throttling API, #833, thanks @jfudally
    • Additional testing:
      • Add latin1 tests with TEXT columns, #291 / #455, thanks @shlomi-noach
      • Add BINARY / VARBINARY tests, #655, thanks @shlomi-noach
      • Add NULL-able INT tests, #692, thanks @shlomi-noach

    Thank you to all contributors!

    $ sha256sum *
e6de44ed76cd7834d7dac5ee4a53459ce18e16fbb482e9ff5d898f7d83444632  gh-ost-1.1.0-1.x86_64.rpm
2c29184d5b7f1340999680c06ae6796639f195d00907214a9baddc9ca9391ca9  gh-ost-binary-linux-20200828140552.tar.gz
c1dc2716dd5b843b4ef11b3fa601885f079363ca5545799bdac7037fd132e723  gh-ost-binary-osx-20200828160625.tar.gz
6ca0c5f0a79e51035e380a00350a535e04381b29ac9bbb847acf31d3b8985390  gh-ost_1.1.0_amd64.deb

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-1.1.0-1.x86_64.rpm(4.88 MB)
    gh-ost-binary-linux-20200828140552.tar.gz(4.90 MB)
    gh-ost-binary-osx-20200828160625.tar.gz(5.51 MB)
    gh-ost_1.1.0_amd64.deb(4.90 MB)

  • v1.0.49(Feb 9, 2020)

    Changes since 1.0.48: https://github.com/github/gh-ost/compare/v1.0.48...v1.0.49

    Notable:

    • --force-named-panic (on panic command, force table name), named throttle, named no-throttle, #695
    • add --skip-strict-mode option, #712, thank @babinomec
    • Improve SSL support #716, thanks @RafeKettler
    • context, status and hooks: add progressPct and CurrentLag, #788
    • Always use NO_AUTO_VALUE_ON_ZERO #723
    • tests: test on MySQL 8.0
    • tests: Bump default-retries to 3 #754 , thanks @zmoazeni
    • Support and require go1.12
    • doc updates/fixes, thanks @mohitm15, Tobias Johansson, @jchambers
    $ sha256sum *
628ad01f4b327ee81f01aee1e4e5d2a966ed681fa391d064424270dbd7b0b0b1  gh-ost-1.0.49-1.x86_64.rpm
a1d7f72e1119bb8a939204a56acbee09eb52c769183a4649e56d6b3b524cb774  gh-ost-binary-linux-20200209110835.tar.gz
667f8800654410749b1af6053f26109769992adb813eb9edaa154d537cc86ef3  gh-ost_1.0.49_amd64.deb
95b3ed74f70ffea9f11f4af06fb6e312f0a857bf4fd3c3ad2da31fc7b3359dd3  gh-ost-binary-osx-20200209114711.tar.gz

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-1.0.49-1.x86_64.rpm(4.72 MB)
    gh-ost-binary-linux-20200209110835.tar.gz(4.74 MB)
    gh-ost-binary-osx-20200209114711.tar.gz(5.38 MB)
    gh-ost_1.0.49_amd64.deb(4.74 MB)

  • v1.0.48(Feb 14, 2019)

    Changes since 1.0.47: https://github.com/github/gh-ost/compare/v1.0.47...v1.0.48

    • Update to latest go-mysql library, which solved multiple issues
    • Fix inspector column types #661, thanks @esnunes
    • inspect: remove redundant join conditions #699, thanks @twotwotwo
    • maintainer email address in .deb package
    • added: --hooks-hint-owner, --hooks-hint-token #703
    • Initial SSL Connection Support #705, thanks @brandonbodnar-wk
    • Fixed DECIMAL datatype #684 (the trigger for updating go-mysql)
    • more... thank you to all contributors!
    $ sha256sum *
cf1f0999c81f194472b9d3f154c21e0848bd1d61697ce1ca9123368b31d718b2  gh-ost-1.0.48-1.x86_64.rpm
8d0a518064c5803e1d5e8676031e5a5b5b180e67700d476e19eaf68e4a4a583a  gh-ost-binary-linux-20190214020851.tar.gz
f57a1361315adba33b51ecd9bf8863722385dbcff741f0a4d467087aaded9bc4  gh-ost-binary-osx-20190214120940.tar.gz
67bf74f1af6f968a59a2725398ccb5167d97e9b0cd1ab00c867425bf86a0d124  gh-ost_1.0.48_amd64.deb

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-1.0.48-1.x86_64.rpm(3.11 MB)
    gh-ost-binary-linux-20190214020851.tar.gz(3.13 MB)
    gh-ost-binary-osx-20190214120940.tar.gz(3.66 MB)
    gh-ost_1.0.48_amd64.deb(3.13 MB)

  • v1.0.47(Oct 16, 2018)

    Changes since 1.0.46: https://github.com/github/gh-ost/compare/v1.0.46...v1.0.47

    • Fix to copyRowsFunc hanging issue. Thank you @wfxiang08 #611
    • Support for GCP (Google Cloud Platform) #634
    • Populate the hooks environment with dry run status. Thank you @jacobbednarz #641
    • Integration test updates
    • Minor documentation updates

    Checksums:

    $ sha256sum *
1e2659498d8f2b66cd911e52c8282f03fd2de08eb86d6d77aad0c7ec3e6ff9e0  gh-ost-1.0.47-1.x86_64.rpm
1b00a14d5b358efd210d5371d63d5c84f3ce35717fbb2c15a24b7248b2dede95  gh-ost-binary-linux-20181016015113.tar.gz
396964f58cb6a72818437cd1a2cf129523b339b6000dd6443f9399841e0d35d9  gh-ost-binary-osx-20181016015113.tar.gz
8e98c38d9d4129929a95850aecc50f010443626e5ff0ba7ef85c90f24444e199  gh-ost_1.0.47_amd64.deb

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-1.0.47-1.x86_64.rpm(2.81 MB)
    gh-ost-binary-linux-20181016015113.tar.gz(2.84 MB)
    gh-ost-binary-osx-20181016015113.tar.gz(2.82 MB)
    gh-ost_1.0.47_amd64.deb(2.84 MB)

  • v1.0.46(May 28, 2018)

    Changes since 1.0.45: https://github.com/github/gh-ost/compare/v1.0.45...v1.0.46

    • Support for GENERATED columns, #595, based on work by @dveeden
    • Support for zero timestamps on MySQL 5.5 #539
    • Sanity: enforcing binlog_row_image=FULL
    • Avoid prepared statements for changelog table #592 @ggunson
    • Community contributions, thanks @SchumacherFM, @atsushi-ishibashi,
    • Release: rpm and deb packages

    Checksums:

    $ sha256sum *
9b0741d1b27aab613cdad12c2bfdd70d6879b116b6d41cef3b58eb21d4f14e62  gh-ost-1.0.46-1.x86_64.rpm
4222af72f19eadfb6ca7335791c8cc9f628f14c751bc324508cf976fc6dd9feb  gh-ost-binary-linux-20180527215024.tar.gz
4f069500b7d3a9f1396fc615acd4913fb12783d8e8494aa5656c9355e6b5d8d3  gh-ost-binary-osx-20180527215024.tar.gz
b6b452c17b44da537f92f29514ab5fdd4dbe795f671c8dd08f5962f31e7ba2ec  gh-ost_1.0.46_amd64.deb

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-1.0.46-1.x86_64.rpm(2.81 MB)
    gh-ost-binary-linux-20180527215024.tar.gz(2.84 MB)
    gh-ost-binary-osx-20180527215024.tar.gz(2.82 MB)
    gh-ost_1.0.46_amd64.deb(2.84 MB)

  • v1.0.45(Apr 17, 2018)

    Changes since previous release, 10.0.44: https://github.com/github/gh-ost/compare/v1.0.44...v1.0.45

    Notable:

    • Support for Aliyun RDS, contribution by @zhangxiaojian, #541
    • Support gbk Encoding , contribution by @ceshihao, #533
    • Support exponential backoff for cutover attempts, contribution by @kurtzur, #563
    • Fix to long JSON values
    • CI to run migration tests (previously executed manually outside CI flow), #546, blog post
    • Tests for GEOMETRY and POINT data types
    • Avoid printing password in output log

    Thank you to all contributors!

    $ sha256sum *
c0319eee757e544eb2be298ba115c3a6e6296d3c5ae44929d64009f3f86a62d4  gh-ost-binary-linux-20180417090238.tar.gz
68fe507863a323e690a9899395314f52925cc173b38272dd91deb29cb5a1005b  gh-ost-binary-osx-20180417090238.tar.gz

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20180417090238.tar.gz(2.82 MB)
    gh-ost-binary-osx-20180417090238.tar.gz(2.82 MB)

  • v1.0.44(Feb 11, 2018)

    Changes since 1.0.42: https://github.com/github/gh-ost/compare/v1.0.42...v1.0.44

    Notable:

    • --postpone-cut-over-flag-file implies touching indicated file
    • support for PK column UPDATE (fixes #519)
    • Implement teardown (#479 ), now gh-ost can be embedded as a library -- thanks @nikhilmat !
    • Allow gh-ost to modify the server using extra port (#491 ) -- thanks @zmoazeni !
    • Doc updates -- thanks @druud !
    • Remove plaintext password from log
    • Building via go1.9
    $ sha256 *
583eb9fe9055b422521a7a60083860248bf5a52226dceb561cb60d8520bfecc0  gh-ost-binary-linux-20180211093207.tar.gz
6940fde2720179f4b3c4c74b906e746060b18a0a1e3229dcf818cd944a57dfd7  gh-ost-binary-osx-20180211093207.tar.gz

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20180211093207.tar.gz(2.73 MB)
    gh-ost-binary-osx-20180211093207.tar.gz(2.73 MB)

  • v1.0.42(Sep 14, 2017)

    Changes since 1.0.40: https://github.com/github/gh-ost/compare/v1.0.40...v1.0.42

    Notable:

    • Support for 5.7 JSON data type
    • Write optimization: client side prepares statements, reduce network traffic overhead, https://github.com/github/gh-ost/pull/459
    • Read optimization: avoid generating temporary tables in read query, https://github.com/github/gh-ost/pull/471 thanks @ggunson
    • Using autocommit=1, fixes https://github.com/github/gh-ost/issues/480, thanks @renecannao
    • Added --force-table-names controlling, see https://github.com/github/gh-ost/pull/457, thanks @arthurnn
    • Validating password length (capped at 32, see https://github.com/github/gh-ost/pull/489)
    • Stricter data type validation pre-migration
    • more...

    Thank you kindly to all contributors!

    $ md5sum *
07617a0058396336d0d0686253ae602d  gh-ost-binary-linux-20170914095800.tar.gz
673574c92bf18921625758a05e0bed98  gh-ost-binary-osx-20170914095800.tar.gz

    $ sha1sum *
444589e9417653a8636838cb6ecbe7fb908e7609  gh-ost-binary-linux-20170914095800.tar.gz
9a79e741ea28e1f55d7cf49b9813f151d4b52362  gh-ost-binary-osx-20170914095800.tar.gz

    $ sha256sum *
297b14ed1efe9b894e27e63f7aff36ceded7334c174f5a8602af9e74beb2f512  gh-ost-binary-linux-20170914095800.tar.gz
0fd2497b21305dbd420f94d61703a33a70fcf2b0b54bbfb55e178691d061ff2d  gh-ost-binary-osx-20170914095800.tar.gz

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20170914095800.tar.gz(2.62 MB)
    gh-ost-binary-osx-20170914095800.tar.gz(2.62 MB)

  • v1.0.40(Jul 21, 2017)

    Changes since 1.0.36: https://github.com/github/gh-ost/compare/v1.0.36...v1.0.40

    Notable:

    • Fix to data integrity issue: corruption in timestamp(6) and datetime(6) microsecond values when smaller than 0.1sec, https://github.com/github/gh-ost/pull/452
    • Fix to data integrity issue: kill on the master connection could cause a premature cut-over without all the rows being copied, https://github.com/github/gh-ost/pull/445
    • Fix to data integrity issue: support for alter table .. change column X where X is a different letter case than original column, https://github.com/github/gh-ost/pull/447
    • supporting tables that reach up to and including max length (64 characters), https://github.com/github/gh-ost/pull/451
    • Hibernate on critical load: when critical-load-hibernate-seconds > 0, then the behavior of critical-load changes. Instead of panicking and bailing out with error, gh-ost enters hibernation. https://github.com/github/gh-ost/pull/424
    • dml-batch-size is dynamic, supported up to 1000
    • http-throttle outputs clearer throttle messahes
    • http-throttle is freno aware and outputs clearer throttle messahes
    • fix to critical-load hibernation check
    • support for drop + add a column of same name (gh-ost will recognize the before/after columns to be unrelated)
    • RDS documentation, thanks @jacobbednarz
    • improved documentation, thanks @jessbreckenridge
    • more...
    md5sum
55079a0ea0f72bfccf32658f4c6de894  gh-ost-binary-linux-1.0.40-20170721063804.tar.gz
715309fcd4cd85f974b4ac648f849fa3  gh-ost-binary-osx-1.0.40-20170721063804.tar.gz

sha1sum
b19b8af996cef57d117a961cdf2c27865e9c5ff6  gh-ost-binary-linux-1.0.40-20170721063804.tar.gz
d8b3d862e815319ba62c6f55502cdd670f0b4028  gh-ost-binary-osx-1.0.40-20170721063804.tar.gz

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-1.0.40-20170721063804.tar.gz(2.61 MB)
    gh-ost-binary-osx-1.0.40-20170721063804.tar.gz(2.62 MB)

  • v1.0.36(Apr 3, 2017)

    changes since 1.0.35: https://github.com/github/gh-ost/compare/v1.0.35...v1.0.36

    Notable:

    • supporting --throttle-http=<url>: routinely check given URL, throttle when response code is not 200.
    • fixed "close sync with err: sync is been closing..." message

    gh-ost now supports and uses TravisCI. The community will now have visibility into CI tests on both external and internal PRs.

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20170403125842.tar.gz(2.54 MB)
    gh-ost-binary-osx-20170403125842.tar.gz(2.53 MB)

  • v1.0.35(Mar 13, 2017)

    changes since 1.0.34: https://github.com/github/gh-ost/compare/v1.0.34...v1.0.35

    notable:

    • updated go-mysql library
    • supporting --timestamp-old-table
    • mitigating cut-over/write race condition
    • fixed mediumint unsigned problem
    • added various tests
    • proper documentation for shared key and unique keys requirements -- thanks @ggunson
    • interactive command support question mark (?) as argument, in which case this is a Get operation, gh-ost prints out current value
    • fix to throttle-control-replicas dynamic setting; showing number of throttle control replicas in status
    • added "common questions" doc page
    • cut-over failure on test-on-replica kicks replication running again before reattempt
    • collecting and presenting MySQL versions of applier and inspector
    • migration only starting after first replication lag metric collected
    • more...

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20170313063856.tar.gz(2.16 MB)
    gh-ost-binary-osx-20170313063856.tar.gz(2.16 MB)

  • v1.0.34(Jan 9, 2017)

    changes since 1.0.32: https://github.com/github/gh-ost/compare/v1.0.32...v1.0.34

    notable:

    • Batched writes for applying binlog events onto ghost table: solving https://github.com/github/gh-ost/issues/348, this increases write capacity for gh-ost without introducing delays or stalls. See https://github.com/github/gh-ost/issues/352
    • Only using internal heartbeat mechanism. --replication-lag-query is now deprecated and unused. gh-ost has its own subsecond resolution replication lag measurement which it uses both on the connected server as well as on the control-replicas. See https://github.com/github/gh-ost/pull/347
    • support for --master-user and --master-password, for when master and replicas have different credentials.
    • Add possibility to set server-id as command line flag - contribution by @cypiszzz, see https://github.com/github/gh-ost/pull/349
    • --check-flag is a flag with which one can check if other flags are supported. Used for scripting and testing in lieu of evolving and changing versions. See https://github.com/github/gh-ost/pull/351

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20170109081400.tar.gz(2.14 MB)
    gh-ost-binary-osx-20170109081400.tar.gz(2.14 MB)

  • v1.0.32(Dec 14, 2016)

  • v1.0.30(Dec 6, 2016)

    Many additions since 1.0.28. The ultimate breakdown is https://github.com/github/gh-ost/compare/v1.0.28...v1.0.30

    Notable:

    • Fixed cut-over stall: when cut-over happened to start executing even as lag or otherwise a throttling condition also emerged, a race condition would make the cut-over stall until throttling reason went away. The fix is that cut-over does not wait, and at worst, rolls back.
    • fix to cut-over retries -- it didn't throttle properly in between and did not cleanup properly in between.
    • cut-over timeout on "unknown" stalls. An example to "unknown" was the throttling problem described above. I don't know what other unknowns there are, but cut-over is now generically resilient to them.
    • support for --skip-foreign-key-checks - can save inspection time, when one is certain no FKs actually exist
    • Made column-rename detection heuristic more robust (it could wrongly infer a rename by some unfortunate comment or column-name)
    • Reading and reporting replication lag before waiting on initial replication event -- this adds visibility in seemingly-hanging operations where the replica is lagging in the first place
    • changelog table uses same engine as original table
    • on startup, verifying replication works all all the way up to the master.
    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20161206132017.tar.gz(2.13 MB)
    gh-ost-binary-osx-20161206132017.tar.gz(2.13 MB)

  • v1.0.28(Nov 1, 2016)

    Many additions since 1.0.21. The ultimate breakdown is https://github.com/github/gh-ost/compare/v1.0.21...v1.0.28

    Notable:

    • support for DATETIME subsecond resolution
    • user-commanded throttling implies complete silence, not even heartbeat is written
    • fixed log_slave_updates check on --test-on-replica and --migrate-on-replica
    • cleanly closing streamer connection (connection that imposes as a replica) upon completion
    • on --exact-rowcount, --concurrent-rowcount now defaults true
    • added validation to confirm table structure on master & replica is identical (yes, the counter case happened)
    • correct (though suboptimal) behavior on enum being part of the migration key
    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20161101125005.tar.gz(2.51 MB)
    gh-ost-binary-osx-20161101125005.tar.gz(2.50 MB)

  • v1.0.21(Oct 13, 2016)

    Many additions since 1.0.20. The ultimate breakdown is https://github.com/github/gh-ost/compare/v1.0.20...v1.0.21

    Notable:

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20161013142859.tar.gz(2.50 MB)
    gh-ost-binary-osx-20161013142859.tar.gz(2.49 MB)

  • v1.0.20(Sep 13, 2016)

  • v1.0.18-experimental(Sep 8, 2016)

  • v1.0.17(Sep 5, 2016)

    Notable:

    • Support hooks: external processes invoked by gh-ost at specific points in execution
    • Sub-second lag resolution: if you require your replicas to lag within subsecond threshold, this is now supported
    • RDS: this release should work with RDS, see discussion -- we will formalize documentation once behavior is tested more
    • Tungsten: supply --tungsten --assume-master-host=the.master.com
    • master-master improved support: explictly specify the master via --allow-master-master --assume-master-host=the.master.com. See cheatsheet
    • Improved status report (ETA shows estimate even while throttling)
    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-v1.0.17-20160905104058.tar.gz(2.45 MB)
    gh-ost-binary-osx-v1.0.17-20160905104058.tar.gz(2.44 MB)

  • v1.0.14(Aug 26, 2016)

    Notable:

    • --concurrent-rowcount (https://github.com/github/gh-ost/issues/194): with --exact-rowcount, issue the count(*) concurrently even as row copy takes place (row count gets updated throughout the migration)
    • Using STRICT_ALL_TABLES for all operations, as safety net ()
    • Fixed support for 5.5 (https://github.com/github/gh-ost/issues/184)
    • Relaxed config file parser (https://github.com/github/gh-ost/issues/198)
    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20160826083132.tar.gz(2.37 MB)
    gh-ost-binary-osx-20160826083132.tar.gz(2.37 MB)

  • v1.0.13(Aug 24, 2016)

  • v1.0.11(Aug 22, 2016)

  • v1.0.10(Aug 18, 2016)

  • v1.0.9(Aug 15, 2016)

  • v1.0.8(Aug 1, 2016)

Owner
GitHub
How people build software.
GitHub
GitHub
Database migrations. CLI and Golang library.

migrate Database migrations written in Go. Use as CLI or import as library. Migrate reads migrations from sources and applies them in correct order to

null 10.6k Jan 9, 2023
Database migrations written in Go

migrate Database migrations written in Go. Use as CLI or import as library. Migrate reads migrations from sources and applies them in correct order to

null 0 Oct 3, 2021
mysql to mysql 轻量级多线程的库表数据同步

goMysqlSync golang mysql to mysql 轻量级多线程库表级数据同步 测试运行 设置当前binlog位置并且开始运行 go run main.go -position mysql-bin.000001 1 1619431429 查询当前binlog位置，参数n为秒数，查询结

null 14 Nov 15, 2022
Manage Schema for KubeDB managed Databases

schema-manager Manage Schema for KubeDB managed Databases Installation To install KubeDB, please follow the guide here. Using KubeDB Want to learn how

Kubernetes Database 8 Feb 19, 2022
Go library that stores data in Redis with SQL-like schema

Go library that stores data in Redis with SQL-like schema. The goal of this library is we can store data in Redis with table form.

kaharman 2 Mar 14, 2022
Web-based, zero-config, dependency-free database schema change and version control tool for teams

Live Demo • Install • Help • Development • Design Doc Bytebase is a web-based, zero-config, dependency-free database schema change and version control

Bytebase 4.6k Jan 1, 2023
a powerful mysql toolset with Go

go-mysql A pure go library to handle MySQL network protocol and replication. Call for Committer/Maintainer Sorry that I have no enough time to maintai

siddontang 3.9k Dec 28, 2022
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

siddontang 3.9k Dec 30, 2022
A high-performance MySQL proxy

kingshard 中文主页 Overview kingshard is a high-performance proxy for MySQL powered by Go. Just like other mysql proxies, you can use it to split the read

Fei Chen 6.1k Dec 30, 2022
Golang MySql binary log replication listener

Go MySql binary log replication listener Pure Go Implementation of MySQL replication protocol. This allow you to receive event like insert, update, de

Pavel <Ven> Gulbin 189 Oct 25, 2022
MySQL replication topology management and HA

orchestrator [Documentation] orchestrator is a MySQL high availability and replication management tool, runs as a service and provides command line ac

null 4.8k Jan 4, 2023
Vitess is a database clustering system for horizontal scaling of MySQL.

Vitess Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vite

Vitess 15.3k Jan 3, 2023
db-recovery is a tool for recovering MySQL data.

db-recovery is a tool for recovering MySQL data. It is used in scenarios where the database has no backup or binlog. It can parse data files and redo/undo logs to recover data.

null 25 Nov 17, 2022
一个使 mysql，pgsql 数据库表自动生成 go struct 的工具

db2go 一个使 mysql、pgsql 数据库表自动生成 go struct 的工具 快速使用 将项目放入到GOPATH/src目录下

易水韩 19 Nov 25, 2022
🐳 A most popular sql audit platform for mysql

?? A most popular sql audit platform for mysql

Henry Yee 7.3k Jan 6, 2023
Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

?? Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

PingCAP 267 Nov 9, 2022
GoRose(go orm), a mini database ORM for golang, which inspired by the famous php framwork laravle's eloquent. It will be friendly for php developer and python or ruby developer. Currently provides six major database drivers: mysql,sqlite3,postgres,oracle,mssql, Clickhouse.

GoRose ORM _______ ______ .______ ______ _______. _______ / _____| / __ \ | _ \ / __ \ / || ____| |

gohouse 1.1k Dec 24, 2022
Vitess is a database clustering system for horizontal scaling of MySQL.

Vitess Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vite

Vitess 15.3k Jan 4, 2023
Gaea is a mysql proxy, it's developed by xiaomi b2c-dev team.

简介 Gaea是小米中国区电商研发部研发的基于mysql协议的数据库中间件，目前在小米商城大陆和海外得到广泛使用，包括订单、社区、活动等多个业务。Gaea支持分库分表、sql路由、读写分离等基本特性，更多详细功能可以参照下面的功能列表。其中分库分表方案兼容了mycat和kingshard两个项目的路

Xiaomi 2.4k Dec 30, 2022