GitHub's Online Schema Migrations for MySQL

Overview

gh-ost

build status downloads release

GitHub's online schema migration for MySQL

gh-ost is a triggerless online schema migration solution for MySQL. It is testable and provides pausability, dynamic control/reconfiguration, auditing, and many operational perks.

gh-ost produces a light workload on the master throughout the migration, decoupled from the existing workload on the migrated table.

It has been designed based on years of experience with existing solutions, and changes the paradigm of table migrations.

How?

All existing online-schema-change tools operate in similar manner: they create a ghost table in the likeness of your original table, migrate that table while empty, slowly and incrementally copy data from your original table to the ghost table, meanwhile propagating ongoing changes (any INSERT, DELETE, UPDATE applied to your table) to the ghost table. Finally, at the right time, they replace your original table with the ghost table.

gh-ost uses the same pattern. However it differs from all existing tools by not using triggers. We have recognized the triggers to be the source of many limitations and risks.

Instead, gh-ost uses the binary log stream to capture table changes, and asynchronously applies them onto the ghost table. gh-ost takes upon itself some tasks that other tools leave for the database to perform. As result, gh-ost has greater control over the migration process; can truly suspend it; can truly decouple the migration's write load from the master's workload.

In addition, it offers many operational perks that make it safer, trustworthy and fun to use.

gh-ost general flow

Highlights

  • Build your trust in gh-ost by testing it on replicas. gh-ost will issue same flow as it would have on the master, to migrate a table on a replica, without actually replacing the original table, leaving the replica with two tables you can then compare and satisfy yourself that the tool operates correctly. This is how we continuously test gh-ost in production.
  • True pause: when gh-ost throttles, it truly ceases writes on master: no row copies and no ongoing events processing. By throttling, you return your master to its original workload
  • Dynamic control: you can interactively reconfigure gh-ost, even as migration still runs. You may forcibly initiate throttling.
  • Auditing: you may query gh-ost for status. gh-ost listens on unix socket or TCP.
  • Control over cut-over phase: gh-ost can be instructed to postpone what is probably the most critical step: the swap of tables, until such time that you're comfortably available. No need to worry about ETA being outside office hours.
  • External hooks can couple gh-ost with your particular environment.

Please refer to the docs for more information. No, really, read the docs.

Usage

The cheatsheet has it all. You may be interested in invoking gh-ost in various modes:

  • a noop migration (merely testing that the migration is valid and good to go)
  • a real migration, utilizing a replica (the migration runs on the master; gh-ost figures out identities of servers involved. Required mode if your master uses Statement Based Replication)
  • a real migration, run directly on the master (but gh-ost prefers the former)
  • a real migration on a replica (master untouched)
  • a test migration on a replica, the way for you to build trust with gh-ost's operation.

Our tips:

  • Testing above all, try out --test-on-replica first few times. Better yet, make it continuous. We have multiple replicas where we iterate our entire fleet of production tables, migrating them one by one, checksumming the results, verifying migration is good.
  • For each master migration, first issue a noop
  • Then issue the real thing via --execute.

More tips:

  • Use --exact-rowcount for accurate progress indication
  • Use --postpone-cut-over-flag-file to gain control over cut-over timing
  • Get familiar with the interactive commands

Also see:

What's in a name?

Originally this was named gh-osc: GitHub Online Schema Change, in the likes of Facebook online schema change and pt-online-schema-change.

But then a rare genetic mutation happened, and the c transformed into t. And that sent us down the path of trying to figure out a new acronym. gh-ost (pronounce: Ghost), stands for GitHub's Online Schema Transmogrifier/Translator/Transformer/Transfigurator

License

gh-ost is licensed under the MIT license

gh-ost uses 3rd party libraries, each with their own license. These are found here.

Community

gh-ost is released at a stable state, but with mileage to go. We are open to pull requests. Please first discuss your intentions via Issues.

We develop gh-ost at GitHub and for the community. We may have different priorities than others. From time to time we may suggest a contribution that is not on our immediate roadmap but which may appeal to others.

Please see Coding gh-ost for a guide to getting started developing with gh-ost.

Download/binaries/source

gh-ost is now GA and stable.

gh-ost is available in binary format for Linux and Mac OS/X

Download latest release here

gh-ost is a Go project; it is built with Go 1.14 and above. To build on your own, use either:

  • script/build - this is the same build script used by CI hence the authoritative; artifact is ./bin/gh-ost binary.
  • build.sh for building tar.gz artifacts in /tmp/gh-ost

Generally speaking, master branch is stable, but only releases are to be used in production.

Authors

gh-ost is designed, authored, reviewed and tested by the database infrastructure team at GitHub:

Issues
  • Delete records not in sync between table and gho table

    Delete records not in sync between table and gho table

    We recently had an issue where we discovered that a gh-ost migration that was run on a table wasn't syncing DELETEs between the table being migrated from and the new table. I was able to replicate the issue in our staging environment, and it was very clear that the two tables were out of sync while the migration was being run.

    Is there anything we're missing on our end? Did we misunderstand that gh-ost should be keeping those tables in sync during a migration?

    ./gh-ost \
    --user="user" \
    --password="pass" \
    --host=host \
    --database="db" \
    --table="ourTable" \
    --allow-on-master \
    --verbose \
    --postpone-cut-over-flag-file="./postpone-cut.txt" \
    --discard-foreign-keys \
    --alter="PARTITION BY HASH(parkId) PARTITIONS 101" \
    --initially-drop-ghost-table \
    --timestamp-old-table \
    --execute
    

    This is the command that we ran. We use RDS with bin_log format = ROW, if that helps

    opened by hunthunt2010 1
  • fix(lost data in mysql two-phase commit): lost data in mysql two-phas…

    fix(lost data in mysql two-phase commit): lost data in mysql two-phas…

    fix: lost data due to mysql tow-phase-commit

    A Pull Request should be associated with an Issue. We wish to have discussions in Issues. A single issue may be targeted by multiple PRs. If you're offering a new feature or fixing anything, we'd like to know beforehand in Issues, and potentially we'll be able to point development in a particular direction.

    Related issue: https://github.com/github/gh-ost/issues/1030

    Further notes in https://github.com/github/gh-ost/blob/master/.github/CONTRIBUTING.md Thank you! We are open to PRs, but please understand if for technical reasons we are unable to accept each and any PR

    Description This PR [fix lost data due to mysql tow-phase-commit]

    bug low-risk 
    opened by shaohk 0
  • Fix `integer divide by zero` panic in migrator

    Fix `integer divide by zero` panic in migrator

    Related issue: https://github.com/github/gh-ost/1117

    Description

    This PR resolves an integer divide by zero panic when migrationContext.HooksStatusIntervalSec is zero. After this PR, no status will be printed when the interval is set to zero

    In case this PR introduced Go code changes:

    • [x] contributed code is using same conventions as original code
    • [x] script/cibuild returns with no formatting errors, build errors or unit test errors.
    low-risk 
    opened by timvaillancourt 0
  • Changlog table comment

    Changlog table comment

    Related issue: https://github.com/github/gh-ost/issues/1133

    Description

    This PR adds the table-level comment gh-ost changlelog to the changelog table, in order to make it easier to find using information_schema.tables

    After this PR, the following query should find gh-ost changelog tables:

    SELECT table_schema, table_name FROM information_schema.tables
        WHERE table_name LIKE '%_ghc'
        AND table_comment='gh-ost changelog'
    

    In case this PR introduced Go code changes:

    • [x] contributed code is using same conventions as original code
    • [x] script/cibuild returns with no formatting errors, build errors or unit test errors.
    opened by timvaillancourt 0
  • Remove unused code in `go/mysql/binlog.go`

    Remove unused code in `go/mysql/binlog.go`

    Description

    This PR removes unused code in go/mysql/binlog.go to simplify maintenance and a future move of BinlogCoordinates to an interface, in order to facilitate adding GTID support to gh-ost

    Details:

    • Remove unused funcs (and unit tests):
      • .FileSmallerThan()
      • .FileNumberDistance()
      • .FileNumber()
      • .PreviousFileCoordinatesBy()
      • .PreviousFileCoordinates()
      • .NextFileCoordinates()
      • .DetachedCoordinates()
    • Remove unused Type field from BinlogCoordinates struct

    In case this PR introduced Go code changes:

    • [x] contributed code is using same conventions as original code
    • [x] script/cibuild returns with no formatting errors, build errors or unit test errors.
    enhancement 
    opened by timvaillancourt 0
  • Default to `go1.17.11`

    Default to `go1.17.11`

    Description

    This PR defaults the build to go1.17.11

    This resolves https://github.com/github/gh-ost/issues/1128

    In case this PR introduced Go code changes:

    • [x] contributed code is using same conventions as original code
    • [x] script/cibuild returns with no formatting errors, build errors or unit test errors.
    build low-risk 
    opened by timvaillancourt 0
Releases(v1.1.4)
  • v1.1.4(Feb 9, 2022)

    Changes since 1.1.3: https://github.com/github/gh-ost/compare/v1.1.3...v1.1.4

    gh-ost v1.1.4 is based on v1.1.3, plus commits cherry-picked from master branch.

    Notable:

    • Add flag to customize the interval which the onStatus hook is called, #1083, thanks @tknodell-recurly
    • Add docs for hooks-status-interval, #1090
    • Reduce the minimal chunk size from 100 to 10, #1087, thanks @arthurschreiber

    This release is not based on the master branch as it contains a known issue that is currently being investigated - please note that the master branch of gh-ost is not considered stable; only releases are considered production-ready.

    Thank you to all contributors!

    Updated 25th Feb 2022: Binaries have been re-uploaded to fix an issue where RPMs were being generated with an OS for darwin instead of linux. (thanks @kedarvj for the report!)

    Updated 11th Mar 2022: Added macOS arm64 binary to support M1 Macs (thanks @jecepeda for adding support in #1108!)

    $ sha256sum *
    da2b2a943aaf18fb849b52e19b0828c9878b180546d1adb31ad03b3cf7c50d53  gh-ost
    d626f6ae582ca15c58a387c819c410788a78f64dd7ac7ec946df07f0bf66f7a0  gh-ost-1.1.4-1.x86_64.rpm
    ccc055a9fa2ca080b99aeb58a6aacf557582c317b8f36ee13c551d87cd5eeb30  gh-ost-binary-linux-20220225143506.tar.gz
    9feebe056914335d23127db10de8f20631e0df448b1382f30366a462ab5d5473  gh-ost-binary-osx-20220225143506.tar.gz
    66619ad49fb7a2a13edfe08aee110a554f7174b44453451fee2e532661f1f254  gh-ost-binary-osx-arm64-20220311164158.tar.gz
    1dead2c07b73a8730ec4b923433be143dcb4dc18719e2ad640962e389168a22d  gh-ost_1.1.4_amd64.deb
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost(8.68 MB)
    gh-ost-1.1.4-1.x86_64.rpm(4.66 MB)
    gh-ost-binary-linux-20220225143506.tar.gz(4.67 MB)
    gh-ost-binary-osx-amd64-20220225143506.tar.gz(4.61 MB)
    gh-ost-binary-osx-arm64-20220311164158.tar.gz(4.40 MB)
    gh-ost_1.1.4_amd64.deb(4.67 MB)
  • v1.1.3(Jan 31, 2022)

    Changes since 1.1.2: https://github.com/github/gh-ost/compare/v1.1.2...v1.1.3

    gh-ost v1.1.3 is comprised of v1.1.2 with a fix for security vulnerability GHSA-rrp4-2xx3-mv29, plus low-risk documentation updates cherry-picked from master branch.

    This release is not based on the master branch as it contains a known issue that is currently being investigated - please note that the master branch of gh-ost is not considered stable; only releases are considered production-ready.

    Notable:

    • Security vulnerability: Improper Input Validation of -database Parameter, GHSA-rrp4-2xx3-mv29, thanks @dwisiswant0
    • doc/triggerless-design.md: typo, #1011, thanks @eddroid
    • Document the flag critical-load-hibernate-seconds, improve grammar of flags documentation, #1066, thanks @EagleEyeJohn
    • Update hook environment variable docs, #1076
    • Update build script to generate releases, #1078
    • Remove CGO dependency in go-mysql in Gh-ost v1.1.3, #1079

    Thank you to all contributors!

    Updated 8th Feb 2022: Binaries have been re-uploaded with the correct version number; previously they were incorrectly labelled as v1.1.2 instead of v1.1.3. (thanks @dharnett for notifying us!)

    Updated 25th Feb 2022: Binaries have been re-uploaded to fix an issue where RPMs were being generated with an OS for darwin instead of linux. (thanks @kedarvj for the report!)

    $ sha256sum *
    10255fa8690001d7eb37f51c6d41aba0e55ee9ef4c2b949ca4c0604011b74d71  gh-ost
    496ac03a36738e752ad98668a3bb97eca913039431fdff80732fbb4acfac8991  gh-ost-1.1.3-1.x86_64.rpm
    c432e7d711268117bc8c13cebcf3e3cf3c986557a205cc041e0ea63937700ef5  gh-ost-binary-linux-20220225143057.tar.gz
    e55b3c28c655eab28617a47eefa88fb6a508af506652bafdd226293968bebd8e  gh-ost-binary-osx-20220225143057.tar.gz
    116db1f8f5e4bff6a8903f057dad07e14ad36eea0f3b462a16342b6d3b447d1a  gh-ost_1.1.3_amd64.deb
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost(8.68 MB)
    gh-ost-1.1.3-1.x86_64.rpm(4.66 MB)
    gh-ost-binary-linux-20220225143057.tar.gz(4.67 MB)
    gh-ost-binary-osx-20220225143057.tar.gz(4.61 MB)
    gh-ost_1.1.3_amd64.deb(4.67 MB)
  • v1.1.2(Jun 17, 2021)

    Changes since 1.1.1: https://github.com/github/gh-ost/compare/v1.1.1...v1.1.2

    Notable:

    • Copy auto increment, #911, thanks @shlomi-noach
    • Cut-over should wait for heartbeat lag to be low enough to succeed, #921, thanks @ccoffey
    • All MySQL DBs limited to max 3 concurrent/idle connections, #931, thanks @shlomi-noach
    • Report GH_OST_ETA_SECONDS ETA as part of migration context, #936, thanks @shlomi-noach
    • Remove build_id files from rpm, #954, thanks @yakirgb
    • Fix bug with -exact-rowcount on MySQL 8.0.2+, #962, thanks @Fanduzi
    • Support enum to varchar, #963, thanks @shlomi-noach
    • Set the ServerName for TLS configuration, #988, thanks @dbussink
    • Use Golang 1.16 and temporarily disable go modules in builds, #966
    • Use matrix build for replica test CI, #968

    Thank you to all contributors!

    $ sha256sum *
    342a7f4760d51dc05154dd1e6663f701bb024868a7f9b0804e8bf38afe67e0ff  gh-ost
    255dbaed5de7c20b1f11c82ed0275e0ecfaa8f8afa4f38fd36705cfac4b6207f  gh-ost-1.1.2-1.x86_64.rpm
    98ebff834bf930a198d3506cda0ee1a6a3ea67f09e9112ca8a2534ef4f74288c  gh-ost-binary-linux-20210617134741.tar.gz
    0b000805894ae5cc6b5ff0aacb67e4273328354f5e559a9ad39b1e7cc617b316  gh-ost-binary-osx-20210617155835.tar.gz
    afaf95ea239b5e473080d0b58592e6449cfc48d936396a8540722c4772741893  gh-ost_1.1.2_amd64.deb
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost(9.40 MB)
    gh-ost-1.1.2-1.x86_64.rpm(4.46 MB)
    gh-ost-binary-linux-20210617134741.tar.gz(4.49 MB)
    gh-ost-binary-osx-20210617155835.tar.gz(4.96 MB)
    gh-ost_1.1.2_amd64.deb(4.49 MB)
  • v1.1.1(May 3, 2021)

    Changes since 1.1.0: https://github.com/github/gh-ost/compare/v1.1.0...v1.1.1

    Notable:

    • Use sync.Once to avoid dropping cutover sentry table twice, #755, thanks @MOON-CLJ
    • Parsing IPV6 addresses, #640, thanks @shlomi-noach
    • Support Azure Database for MySQL, #807, thanks @elemount
    • Use Golang 1.15 and remove Debian Jessie (end-of-life) tarball build, #906
    • Improved connection type logging, #515, thanks @akshaychhajed
    • Fix: Data loss on tables with binary primary key, #915, thanks @cenkore
    • Add applier and inspector interactive commands to socket/TCP server, #951

    Thank you to all contributors!

    $ sha256sum *
    8e802e8685d4b7d757ae47513b918d6de1d0402511cc3fadb2d961713beb6d89  gh-ost
    6b49bbcf1a985d96fa98954d6aba5b2e8fa9f8b0ac9aee2c10bd47755f986254  gh-ost-1.1.1-1.x86_64.rpm
    1188bb2d4539d25fc8a4b8a96cccb9b56a217124a9d4c8b9faab4534acd9258b  gh-ost-binary-linux-20210503150942.tar.gz
    1ef5585f926af0d93af53007b563c86a34d52bd4c05e1be5c6afaf8064199d3e  gh-ost-binary-osx-20210503171035.tar.gz
    02d39afd96e0b5d69baf6a7259871b664445eaa811ccb6c3928376793b56f997  gh-ost_1.1.1_amd64.deb
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost(9.11 MB)
    gh-ost-1.1.1-1.x86_64.rpm(4.57 MB)
    gh-ost-binary-linux-20210503150942.tar.gz(4.60 MB)
    gh-ost-binary-osx-20210503171035.tar.gz(4.95 MB)
    gh-ost_1.1.1_amd64.deb(4.60 MB)
  • v1.1.0(Aug 28, 2020)

    Changes since 1.0.49: https://github.com/github/gh-ost/compare/v1.0.49...v1.1.0

    Notable:

    • Support a complete ALTER TABLE statement in --alter, #865 / #878, thanks @shlomi-noach
    • Support --mysql-timeout flag, #824, thanks @shlomi-noach
    • Add a check to rows.Err after processing all rows, #835, thanks @ajm188
    • Update go-sql-driver to latest, #823, thanks @shlomi-noach
    • Update build to Golang 1.14, #861 / #876, thanks @shlomi-noach
    • Add error checking for an err variable that was left unchecked, #810, thanks @yaserazfar
    • Implement a logging interface, #789 / #864, thanks @abeyum and @jfudally
    • Throttle on HTTP error to throttling API, #833, thanks @jfudally
    • Additional testing:
      • Add latin1 tests with TEXT columns, #291 / #455, thanks @shlomi-noach
      • Add BINARY / VARBINARY tests, #655, thanks @shlomi-noach
      • Add NULL-able INT tests, #692, thanks @shlomi-noach

    Thank you to all contributors!

    $ sha256sum *
    e6de44ed76cd7834d7dac5ee4a53459ce18e16fbb482e9ff5d898f7d83444632  gh-ost-1.1.0-1.x86_64.rpm
    2c29184d5b7f1340999680c06ae6796639f195d00907214a9baddc9ca9391ca9  gh-ost-binary-linux-20200828140552.tar.gz
    c1dc2716dd5b843b4ef11b3fa601885f079363ca5545799bdac7037fd132e723  gh-ost-binary-osx-20200828160625.tar.gz
    6ca0c5f0a79e51035e380a00350a535e04381b29ac9bbb847acf31d3b8985390  gh-ost_1.1.0_amd64.deb
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-1.1.0-1.x86_64.rpm(4.88 MB)
    gh-ost-binary-linux-20200828140552.tar.gz(4.90 MB)
    gh-ost-binary-osx-20200828160625.tar.gz(5.51 MB)
    gh-ost_1.1.0_amd64.deb(4.90 MB)
  • v1.0.49(Feb 9, 2020)

    Changes since 1.0.48: https://github.com/github/gh-ost/compare/v1.0.48...v1.0.49

    Notable:

    • --force-named-panic (on panic command, force table name), named throttle, named no-throttle, #695
    • add --skip-strict-mode option, #712, thank @babinomec
    • Improve SSL support #716, thanks @RafeKettler
    • context, status and hooks: add progressPct and CurrentLag, #788
    • Always use NO_AUTO_VALUE_ON_ZERO #723
    • tests: test on MySQL 8.0
    • tests: Bump default-retries to 3 #754 , thanks @zmoazeni
    • Support and require go1.12
    • doc updates/fixes, thanks @mohitm15, Tobias Johansson, @jchambers
    $ sha256sum *
    628ad01f4b327ee81f01aee1e4e5d2a966ed681fa391d064424270dbd7b0b0b1  gh-ost-1.0.49-1.x86_64.rpm
    a1d7f72e1119bb8a939204a56acbee09eb52c769183a4649e56d6b3b524cb774  gh-ost-binary-linux-20200209110835.tar.gz
    667f8800654410749b1af6053f26109769992adb813eb9edaa154d537cc86ef3  gh-ost_1.0.49_amd64.deb
    95b3ed74f70ffea9f11f4af06fb6e312f0a857bf4fd3c3ad2da31fc7b3359dd3  gh-ost-binary-osx-20200209114711.tar.gz
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-1.0.49-1.x86_64.rpm(4.72 MB)
    gh-ost-binary-linux-20200209110835.tar.gz(4.74 MB)
    gh-ost-binary-osx-20200209114711.tar.gz(5.38 MB)
    gh-ost_1.0.49_amd64.deb(4.74 MB)
  • v1.0.48(Feb 14, 2019)

    Changes since 1.0.47: https://github.com/github/gh-ost/compare/v1.0.47...v1.0.48

    • Update to latest go-mysql library, which solved multiple issues
    • Fix inspector column types #661, thanks @esnunes
    • inspect: remove redundant join conditions #699, thanks @twotwotwo
    • maintainer email address in .deb package
    • added: --hooks-hint-owner, --hooks-hint-token #703
    • Initial SSL Connection Support #705, thanks @brandonbodnar-wk
    • Fixed DECIMAL datatype #684 (the trigger for updating go-mysql)
    • more... thank you to all contributors!
    $ sha256sum *
    cf1f0999c81f194472b9d3f154c21e0848bd1d61697ce1ca9123368b31d718b2  gh-ost-1.0.48-1.x86_64.rpm
    8d0a518064c5803e1d5e8676031e5a5b5b180e67700d476e19eaf68e4a4a583a  gh-ost-binary-linux-20190214020851.tar.gz
    f57a1361315adba33b51ecd9bf8863722385dbcff741f0a4d467087aaded9bc4  gh-ost-binary-osx-20190214120940.tar.gz
    67bf74f1af6f968a59a2725398ccb5167d97e9b0cd1ab00c867425bf86a0d124  gh-ost_1.0.48_amd64.deb
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-1.0.48-1.x86_64.rpm(3.11 MB)
    gh-ost-binary-linux-20190214020851.tar.gz(3.13 MB)
    gh-ost-binary-osx-20190214120940.tar.gz(3.66 MB)
    gh-ost_1.0.48_amd64.deb(3.13 MB)
  • v1.0.47(Oct 16, 2018)

    Changes since 1.0.46: https://github.com/github/gh-ost/compare/v1.0.46...v1.0.47

    • Fix to copyRowsFunc hanging issue. Thank you @wfxiang08 #611
    • Support for GCP (Google Cloud Platform) #634
    • Populate the hooks environment with dry run status. Thank you @jacobbednarz #641
    • Integration test updates
    • Minor documentation updates

    Checksums:

    $ sha256sum *
    1e2659498d8f2b66cd911e52c8282f03fd2de08eb86d6d77aad0c7ec3e6ff9e0  gh-ost-1.0.47-1.x86_64.rpm
    1b00a14d5b358efd210d5371d63d5c84f3ce35717fbb2c15a24b7248b2dede95  gh-ost-binary-linux-20181016015113.tar.gz
    396964f58cb6a72818437cd1a2cf129523b339b6000dd6443f9399841e0d35d9  gh-ost-binary-osx-20181016015113.tar.gz
    8e98c38d9d4129929a95850aecc50f010443626e5ff0ba7ef85c90f24444e199  gh-ost_1.0.47_amd64.deb
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-1.0.47-1.x86_64.rpm(2.81 MB)
    gh-ost-binary-linux-20181016015113.tar.gz(2.84 MB)
    gh-ost-binary-osx-20181016015113.tar.gz(2.82 MB)
    gh-ost_1.0.47_amd64.deb(2.84 MB)
  • v1.0.46(May 28, 2018)

    Changes since 1.0.45: https://github.com/github/gh-ost/compare/v1.0.45...v1.0.46

    • Support for GENERATED columns, #595, based on work by @dveeden
    • Support for zero timestamps on MySQL 5.5 #539
    • Sanity: enforcing binlog_row_image=FULL
    • Avoid prepared statements for changelog table #592 @ggunson
    • Community contributions, thanks @SchumacherFM, @atsushi-ishibashi,
    • Release: rpm and deb packages

    Checksums:

    $ sha256sum *
    9b0741d1b27aab613cdad12c2bfdd70d6879b116b6d41cef3b58eb21d4f14e62  gh-ost-1.0.46-1.x86_64.rpm
    4222af72f19eadfb6ca7335791c8cc9f628f14c751bc324508cf976fc6dd9feb  gh-ost-binary-linux-20180527215024.tar.gz
    4f069500b7d3a9f1396fc615acd4913fb12783d8e8494aa5656c9355e6b5d8d3  gh-ost-binary-osx-20180527215024.tar.gz
    b6b452c17b44da537f92f29514ab5fdd4dbe795f671c8dd08f5962f31e7ba2ec  gh-ost_1.0.46_amd64.deb
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-1.0.46-1.x86_64.rpm(2.81 MB)
    gh-ost-binary-linux-20180527215024.tar.gz(2.84 MB)
    gh-ost-binary-osx-20180527215024.tar.gz(2.82 MB)
    gh-ost_1.0.46_amd64.deb(2.84 MB)
  • v1.0.45(Apr 17, 2018)

    Changes since previous release, 10.0.44: https://github.com/github/gh-ost/compare/v1.0.44...v1.0.45

    Notable:

    • Support for Aliyun RDS, contribution by @zhangxiaojian, #541
    • Support gbk Encoding , contribution by @ceshihao, #533
    • Support exponential backoff for cutover attempts, contribution by @kurtzur, #563
    • Fix to long JSON values
    • CI to run migration tests (previously executed manually outside CI flow), #546, blog post
    • Tests for GEOMETRY and POINT data types
    • Avoid printing password in output log

    Thank you to all contributors!

    $ sha256sum *
    c0319eee757e544eb2be298ba115c3a6e6296d3c5ae44929d64009f3f86a62d4  gh-ost-binary-linux-20180417090238.tar.gz
    68fe507863a323e690a9899395314f52925cc173b38272dd91deb29cb5a1005b  gh-ost-binary-osx-20180417090238.tar.gz
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20180417090238.tar.gz(2.82 MB)
    gh-ost-binary-osx-20180417090238.tar.gz(2.82 MB)
  • v1.0.44(Feb 11, 2018)

    Changes since 1.0.42: https://github.com/github/gh-ost/compare/v1.0.42...v1.0.44

    Notable:

    • --postpone-cut-over-flag-file implies touching indicated file
    • support for PK column UPDATE (fixes #519)
    • Implement teardown (#479 ), now gh-ost can be embedded as a library -- thanks @nikhilmat !
    • Allow gh-ost to modify the server using extra port (#491 ) -- thanks @zmoazeni !
    • Doc updates -- thanks @druud !
    • Remove plaintext password from log
    • Building via go1.9
    $ sha256 *
    583eb9fe9055b422521a7a60083860248bf5a52226dceb561cb60d8520bfecc0  gh-ost-binary-linux-20180211093207.tar.gz
    6940fde2720179f4b3c4c74b906e746060b18a0a1e3229dcf818cd944a57dfd7  gh-ost-binary-osx-20180211093207.tar.gz
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20180211093207.tar.gz(2.73 MB)
    gh-ost-binary-osx-20180211093207.tar.gz(2.73 MB)
  • v1.0.42(Sep 14, 2017)

    Changes since 1.0.40: https://github.com/github/gh-ost/compare/v1.0.40...v1.0.42

    Notable:

    • Support for 5.7 JSON data type
    • Write optimization: client side prepares statements, reduce network traffic overhead, https://github.com/github/gh-ost/pull/459
    • Read optimization: avoid generating temporary tables in read query, https://github.com/github/gh-ost/pull/471 thanks @ggunson
    • Using autocommit=1, fixes https://github.com/github/gh-ost/issues/480, thanks @renecannao
    • Added --force-table-names controlling, see https://github.com/github/gh-ost/pull/457, thanks @arthurnn
    • Validating password length (capped at 32, see https://github.com/github/gh-ost/pull/489)
    • Stricter data type validation pre-migration
    • more...

    Thank you kindly to all contributors!

    $ md5sum *
    07617a0058396336d0d0686253ae602d  gh-ost-binary-linux-20170914095800.tar.gz
    673574c92bf18921625758a05e0bed98  gh-ost-binary-osx-20170914095800.tar.gz
    
    $ sha1sum *
    444589e9417653a8636838cb6ecbe7fb908e7609  gh-ost-binary-linux-20170914095800.tar.gz
    9a79e741ea28e1f55d7cf49b9813f151d4b52362  gh-ost-binary-osx-20170914095800.tar.gz
    
    $ sha256sum *
    297b14ed1efe9b894e27e63f7aff36ceded7334c174f5a8602af9e74beb2f512  gh-ost-binary-linux-20170914095800.tar.gz
    0fd2497b21305dbd420f94d61703a33a70fcf2b0b54bbfb55e178691d061ff2d  gh-ost-binary-osx-20170914095800.tar.gz
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20170914095800.tar.gz(2.62 MB)
    gh-ost-binary-osx-20170914095800.tar.gz(2.62 MB)
  • v1.0.40(Jul 21, 2017)

    Changes since 1.0.36: https://github.com/github/gh-ost/compare/v1.0.36...v1.0.40

    Notable:

    • Fix to data integrity issue: corruption in timestamp(6) and datetime(6) microsecond values when smaller than 0.1sec, https://github.com/github/gh-ost/pull/452
    • Fix to data integrity issue: kill on the master connection could cause a premature cut-over without all the rows being copied, https://github.com/github/gh-ost/pull/445
    • Fix to data integrity issue: support for alter table .. change column X where X is a different letter case than original column, https://github.com/github/gh-ost/pull/447
    • supporting tables that reach up to and including max length (64 characters), https://github.com/github/gh-ost/pull/451
    • Hibernate on critical load: when critical-load-hibernate-seconds > 0, then the behavior of critical-load changes. Instead of panicking and bailing out with error, gh-ost enters hibernation. https://github.com/github/gh-ost/pull/424
    • dml-batch-size is dynamic, supported up to 1000
    • http-throttle outputs clearer throttle messahes
    • http-throttle is freno aware and outputs clearer throttle messahes
    • fix to critical-load hibernation check
    • support for drop + add a column of same name (gh-ost will recognize the before/after columns to be unrelated)
    • RDS documentation, thanks @jacobbednarz
    • improved documentation, thanks @jessbreckenridge
    • more...
    md5sum
    55079a0ea0f72bfccf32658f4c6de894  gh-ost-binary-linux-1.0.40-20170721063804.tar.gz
    715309fcd4cd85f974b4ac648f849fa3  gh-ost-binary-osx-1.0.40-20170721063804.tar.gz
    
    sha1sum
    b19b8af996cef57d117a961cdf2c27865e9c5ff6  gh-ost-binary-linux-1.0.40-20170721063804.tar.gz
    d8b3d862e815319ba62c6f55502cdd670f0b4028  gh-ost-binary-osx-1.0.40-20170721063804.tar.gz
    

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-1.0.40-20170721063804.tar.gz(2.61 MB)
    gh-ost-binary-osx-1.0.40-20170721063804.tar.gz(2.62 MB)
  • v1.0.36(Apr 3, 2017)

    changes since 1.0.35: https://github.com/github/gh-ost/compare/v1.0.35...v1.0.36

    Notable:

    • supporting --throttle-http=<url>: routinely check given URL, throttle when response code is not 200.
    • fixed "close sync with err: sync is been closing..." message

    gh-ost now supports and uses TravisCI. The community will now have visibility into CI tests on both external and internal PRs.

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20170403125842.tar.gz(2.54 MB)
    gh-ost-binary-osx-20170403125842.tar.gz(2.53 MB)
  • v1.0.35(Mar 13, 2017)

    changes since 1.0.34: https://github.com/github/gh-ost/compare/v1.0.34...v1.0.35

    notable:

    • updated go-mysql library
    • supporting --timestamp-old-table
    • mitigating cut-over/write race condition
    • fixed mediumint unsigned problem
    • added various tests
    • proper documentation for shared key and unique keys requirements -- thanks @ggunson
    • interactive command support question mark (?) as argument, in which case this is a Get operation, gh-ost prints out current value
    • fix to throttle-control-replicas dynamic setting; showing number of throttle control replicas in status
    • added "common questions" doc page
    • cut-over failure on test-on-replica kicks replication running again before reattempt
    • collecting and presenting MySQL versions of applier and inspector
    • migration only starting after first replication lag metric collected
    • more...

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20170313063856.tar.gz(2.16 MB)
    gh-ost-binary-osx-20170313063856.tar.gz(2.16 MB)
  • v1.0.34(Jan 9, 2017)

    changes since 1.0.32: https://github.com/github/gh-ost/compare/v1.0.32...v1.0.34

    notable:

    • Batched writes for applying binlog events onto ghost table: solving https://github.com/github/gh-ost/issues/348, this increases write capacity for gh-ost without introducing delays or stalls. See https://github.com/github/gh-ost/issues/352
    • Only using internal heartbeat mechanism. --replication-lag-query is now deprecated and unused. gh-ost has its own subsecond resolution replication lag measurement which it uses both on the connected server as well as on the control-replicas. See https://github.com/github/gh-ost/pull/347
    • support for --master-user and --master-password, for when master and replicas have different credentials.
    • Add possibility to set server-id as command line flag - contribution by @cypiszzz, see https://github.com/github/gh-ost/pull/349
    • --check-flag is a flag with which one can check if other flags are supported. Used for scripting and testing in lieu of evolving and changing versions. See https://github.com/github/gh-ost/pull/351

    downloads

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20170109081400.tar.gz(2.14 MB)
    gh-ost-binary-osx-20170109081400.tar.gz(2.14 MB)
  • v1.0.32(Dec 14, 2016)

  • v1.0.30(Dec 6, 2016)

    Many additions since 1.0.28. The ultimate breakdown is https://github.com/github/gh-ost/compare/v1.0.28...v1.0.30

    Notable:

    • Fixed cut-over stall: when cut-over happened to start executing even as lag or otherwise a throttling condition also emerged, a race condition would make the cut-over stall until throttling reason went away. The fix is that cut-over does not wait, and at worst, rolls back.
    • fix to cut-over retries -- it didn't throttle properly in between and did not cleanup properly in between.
    • cut-over timeout on "unknown" stalls. An example to "unknown" was the throttling problem described above. I don't know what other unknowns there are, but cut-over is now generically resilient to them.
    • support for --skip-foreign-key-checks - can save inspection time, when one is certain no FKs actually exist
    • Made column-rename detection heuristic more robust (it could wrongly infer a rename by some unfortunate comment or column-name)
    • Reading and reporting replication lag before waiting on initial replication event -- this adds visibility in seemingly-hanging operations where the replica is lagging in the first place
    • changelog table uses same engine as original table
    • on startup, verifying replication works all all the way up to the master.
    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20161206132017.tar.gz(2.13 MB)
    gh-ost-binary-osx-20161206132017.tar.gz(2.13 MB)
  • v1.0.28(Nov 1, 2016)

    Many additions since 1.0.21. The ultimate breakdown is https://github.com/github/gh-ost/compare/v1.0.21...v1.0.28

    Notable:

    • support for DATETIME subsecond resolution
    • user-commanded throttling implies complete silence, not even heartbeat is written
    • fixed log_slave_updates check on --test-on-replica and --migrate-on-replica
    • cleanly closing streamer connection (connection that imposes as a replica) upon completion
    • on --exact-rowcount, --concurrent-rowcount now defaults true
    • added validation to confirm table structure on master & replica is identical (yes, the counter case happened)
    • correct (though suboptimal) behavior on enum being part of the migration key
    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20161101125005.tar.gz(2.51 MB)
    gh-ost-binary-osx-20161101125005.tar.gz(2.50 MB)
  • v1.0.21(Oct 13, 2016)

    Many additions since 1.0.20. The ultimate breakdown is https://github.com/github/gh-ost/compare/v1.0.20...v1.0.21

    Notable:

    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20161013142859.tar.gz(2.50 MB)
    gh-ost-binary-osx-20161013142859.tar.gz(2.49 MB)
  • v1.0.20(Sep 13, 2016)

  • v1.0.18-experimental(Sep 8, 2016)

  • v1.0.17(Sep 5, 2016)

    Notable:

    • Support hooks: external processes invoked by gh-ost at specific points in execution
    • Sub-second lag resolution: if you require your replicas to lag within subsecond threshold, this is now supported
    • RDS: this release should work with RDS, see discussion -- we will formalize documentation once behavior is tested more
    • Tungsten: supply --tungsten --assume-master-host=the.master.com
    • master-master improved support: explictly specify the master via --allow-master-master --assume-master-host=the.master.com. See cheatsheet
    • Improved status report (ETA shows estimate even while throttling)
    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-v1.0.17-20160905104058.tar.gz(2.45 MB)
    gh-ost-binary-osx-v1.0.17-20160905104058.tar.gz(2.44 MB)
  • v1.0.14(Aug 26, 2016)

    Notable:

    • --concurrent-rowcount (https://github.com/github/gh-ost/issues/194): with --exact-rowcount, issue the count(*) concurrently even as row copy takes place (row count gets updated throughout the migration)
    • Using STRICT_ALL_TABLES for all operations, as safety net ()
    • Fixed support for 5.5 (https://github.com/github/gh-ost/issues/184)
    • Relaxed config file parser (https://github.com/github/gh-ost/issues/198)
    Source code(tar.gz)
    Source code(zip)
    gh-ost-binary-linux-20160826083132.tar.gz(2.37 MB)
    gh-ost-binary-osx-20160826083132.tar.gz(2.37 MB)
  • v1.0.13(Aug 24, 2016)

  • v1.0.11(Aug 22, 2016)

  • v1.0.10(Aug 18, 2016)

  • v1.0.9(Aug 15, 2016)

Owner
GitHub
How people build software.
GitHub
Database migrations. CLI and Golang library.

migrate Database migrations written in Go. Use as CLI or import as library. Migrate reads migrations from sources and applies them in correct order to

null 9k Jun 29, 2022
Database migrations written in Go

migrate Database migrations written in Go. Use as CLI or import as library. Migrate reads migrations from sources and applies them in correct order to

null 0 Oct 3, 2021
mysql to mysql 轻量级多线程的库表数据同步

goMysqlSync golang mysql to mysql 轻量级多线程库表级数据同步 测试运行 设置当前binlog位置并且开始运行 go run main.go -position mysql-bin.000001 1 1619431429 查询当前binlog位置,参数n为秒数,查询结

null 13 Jun 14, 2022
Manage Schema for KubeDB managed Databases

schema-manager Manage Schema for KubeDB managed Databases Installation To install KubeDB, please follow the guide here. Using KubeDB Want to learn how

Kubernetes Database 8 Feb 19, 2022
Go library that stores data in Redis with SQL-like schema

Go library that stores data in Redis with SQL-like schema. The goal of this library is we can store data in Redis with table form.

kaharman 2 Mar 14, 2022
Web-based, zero-config, dependency-free database schema change and version control tool for teams

Live Demo • Install • Help • Development • Design Doc Bytebase is a web-based, zero-config, dependency-free database schema change and version control

Bytebase 3k Jun 23, 2022
a powerful mysql toolset with Go

go-mysql A pure go library to handle MySQL network protocol and replication. Call for Committer/Maintainer Sorry that I have no enough time to maintai

siddontang 3.7k Jun 24, 2022
Sync MySQL data into elasticsearch

go-mysql-elasticsearch is a service syncing your MySQL data into Elasticsearch automatically. It uses mysqldump to fetch the origin data at first, the

siddontang 3.8k Jun 25, 2022
A high-performance MySQL proxy

kingshard 中文主页 Overview kingshard is a high-performance proxy for MySQL powered by Go. Just like other mysql proxies, you can use it to split the read

Fei Chen 6k Jun 28, 2022
Golang MySql binary log replication listener

Go MySql binary log replication listener Pure Go Implementation of MySQL replication protocol. This allow you to receive event like insert, update, de

Pavel <Ven> Gulbin 185 Apr 13, 2022
MySQL replication topology management and HA

orchestrator [Documentation] orchestrator is a MySQL high availability and replication management tool, runs as a service and provides command line ac

null 4.6k Jun 28, 2022
Vitess is a database clustering system for horizontal scaling of MySQL.

Vitess Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vite

Vitess 14.3k Jun 26, 2022
db-recovery is a tool for recovering MySQL data.

db-recovery is a tool for recovering MySQL data. It is used in scenarios where the database has no backup or binlog. It can parse data files and redo/undo logs to recover data.

null 22 Jun 13, 2022
一个使 mysql,pgsql 数据库表自动生成 go struct 的工具

db2go 一个使 mysql、pgsql 数据库表自动生成 go struct 的工具 快速使用 将项目放入到GOPATH/src目录下

易水韩 18 Jun 1, 2022
🐳 A most popular sql audit platform for mysql

?? A most popular sql audit platform for mysql

Henry Yee 6.6k Jul 1, 2022
Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).

?? Dumpling Dumpling is a tool and a Go library for creating SQL dump from a MySQL-compatible database. It is intended to replace mysqldump and mydump

PingCAP 260 Jun 10, 2022
Vitess is a database clustering system for horizontal scaling of MySQL.

Vitess Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding. By encapsulating shard-routing logic, Vite

Vitess 14.3k Jun 29, 2022
Gaea is a mysql proxy, it's developed by xiaomi b2c-dev team.

简介 Gaea是小米中国区电商研发部研发的基于mysql协议的数据库中间件,目前在小米商城大陆和海外得到广泛使用,包括订单、社区、活动等多个业务。Gaea支持分库分表、sql路由、读写分离等基本特性,更多详细功能可以参照下面的功能列表。其中分库分表方案兼容了mycat和kingshard两个项目的路

Xiaomi 2.3k Jun 26, 2022