Database migrations. CLI and Golang library.

Overview

CircleCI - Build Status GoDoc Coverage Status packagecloud.io Docker Pulls Supported Go Versions GitHub Release Go Report Card

migrate

Database migrations written in Go. Use as CLI or import as library.

  • Migrate reads migrations from sources and applies them in correct order to a database.
  • Drivers are "dumb", migrate glues everything together and makes sure the logic is bulletproof. (Keeps the drivers lightweight, too.)
  • Database drivers don't assume things or try to correct user input. When in doubt, fail.

Forked from mattes/migrate

Databases

Database drivers run migrations. Add a new database?

Database URLs

Database connection strings are specified via URLs. The URL format is driver dependent but generally has the form: dbdriver://username:[email protected]:port/dbname?param1=true&param2=false

Any reserved URL characters need to be escaped. Note, the % character also needs to be escaped

Explicitly, the following characters need to be escaped: !, #, $, %, &, ', (, ), *, +, ,, /, :, ;, =, ?, @, [, ]

It's easiest to always run the URL parts of your DB connection URL (e.g. username, password, etc) through an URL encoder. See the example Python snippets below:

$ python3 -c 'import urllib.parse; print(urllib.parse.quote(input("String to encode: "), ""))'
String to encode: FAKEpassword!#$%&'()*+,/:;=?@[]
FAKEpassword%21%23%24%25%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D
$ python2 -c 'import urllib; print urllib.quote(raw_input("String to encode: "), "")'
String to encode: FAKEpassword!#$%&'()*+,/:;=?@[]
FAKEpassword%21%23%24%25%26%27%28%29%2A%2B%2C%2F%3A%3B%3D%3F%40%5B%5D
$

Migration Sources

Source drivers read migrations from local or remote sources. Add a new source?

CLI usage

  • Simple wrapper around this library.
  • Handles ctrl+c (SIGINT) gracefully.
  • No config search paths, no config files, no magic ENV var injections.

CLI Documentation

Basic usage

$ migrate -source file://path/to/migrations -database postgres://localhost:5432/database up 2

Docker usage

$ docker run -v {{ migration dir }}:/migrations --network host migrate/migrate
    -path=/migrations/ -database postgres://localhost:5432/database up 2

Use in your Go project

  • API is stable and frozen for this release (v3 & v4).
  • Uses Go modules to manage dependencies.
  • To help prevent database corruptions, it supports graceful stops via GracefulStop chan bool.
  • Bring your own logger.
  • Uses io.Reader streams internally for low memory overhead.
  • Thread-safe and no goroutine leaks.

Go Documentation

import (
    "github.com/golang-migrate/migrate/v4"
    _ "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/github"
)

func main() {
    m, err := migrate.New(
        "github://mattes:[email protected]/migrate_test",
        "postgres://localhost:5432/database?sslmode=enable")
    m.Steps(2)
}

Want to use an existing database client?

import (
    "database/sql"
    _ "github.com/lib/pq"
    "github.com/golang-migrate/migrate/v4"
    "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/file"
)

func main() {
    db, err := sql.Open("postgres", "postgres://localhost:5432/database?sslmode=enable")
    driver, err := postgres.WithInstance(db, &postgres.Config{})
    m, err := migrate.NewWithDatabaseInstance(
        "file:///migrations",
        "postgres", driver)
    m.Steps(2)
}

Getting started

Go to getting started

Tutorials

(more tutorials to come)

Migration files

Each migration has an up and down migration. Why?

1481574547_create_users_table.up.sql
1481574547_create_users_table.down.sql

Best practices: How to write migrations.

Versions

Version Supported? Import Notes
master import "github.com/golang-migrate/migrate/v4" New features and bug fixes arrive here first
v4 import "github.com/golang-migrate/migrate/v4" Used for stable releases
v3 import "github.com/golang-migrate/migrate" (with package manager) or import "gopkg.in/golang-migrate/migrate.v3" (not recommended) DO NOT USE - No longer supported

Development and Contributing

Yes, please! Makefile is your friend, read the development guide.

Also have a look at the FAQ.


Looking for alternatives? https://awesome-go.com/#database.

Comments
  • Getting

    Getting "error: file does not exist"

    We have run the migration successfully, and now i need to run it again in case there are any new migrations.

    This is he command we are using:

    migrate -database 'mysql://xxx:[email protected]/zzzz' -source file://go/src/github.com/me/myproj.org/db/schema up

    I am in the root directory of the user (i.e. /home/myuser), and underneath is the structure: go/src/github.com/me/myproj.org/db/schema which contains files like:

    201802221525_init_schema.down.sql 201805211743_us_states.down.sql 201805291420_countries.down.sql 201806051200_currencies.down.sql 201802221525_init_schema.up.sql 201805211743_us_states.up.sql 201805291420_countries.up.sql 201806051200_currencies.up.sql etc.

    We see that with mattes migrate, there were many similar issues with no obvious resolution.

    in the db in schema_migrations table, I see it says version 201807191200, although i am not sure this has any bearing on the issue.

    If I change the file path to something which doesn't exist, e.g.

    migrate -database 'mysql://xxx:[email protected]/zzzz' -source file://go/src/github.com/me/myproj.org/notexist up

    I get a different error message:

    error: open /home/muyuser/go/src/github.com/me/myproj.org/notexist : no such file or directory

    This is how we installed migrate:

    wget https://github.com/golang-migrate/migrate/releases/download/v3.2.0/migrate.linux-amd64.tar.gz gunzip migrate.linux-amd64.tar.gz tar xvf migrate.linux-amd64.tar mv migrate.linux-amd64 /usr/bin/migrate

    opened by nutmix 27
  • Dirty database version 1. Fix and force version

    Dirty database version 1. Fix and force version

    the error appears as below when migrate timeout:

    Dirty database version 1. Fix and force version
    

    Steps to Reproduce Steps to reproduce the behavior:

    1. do migrate timeout
    2. migrate again

    Expected Behavior migrate again could automatically solve this problem and do migrate normally.

    wontfix 
    opened by duyanghao 22
  • Postgres - Add schema name support in x-migrations-table value (#95)

    Postgres - Add schema name support in x-migrations-table value (#95)

    @dhui what do you think about this commit which add x-schema-name URL Query support as suggest by this comment?

    Is anything missing to approve this Pull Request? Some documention, unit test or other?

    Best regards,
    Stéphane


    Edit:

    The second version of this patch add schema name support in x-migrations-table value.

    Now, you can use:

    migrate ... -database ...?x-migrations-table=gomigrate.schema_migrations
    

    Edit: alternative implementation: https://github.com/golang-migrate/migrate/pull/533

    opened by stephane-klein 20
  • feat: embed, io/fs support

    feat: embed, io/fs support

    Is your feature request related to a problem? Please describe.

    The next Go 1.16 will add official file system interface and file embedding.

    https://github.com/golang/go/issues/41190 https://github.com/golang/go/tree/master/src/io/fs

    https://github.com/golang/go/issues/41191 https://github.com/golang/go/tree/master/src/embed

    It would be great if golang-migrate supports these as source.Driver.

    It replaces statik, go-bindata, and pkger. Of course, this does not mean that they cannot be used.

    Describe the solution you'd like

    Describe alternatives you've considered

    Additional context

    We will also need to use the build tag properly.

    opened by johejo 17
  • fix example of install by go toolchain at README

    fix example of install by go toolchain at README

    Currently, result of install by go toolchain as follows

    > go get -tags 'postgres' -u github.com/golang-migrate/migrate/cmd/migrate
    go: finding github.com/golang-migrate/migrate/cmd/migrate latest
    go: finding github.com/golang-migrate/migrate/cmd latest
    go get github.com/golang-migrate/migrate/cmd/migrate: no matching versions for query "latest"
    

    So, I fixed Installation->With Go toolchain.

    opened by stobita 17
  • postgres: Move lock out of ensureVersionTable, for consistency with other SQL operations

    postgres: Move lock out of ensureVersionTable, for consistency with other SQL operations

    Fixes #164 .

    ensureVersionTable seems to be the only SQL operation in the postgres driver which handles locking of database itself, meaning it seems to break the convention of having the caller responsible for locking and not the callee.

    This removes the locking logic from ensureVersionTable and makes sure to properly lock the database before calling ensureVersionTable from the WithInstance method.

    opened by lukaspj 15
  • adding /v3/ to urls is a breaking change when building.

    adding /v3/ to urls is a breaking change when building.

    When building the project there is no such place as

    ```https://github.com/golang-migrate/migrate/v3/``

    so the build cannot import its dependencies and the build fails.

    Steps to Reproduce

    Build the project as below

    go build -tags 'postgres' -o ~/go/bin/migrate github.com/golang-migrate/migrate/cli

    Expected Behavior I expect the binary to be built and deployed to $GOPATH/bin/migrate

    Migrate Version v3.5.2

    opened by hayres 15
  • Postgres - add x-migrations-table-quoted url query option (#95)

    Postgres - add x-migrations-table-quoted url query option (#95)

    Following this comment, this is a new Pull Request.

    By default, gomigrate quote migrations table name, if x-migrations-table-quoted is enabled, then you must to quote migrations table name manually, for instance "gomigrate"."schema_migrations".

    I have used this playground to test this branch.

    opened by stephane-klein 14
  • Packr Box as migration source.

    Packr Box as migration source.

    Is your feature request related to a problem? Please describe. go-bindata has been left unmaintained, unwillingly or otherwise. See https://github.com/jteeuwen/go-bindata/issues/5

    This presents a problem as we're using go-bindata to embed our migration files into our Go binaries. The next best alternative is https://github.com/gobuffalo/packr

    Describe the solution you'd like Migration source support for packr boxes.

    Describe alternatives you've considered N/A

    Additional context N/A

    opened by chakrit 13
  • mysql database registered twice

    mysql database registered twice

    Step to reproduce Execute the sample code of this page: https://github.com/golang-migrate/migrate/tree/master/database/mysql

    Output It gives the following error:

    panic: sql: Register called twice for driver mysql
    
    goroutine 1 [running]:
    database/sql.Register(0x127dc81, 0x5, 0x12a7a20, 0x13d2508)
    	/usr/local/Cellar/go/1.10.3/libexec/src/database/sql/sql.go:50 +0x1a4
    github.com/golang-migrate/migrate/vendor/github.com/go-sql-driver/mysql.init.0()
    	/Users/martin/go/src/github.com/golang-migrate/migrate/vendor/github.com/go-sql-driver/mysql/driver.go:168 +0x5c
    exit status 2
    

    Root cause The migrate/database/mysql package uses github.com/go-sql-driver/mysql which already registers the "mysql" driver. So when migrate/database/mysql tries to register it as well, it creates this error.

    opened by mdestagnol 13
  • goreleaser configuration and automatic release workflow

    goreleaser configuration and automatic release workflow

    close #74

    Required settings

    GitHub Actions secrets

    • DOCKERHUB_USERNAME
    • DOCKERHUB_PASSWORD
    • PACKAGECLOUD_TOKEN

    How to release

    I imagined as much as possible what the existing release workflow would look like, and followed the steps below. If you don't like it, please send comments. We can also automatically generate a change log for the release notes.

    1. push a new tag (manual)
    2. publish draft release (automatic)
    3. edit draft release and publish release (manual)
    opened by johejo 12
  • clickhouse: Quote db name in ensureVersionTable

    clickhouse: Quote db name in ensureVersionTable

    Hi all!

    Today I wanted to use migrate on a clickhouse cloud cluster where we were setting up databases with dashes in their name, e.g. database-name. This broke migrate since the SHOW TABLES query for clickhouse did not quote the database name, hence this PR.

    opened by luca-nardelli 1
  • Option to migrate create without down.sql

    Option to migrate create without down.sql

    Is your feature request related to a problem? Please describe. I'm always frustrated when migrate create creates an up and down sql even though I decided to not use down migrations at all.

    Describe the solution you'd like An option that allows us to disable the generation of the down sql file.

    Describe alternatives you've considered Delete the generated down sql file every time after I ran the create command.

    Additional context

    opened by weeco 0
  • Upgraded Neo4j driver and added support for Neo4j v5

    Upgraded Neo4j driver and added support for Neo4j v5

    The Neo4j driver has been updated to v4 and the migration queries have been updated to add compatibility with Neo4j version 5.x that changed cypher syntax

    opened by fabiomarini 0
  • Add non-cgo sqlite support to released binaries

    Add non-cgo sqlite support to released binaries

    In https://github.com/golang-migrate/migrate/commit/15611d4559ac626305655c5b2fa18b2cfe995f60 the sqlite3 driver was removed from builds because of on-going cgo issues.

    But the sqlite non-cgo version still works and can be built and released. This PR enables it.

    opened by tmm1 1
Releases(v4.15.2)
Owner
null
GitHub's Online Schema Migrations for MySQL

gh-ost GitHub's online schema migration for MySQL gh-ost is a triggerless online schema migration solution for MySQL. It is testable and provides paus

GitHub 10.6k Dec 2, 2022
Simple MySQL migrations in Go

gomigrator This package contains a very small MySQL migration solution for go written in 1 file. The aim of this package is to be very small but highl

null 0 Nov 29, 2021
[mirror] the database client and tools for the Go vulnerability database

The Go Vulnerability Database golang.org/x/vulndb This repository is a prototype of the Go Vulnerability Database. Read the Draft Design. Neither the

Go 201 Nov 25, 2022
Database - Example project of database realization using drivers and models

database Golang based database realization Description Example project of databa

Denis 1 Feb 10, 2022
sqlx is a library which provides a set of extensions on go's standard database/sql library

sqlx is a library which provides a set of extensions on go's standard database/sql library. The sqlx versions of sql.DB, sql.TX, sql.Stmt, et al. all leave the underlying interfaces untouched, so that their interfaces are a superset on the standard ones. This makes it relatively painless to integrate existing codebases using database/sql with sqlx.

Jason Moiron 12.9k Dec 6, 2022
Interactive terminal user interface and CLI for database connections. MySQL, PostgreSQL. More to come.

?? dbui dbui is the terminal user interface and CLI for database connections. It provides features like, Connect to multiple data sources and instance

Kanan Rahimov 116 Dec 3, 2022
Library for scanning data from a database into Go structs and more

scany Overview Go favors simplicity, and it's pretty common to work with a database via driver directly without any ORM. It provides great control and

Georgy Savva 796 Dec 7, 2022
A go Library for scan database/sql rows to struct、slice、other types. And it support multiple databases connection management

ploto A go Library for scan database/sql rows to struct、slice、other types. And it support multiple databases connection management It's not an ORM. wo

solar 3 Nov 3, 2022
Database Access Layer for Golang - Testable, Extendable and Crafted Into a Clean and Elegant API

REL Modern Database Access Layer for Golang. REL is golang orm-ish database layer for layered architecture. It's testable and comes with its own test

REL 605 Dec 6, 2022
A Go rest API project that is following solid and common principles and is connected to local MySQL database.

This is an intermediate-level go project that running with a project structure optimized RESTful API service in Go. API's of that project is designed based on solid and common principles and connected to the local MySQL database.

Kıvanç Aydoğmuş 21 Jun 6, 2022
An observability database aims to ingest, analyze and store Metrics, Tracing and Logging data.

BanyanDB BanyanDB, as an observability database, aims to ingest, analyze and store Metrics, Tracing and Logging data. It's designed to handle observab

The Apache Software Foundation 138 Dec 5, 2022
database to golang struct

中文文档 mysql database to golang struct conversion tools base on gorm(v1/v2),You can automatically generate golang sturct from mysql database. big Camel-

xxj 2.1k Dec 2, 2022
Constant Database native golang implementation

CDB golang implementation cdb is a fast, reliable, simple package for creating and reading constant databases see docs for more details Advantages Ite

Alexander Petrov 32 Jul 15, 2022
Golang restAPI crud project with mySql database.

Golang RestAPI using gorilla/mux Golang restAPI crud project with mySql database. Test Api with Thunder Client vs code beautiful Extension. and use Be

Md Abu. Raihan 6 Mar 26, 2022
A simple Golang-based application that queries a PostgreSQL database

Qwik-E-Mart Demo App A simple Golang-based application that queries a PostgreSQL database named qwikemart to read and return customer data stored in t

Joe Garcia 0 Nov 6, 2021
Implementasi database oracle kedalam golang

Go with Oracle database Implementasi database oracle kedalam golang How to using swagger Install generator swagger menggunakan perintah : go get -u gi

Muhammad Rais Adlani 0 Nov 20, 2021
Golang database driver for SQLite

go-sqlite Golang database driver for SQLite. Does not use cgo. This driver is based on pure-Go SQLite implementation (https://gitlab.com/cznic/sqlite)

glebarez 153 Dec 5, 2022
BigQuery database/sql golang driver

BigQuery SQL Driver This library is compatible with Go 1.17+ Please refer to CHA

Viant, Inc 6 Aug 31, 2022