A Go package to help write migrations with go-pg/pg.

Overview

go-pg-migrations

Version GoDoc Build Status Coverage Status Go Report Card

A Go package to help write migrations with go-pg/pg.

Usage

Installation

Because go-pg now has Go modules support, go-pg-migrations also has modules support; it currently depends on v10 of go-pg. To install it, use the following command in a project with a go.mod:

$ go get github.com/robinjoseph08/go-pg-migrations/v3

If you are not yet using Go modules, you can still use v1 of this package.

Running

To see how this package is intended to be used, you can look at the example directory. All you need to do is have a main package (e.g. example); call migrations.Run with the directory you want the migration files to be saved in (which will be the same directory of the main package, e.g. example), an instance of *pg.DB, and os.Args; and log any potential errors that could be returned.

Once this has been set up, then you can use the create, migrate, rollback, help commands like so:

$ go run example/*.go create create_users_table
Creating example/20180812001528_create_users_table.go...

$ go run example/*.go migrate
Running batch 1 with 1 migration(s)...
Finished running "20180812001528_create_users_table"

$ go run example/*.go rollback
Rolling back batch 1 with 1 migration(s)...
Finished rolling back "20180812001528_create_users_table"

$ go run example/*.go help
Usage:
  go run example/*.go [command]

Commands:
  create   - create a new migration in example with the provided name
  migrate  - run any migrations that haven't been run yet
  rollback - roll back the previous run batch of migrations
  help     - print this help text

Examples:
  go run example/*.go create create_users_table
  go run example/*.go migrate
  go run example/*.go rollback
  go run example/*.go help

While this works when you have the Go toolchain installed, there might be a scenario where you have to run migrations and you don't have the toolchain available (e.g. in a scratch or alpine Docker image deployed to production). In that case, you should compile another binary (in addition to your actual application) and copy it into the final image. This will include all of your migrations and allow you to run it by overriding the command when running the Docker container.

This would look something like this:

# Dockerfile
FROM golang:1.13.3 as build

WORKDIR /app

COPY go.mod go.mod
COPY go.sum go.sum
RUN go mod download

COPY . .

RUN CGO_ENABLED=0 GOOS=linux go build -installsuffix cgo -ldflags '-w -s' -o ./bin/serve ./cmd/serve
RUN CGO_ENABLED=0 GOOS=linux go build -installsuffix cgo -ldflags '-w -s' -o ./bin/migrations ./cmd/migrations

FROM alpine:3.8

RUN apk --no-cache add ca-certificates
COPY --from=build /app/bin /bin

CMD ["serve"]
$ docker build -t service:latest .
$ docker run --rm service:latest migrations migrate

Why?

While go-pg has its own migrations package, it leaves a bit to be desired. Some additional features that this package supports:

  • Complete migration diffing to determine which migrations still need to be run. Since go-pg/migrations checks the current version of migrations and runs any migrations after that, some migrations can be missed, especially when several people are working on the same project.
  • Timestamp-based prefixes to prevent two people creating a migration with the same version on two separate branches. If the current version is 3, and more than one person branches off and creates a new migration, all of them will be version 4.
  • The ability to run migrations in a transaction on a case-by-case basis. Most of the time, running migrations within a transaction is desirable, so that if it errs out within the "up" function, the whole migration is reverted. But since some long-running migrations might have a statement with a relatively exclusive lock, you might opt out of running that specific migration within a transaction.
  • A migration locking mechanism. This is to avoid two people (or an automated deployment system) attempting to run migrations at the same time against the same database, which could lead to undesired behavior.
  • An expected workflow of how this package should be used within a project. While go-pg/migrations has some recommendations and examples, this package takes a more opinionated approach which makes it so you don't have to think about it as much, and there's less code for you to write and maintain.
  • Batch-level rollbacks. When there are multiple migration files run during the same migration invocation, they are all grouped together into a "batch". During rollbacks, each batch gets rolled back together. This tends to be more desireable since this usually means the application is reverting back to a previous release, so the database should be in the state expected for that release.

Many of these features and expected behaviors come from using Knex.js migrations in production for many years. This project is heavily inspired by Knex to provide a robust and safe migration experience.

go-pg is a great and performant project, and hopefully, this makes it a little better.

Development

To develop on this project, you'll need to have Postgres running because the tests depend on it.

If you have it running on your machine because it was installed through your package manager (like brew or apt-get), you just need to run the following to get it set up correctly:

make setup

If you don't have it on your laptop, you can run the following to start it within Docker:

make postgres

That should start the container and keep it running while you develop. Once you're done, you can ^C out of it and it will stop the container.

To run the tests, you should run:

make test

To run the linter, you should run:

make lint
Comments
  • Need help with usage

    Need help with usage

    I try to use your lib, took code from example and it doesn't work.

    When I do go run cmd/migrate/*.go create create_users_table it works fine. After inserting code into Exec I do migrate and in my db 2 tables for migrations were appeared

    go run cmd/migrate/*.go create create_users_table 
    Creating database/migrations/20200128144423_create_users_table.go...
    
    go run cmd/migrate/*.go migrate                  
    Migrations already up to date
    
    go run cmd/migrate/*.go rollback                 
    No migrations have been run yet
    
    

    But main table isn't present nowhere

    opened by dink10 6
  • Getting an Error is_locked violates not-null constraint

    Getting an Error is_locked violates not-null constraint

    ERROR #23502 null value in column "is_locked" violates not-null constraint

    This occurs when the migration is running any command even the help command as it is trying to insert a migration lock and failing with that error. The SQL being run is the following.

    INSERT INTO migration_lock ("id", "is_locked") VALUES ('lock', false) RETURNING "is_locked"

    Edit the actual query is: INSERT INTO migration_lock ("id", "is_locked") VALUES ('lock', DEFAULT) RETURNING "is_locked"

    opened by KaffineAddict 6
  • Migrate and Rollback takes in directory but doesn't use it

    Migrate and Rollback takes in directory but doesn't use it

    See: https://github.com/robinjoseph08/go-pg-migrations/blob/master/migrate.go#L25 https://github.com/robinjoseph08/go-pg-migrations/blob/master/rollback.go#L10

    directory is passed in, but not used at all, hence migration and rollbacks don't work when migration is run from a separate directory?

    opened by ernsheong 4
  • refactor(migrate): acquire lock in a single statement

    refactor(migrate): acquire lock in a single statement

    This can be accomplished in a single statement. UPDATE lock SET is_locked = true WHERE id = 'lock' and is_locked = false. You can tell if the unlock was successful by checking if any rows were updated.

    opened by mgartner 4
  • CreateTable assignment mismatch

    CreateTable assignment mismatch

    go-getting this repo produces a compiler error:

    ../../robinjoseph08/go-pg-migrations/setup.go:57:9: assignment mismatch: 2 variables but 1 values
    

    go-pg/pg#957 changed the signature of CreateTable to omit the Result and only return an error, so createTable in setup.go should be changed to just return orm.CreateTable(...).

    opened by jakebailey 3
  • Do not require postgres connection on create

    Do not require postgres connection on create

    Right now creating migration files requires postgresql connection even if it's not used during the process

    There's also no possibility to make custom wrapper, since create function is private and can't be imported.

    It would help a lot if any one of these two were possible

    opened by fizyk 2
  • Create.go doesn't reference new /v2 lib

    Create.go doesn't reference new /v2 lib

    The template on line 16 still says migrations "github.com/robinjoseph08/go-pg-migrations" when it should read migrations "github.com/robinjoseph08/go-pg-migrations/v2"

    opened by bennichols 2
  • create uses wrong package name

    create uses wrong package name

    Hi there, i would like to have my migrations in a separate folder like so:

    err := gpgm.Run(db, "migrations", os.Args)
    

    Using the "create" command creates the migration in the right place but the package name is wrong. It should be the name of the folder but it is "main" instead.

    opened by sgoldenb 1
  • feat(go-pg): update from go-pg v9 to v10

    feat(go-pg): update from go-pg v9 to v10

    what

    fixes #22

    • update from go-pg v9 to v10
    • update from go-pg-migrations v2 to v3
    • fix changes/deprecations with the new version
    • make testing more verbose so that we can more easily catch deprecations
    • ensure that errors from releaseLock get surfaced
    opened by robinjoseph08 0
  • fix(db): remove need of db connection for create and help

    fix(db): remove need of db connection for create and help

    what

    fixes #21

    • update Makefile to make it easier to run local
    • update README.md with these changes
    • only ensure migration tables for migrate and rollback
    • update the tests to not have a valid connection for create and help
    opened by robinjoseph08 0
  • feat(modules): Add go.mod for Go modules

    feat(modules): Add go.mod for Go modules

    What

    • use go v1.13.3
    • switch from dep to go modules
    • switch from gometalinter to golangc-lint
    • ignore bin directory since thats where all tools are now being installed into
    • build against multiple versions of go
    • add version badge to readme
    • change module path to github.com/robinjoseph08/go-pg-migrations/v2 to allow those who are not using go modules yet to keep using v1
    • pin to version v9 of go-pg, update all paths to use v9, and update some code for v9
    • update example dockerfile in readme to use go modules
    • add a tools.go
    opened by robinjoseph08 0
  • Move migrations tables to seperate schema

    Move migrations tables to seperate schema

    Is there any way to move the tables crated by this package migrations & migration_lock to a separate schema ? (so that it is not in the public Schema & does not conflict with existing tables)

    opened by vijayrajah 0
  • adds a 'status' command that checks for pending migrations.

    adds a 'status' command that checks for pending migrations.

    This adds a new command called 'status', that you run like this: go run example/*.go status. If there are no pending migrations, it will print a message indicating that migrations are up to date, and return nil. If there are pending migrations, it will return an error with the message indicating how many are pending. If things are set up as described in the example dir, this will also cause the program to exit with status 1.

    This is useful in deployment, for example, in kubernetes, new pods can use this as a check to determine if they are ready to start receiving traffic.

    opened by blunckr 1
  • feat(status): Add status command

    feat(status): Add status command

    Add status command for viewing status of migrations:

    $ go run *.go status
    +---------+------------------------------------------------+-------+
    | Applied | Migration                                      | Batch |
    +---------+------------------------------------------------+-------+
    |    √    | 2021_02_26_151503_dump                         |     1 |
    |    √    | 2021_02_26_151504_create_a_dump_table_for_test |     2 |
    |         | 2021_02_26_151502_create_2nd_dump_table        |       |
    |         | 2021_02_26_151505_create_3rd_dump_table        |       |
    +---------+------------------------------------------------+-------+
    
    opened by psucodervn 0
Owner
Robin Joseph
Robin Joseph
Database migrations. CLI and Golang library.

Database migrations written in Go. Use as CLI or import as library.

null 27 May 30, 2021
Opinionated tool for database structure management and migrations

trek Requirements At least version 13 of postgres is needed. Installation go install . Setup Create config.yaml: model_name: <model_name> db_name: <db

Stack11 3 Jan 7, 2022
Tool to handle versioned migrations with gorm

GORM Migrations About Gorm Migrations Gorm Migrations is a tool designed for go-gorm. Gorm Migration takes the pain out of development of migrations v

null 5 Mar 14, 2022
GitHub's Online Schema Migrations for MySQL

gh-ost GitHub's online schema migration for MySQL gh-ost is a triggerless online schema migration solution for MySQL. It is testable and provides paus

Leon 0 Apr 3, 2020
Database migrations. CLI and Golang library.

migrate Database migrations written in Go. Use as CLI or import as library. Migrate reads migrations from sources and applies them in correct order to

null 9.7k Sep 19, 2022
Database migrations. CLI and Golang library.

Database migrations written in Go. Use as CLI or import as library.

null 27 May 30, 2021
GitHub's Online Schema Migrations for MySQL

gh-ost GitHub's online schema migration for MySQL gh-ost is a triggerless online schema migration solution for MySQL. It is testable and provides paus

GitHub 10.4k Sep 26, 2022
Database migrations. CLI and Golang library.

migrate Database migrations written in Go. Use as CLI or import as library. Migrate reads migrations from sources and applies them in correct order to

null 9.8k Sep 25, 2022
Database migrations written in Go

migrate Database migrations written in Go. Use as CLI or import as library. Migrate reads migrations from sources and applies them in correct order to

null 0 Oct 3, 2021
Opinionated tool for database structure management and migrations

trek Requirements At least version 13 of postgres is needed. Installation go install . Setup Create config.yaml: model_name: <model_name> db_name: <db

Stack11 3 Jan 7, 2022
Simple MySQL migrations in Go

gomigrator This package contains a very small MySQL migration solution for go written in 1 file. The aim of this package is to be very small but highl

null 0 Nov 29, 2021
Tool to handle versioned migrations with gorm

GORM Migrations About Gorm Migrations Gorm Migrations is a tool designed for go-gorm. Gorm Migration takes the pain out of development of migrations v

null 5 Mar 14, 2022
GitHub's Online Schema Migrations for MySQL

gh-ost GitHub's online schema migration for MySQL gh-ost is a triggerless online schema migration solution for MySQL. It is testable and provides paus

Leon 0 Apr 3, 2020
Write controller-runtime based k8s controllers that read/write to git, not k8s

Git Backed Controller The basic idea is to write a k8s controller that runs against git and not k8s apiserver. So the controller is reading and writin

Darren Shepherd 50 Dec 10, 2021
This is a template project to help beginners learn, or to help developers develop some interesting small projects

This is a template project to help beginners learn, or to help developers develop some interesting small projects

null 6 Aug 24, 2022
Elastos.ELA.Rosetta.API - How to write a Rosetta server and use either the Client package or Fetcher package to communicate

Examples This folder demonstrates how to write a Rosetta server and how to use e

Elastos Foundation 3 Jan 17, 2022
Package goth provides a simple, clean, and idiomatic way to write authentication packages for Go web applications.

Goth: Multi-Provider Authentication for Go Package goth provides a simple, clean, and idiomatic way to write authentication packages for Go web applic

Mark Bates 3.8k Sep 20, 2022
Package ini provides INI file read and write functionality in Go.

INI Package ini provides INI file read and write functionality in Go. Features Load from multiple data sources(file, []byte, io.Reader and io.ReadClos

INI 3.1k Sep 22, 2022
peanut is a Go package to write tagged data structs to disk in a variety of formats.

peanut peanut is a Go package to write tagged data structs to disk in a variety of formats. Its primary purpose is to provide a single consistent inte

Jim Smart 3 Apr 21, 2021