wrench - Schema management tool for Cloud Spanner -

Overview

wrench

cloudspannerecosystem

wrench is a schema management tool for Cloud Spanner.

Please feel free to report issues and send pull requests, but note that this application is not officially supported as part of the Cloud Spanner product.

$ cat ./_examples/schema.sql
CREATE TABLE Singers (
  SingerID STRING(36) NOT NULL,
  FirstName STRING(1024),
) PRIMARY KEY(SingerID);

# create database with ./_examples/schema.sql
$ wrench create --directory ./_examples

# create migration file
$ wrench migrate create --directory ./_examples
_examples/migrations/000001.sql is created

# edit _examples/migrations/000001.sql
$ cat ./_examples/migrations/000001.sql
ALTER TABLE Singers ADD COLUMN LastName STRING(1024);

# execute migration
$ wrench migrate up --directory ./_examples

# load ddl from database to file ./_examples/schema.sql
$ wrench load --directory ./_examples

# finally, we have successfully migrated database!
$ cat ./_examples/schema.sql
CREATE TABLE SchemaMigrations (
  Version INT64 NOT NULL,
  Dirty BOOL NOT NULL,
) PRIMARY KEY(Version);

CREATE TABLE Singers (
  SingerID STRING(36) NOT NULL,
  FirstName STRING(1024),
  LastName STRING(1024),
) PRIMARY KEY(SingerID);

Installation

Get binary from release page. Or, you can use Docker container: mercari/wrench.

Usage

Prerequisite

export SPANNER_PROJECT_ID=your-project-id
export SPANNER_INSTANCE_ID=your-instance-id
export SPANNER_DATABASE_ID=your-database-id

You can also specify project id, instance id and database id by passing them as command arguments.

Create database

$ wrench create --directory ./_examples

This creates the database with ./_examples/schema.sql.

Drop database

$ wrench drop

This just drops the database.

Reset database

wrench reset --directory ./_examples

This drops the database and then re-creates with ./_examples/schema.sql. Equivalent to drop and then create.

Load schema from database to file

$ wrench load --directory ./_examples

This loads schema DDL from database and writes it to ./_examples/schema.sql.

Create migration file

$ wrench migrate create --directory ./_examples

This creates a next migration file like _examples/migrations/000001.sql. You will write your own migration DDL to this file.

Execute migrations

$ wrench migrate up --directory ./_examples

This executes migrations. This also creates SchemaMigrations table into your database to manage schema version if it does not exist.

Apply single DDL/DML

$ wrench apply --ddl ./_examples/ddl.sql

This applies single DDL or DML.

Use wrench [command] --help for more information about a command.

Contributions

Please read the contribution guidelines before submitting pull requests.

License

Copyright 2019 Mercari, Inc.

Licensed under the MIT License.

Comments
  • Add comment support #15

    Add comment support #15

    WHAT

    • Add supports comments in DDL; only manual addition
      • I know there's already issue to store the comments both in importing and exporting on #15, but it supports only manual addition of comments so that we can leave some comments and restored them on local if they are missed by other wrench operation, temporarily

    WHY

    • Supporting comments on DDL is helpful, if it is only manual addition temporarily
    opened by mookjp 7
  • Raise error message when migration scripts have common version number

    Raise error message when migration scripts have common version number

    WHAT

    This PR addresses the issue https://github.com/cloudspannerecosystem/wrench/issues/47.

    The bug was that if more migration scripts have the same version number then, instead of reporting this as an issue, wrench pretended as if everything was OK and simply ignored the later migration scripts.

    WHY

    This bug caused me several hours of bug hunting. I hope with this PR, others will find their bug more easily.

    Note: there was another minor issue reported by golangci-lint which I also fixed.

    opened by halmai 6
  • No binary release for v1.1.0 and embedded version is still 1.0.4

    No binary release for v1.1.0 and embedded version is still 1.0.4

    Hello.

    • There is no binary attached to release page. And also no 1.1.0 image is published to docker hub.
    • Version number in https://github.com/cloudspannerecosystem/wrench/blob/v1.1.0/BUILD.bazel#L1 is still 1.0.4.
    opened by darklore 4
  • Add support (or documentation if support already exists) for using wrench with the Cloud Spanner emulator

    Add support (or documentation if support already exists) for using wrench with the Cloud Spanner emulator

    WHAT

    Hi, is there support for using wrench with the local Cloud Spanner emulator? So far, I haven't been able to make it work. I've tried the following:

    Shell session 1

    $ gcloud beta emulators spanner start
    Executing: docker run -p 127.0.0.1:9010:9010 -p 127.0.0.1:9020:9020 gcr.io/cloud-spanner-emulator/emulator:0.7.3
    [cloud-spanner-emulator] 2020/06/12 21:59:57 gateway.go:135: Cloud Spanner emulator running.
    [cloud-spanner-emulator] 2020/06/12 21:59:57 gateway.go:136: REST server listening at 0.0.0.0:9020
    [cloud-spanner-emulator] 2020/06/12 21:59:57 gateway.go:137: gRPC server listening at 0.0.0.0:9010
    

    Shell session 2

    $  $(gcloud beta emulators spanner env-init)  # sets SPANNER_EMULATOR_HOST
    $ ./wrench_darwin_amd64 create --directory . --project my-project --instance my-instance --database my-database
    Error command: create, version: 1.0.0
    	Failed to execute the operation to Cloud Spanner, Either caller is missing IAM permission spanner.databases.create on resource projects/my-project/instances/my-instance, or the CreateDatabaseRequest.create_statement field is malformed and the database name could not be identified to verify Cloud IAM Conditions.
    

    It looks like wrench isn't picking up on the SPANNER_EMULATOR_HOST environment variable, and therefore is failing when hitting the actual Cloud Spanner API only to find that my dummy project / instance / database doesn't exist.

    I'm guessing wrench just uses the Go client library for Cloud Spanner – which according to the emulator docs should honor the SPANNER_EMULATOR_HOST environment variable. So I might be missing something that's needed to make this work.

    WHY

    We rely on the Cloud Spanner emulator for local development and are looking for a good tool to manage schema migrations. It would be great if we could start using wrench for both local and Cloud development.

    Thanks in advance.

    EDIT

    Perhaps https://github.com/googleapis/google-cloud-go/issues/1602 is related?

    enhancement 
    opened by ghost 3
  • Add a flag to set context timeout

    Add a flag to set context timeout

    WHAT

    SSIA

    WHY

    As you can see in the following code, gRPC client defines the default timeout as 30 seconds. https://github.com/googleapis/google-cloud-go/blob/spanner/v1.41.0/spanner/apiv1/spanner_client.go#L551-L555 It's too short in some cases.

    opened by iwata 2
  • Add context timeout option

    Add context timeout option

    WHAT

    Add option to set context timeout duration. It is almost the same as the PR here, but I thought it would be better to be able to specify by option. https://github.com/cloudspannerecosystem/wrench/pull/46

    WHY

    sometimes error would occur like this

    Failed to execute the operation to Cloud Spanner, context deadline exceeded
    

    As far as I can see this documents, the deadline of the context can be overwritten. https://pkg.go.dev/cloud.google.com/go/spanner/admin/database/apiv1#hdr-Use_of_Context https://pkg.go.dev/cloud.google.com/go#hdr-Timeouts_and_Cancellation

    opened by shiv3 2
  • Migration files with colliding numbers are not reported as error

    Migration files with colliding numbers are not reported as error

    Expected Behavior

    If there are more than one files starting with the same number in their names then they should be either processed individually or reported as an error. Silently pretending as if there were only one of them is pretty misleading.

    If there are two files with the same number in their names then either process all of them or report an error saying "File names must have different numbers in their names."

    Current Behavior

    Wrench executes only one of these files and ignores the rest.

    Steps to Reproduce

    1. Create two migration scripts and name them with the same number. For example, 20210831_123456_first.sql and 20210831_123456_second.sql
    2. Define a CREATE TABLE first(...) in the first one and CREATE TABLE second(...) in the second one.
    3. Do a migrate up
    4. See your database. Table first will be there, table second will not.
    bug 
    opened by halmai 2
  • Fix the missing

    Fix the missing "context" pacakge

    WHAT

    fix build error by fixing the missing "context" package.

    WHY

    • https://github.com/cloudspannerecosystem/wrench/pull/43 introduced a build error (the file is using context package but the package is not imported).
    • Out projects are using the tool, and it has the error when we don't specify a version.
    Screen Shot 2021-08-02 at 15 51 15
    opened by adlerhsieh 2
  • bazel: use distroless/static to base image

    bazel: use distroless/static to base image

    WHAT

    Use distroless/static:latest to base image.

    WHY

    More smallest container image.

    Note

    You can create distroless/static:debug base image with bin/bazelisk build -c dbg --platforms=@io_bazel_rules_go//go/toolchain:linux_amd64 //:image command.

    Ref

    • https://github.com/bazelbuild/rules_docker/blob/v0.12.1/go/image.bzl#L113
    opened by zchee 2
  • Some Binary Assets included before v.1.1.0 do not include in the release tags, v1.2.0 and v1.2.1.

    Some Binary Assets included before v.1.1.0 do not include in the release tags, v1.2.0 and v1.2.1.

    Expected Behavior

    Expected v1.2.0 and v1.2.1 release tags include binary Assets:

    • wrench_darwin_amd64
    • wrench_darwin_arm64
    • wrench_linux_amd64
    • wrench_linux_arm64

    Allow developers to download such binary Assets directly.

    Current Behavior

    Not included Binary Assets of the recent release tags, v1.2.0 and v1.2.1. Only source files are available to download as release tag Assets.

    https://github.com/cloudspannerecosystem/wrench/releases/tag/v1.2.0 https://github.com/cloudspannerecosystem/wrench/releases/tag/v1.2.1

    Steps to Reproduce

    1. RUN curl -L https://github.com/cloudspannerecosystem/wrench/releases/download/v1.2.0/wrench_linux_amd64 -o ./wrench && chmod +x ./wrench in the container didn't work because the Assets of release tag didn't include such the binary wrench_linux_amd64.

    Context (Environment)

    • wrench version: v1.2.0 and v1.2.1
    bug 
    opened by nsega 1
  • could not download Bazel

    could not download Bazel

    Expected Behavior

    make build should run ok

    Current Behavior

    make build bin/bazelisk build //:wrench 2022/04/27 18:44:31 could not download Bazel: could not determine filename to use for Bazel binary: unsupported machine architecture "arm64", must be x86_64 make: *** [build] Error 1

    Steps to Reproduce

    1. make build

    Context (Environment)

    Use a MacBook Pro M1

    • wrench version: master

    I cannot build wrench, probably bin/bazelisk need to be updated as well as the bazel version.

    bug 
    opened by plemanach 1
  • wrench 1.1.0 fails to parse `TIMESTAMP_SUB`

    wrench 1.1.0 fails to parse `TIMESTAMP_SUB`

    Expected Behavior

    wrench can apply DML including TIMESTAMP_SUB function.

    Current Behavior

    It seems that wrench 1.1.0 fails to parse DML including TIMESTAMP_SUB. wrench 1.0.4 can apply such DML correctly.

    Steps to Reproduce

    The following reproduction steps assumes bash or zsh.

    # $PROJECT, $INSTANCE, $DATABASE should be set properly
    
    # setup
    $ wrench apply --project $PROJECT --instance $INSTANCE --database $DATABASE --ddl <(echo "CREATE TABLE ParsingTimestampSub (CreatedAt TIMESTAMP NOT NULL) PRIMARY KEY (CreatedAt);")
    
    # OK case
    $ wrench apply --project $PROJECT --instance $INSTANCE --database $DATABASE --dml <(echo "DELETE FROM ParsingTimestampSub WHERE CreatedAt < CURRENT_TIMESTAMP();")
    0 rows affected.
    
    # error case
    $ wrench apply --project $PROJECT --instance $INSTANCE --database $DATABASE --dml <(echo "DELETE FROM ParsingTimestampSub WHERE CreatedAt < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 31 DAY);")
    Error command: apply, version: unknown
            /proc/self/fd/11:1.94: got "31", want ")" or ","
    
    # cleanup
    $ wrench apply --project $PROJECT --instance $INSTANCE --database $DATABASE --ddl <(echo "DROP TABLE ParsingTimestampSub;")
    

    Context (Environment)

    • wrench version: 1.1.0
    bug 
    opened by youxkei 4
  • Support passing option.ClientOption to NewClient in the spanner pkg

    Support passing option.ClientOption to NewClient in the spanner pkg

    WHAT

    Make it so it's possible to instantiate the client in wrench/pkg/spanner with option.ClientOption - perhaps like this:

    func NewClient(ctx context.Context, config *Config, opts ...option.ClientOption) (*Client, error)

    WHY

    I'm trying to use spannertest for unit testing and would like to perform a wrench migration on a new inmem fake db instance. However, I'd also like to avoid using SPANNER_EMULATOR_HOST as I'd like to have multiple instances of spannertest running in the unit test suite.

    It is the case that a normal spanner Client can be instantiated such that it uses the spannertest server by passing option.WithGRPCConn(conn) to spanner.NewClient with conn being a gRPC connection to the spannertest server. Perhaps the wrench NewClient could also be able to interact with a spannertest server by instantiating the spanner.Client and the apiv1.DatabaseAdminClient with this option?

    enhancement 
    opened by mskwon 0
  • Add install.sh to conveniently add mercari wrench during CircleCI

    Add install.sh to conveniently add mercari wrench during CircleCI

    WHAT

    It would be great if we can add an install.sh script like in golangcli-lint

    Exampe of Install.sh -> https://github.com/reviewdog/reviewdog/blob/master/install.sh

    Example

    jobs:
      lint:
        working_directory: ~/reviewdog
        docker:
          - image: circleci/golang:1.17
        steps:
          - checkout
          - run:
              name: System information
              command: 'echo "Golang $(go version)"'
          - run: 'curl -sfL https://raw.githubusercontent.com/reviewdog/reviewdog/master/install.sh| sh -s -- -b $(go env GOPATH)/bin'
          - run:
              command: |-
                set +o pipefail
                golint ./... | reviewdog -f=golint -name=golint-circleci -reporter=github-pr-review
    
    

    WHY

    It would make it easy to install mercari-wrench for Ci

    enhancement 
    opened by Rishabh-malhotraa 0
  • pkg/spanner - breaking - remove migration tableName parameter from API

    pkg/spanner - breaking - remove migration tableName parameter from API

    Refactor responsibility of setting the migration table name

    Previously pkg/spanner Client API expected caller to explicitly pass in the migration table name, except when doing a truncate operation, where the migration table name was hardcoded and the caller could not control it.

    BEWARE: this commit introduces a breaking change to the Client API and removes tableName. Instead, tableName can optionally be set when a Client is created. See: pkg/spanner/config.go

    Reworked cmd/ to use the new pkg/spanner API.

    WHAT

    • rework pkg/spanner Client API not to require migration table name to be passed in each method call
    • make it possible to configure migration table name through pkg/spanner Config

    WHY

    see: https://github.com/cloudspannerecosystem/wrench/issues/50

    opened by fcostin 1
  • refactoring suggestion - migration table name is hardcoded in both pkg/spanner and cmd/

    refactoring suggestion - migration table name is hardcoded in both pkg/spanner and cmd/

    while reviewing the code I noticed a minor issue with internal code structure, does not impact functionality of wrench tool:

    Currently, it is not clear which layer of the code is responsible for choosing the default migration table name:

    1. the public API exposed by pkg/spanner Client has a tableName parameter on most operations that interact with the migration table, with the exception of TruncateAllTables, which hardcodes the table name instead of letting the caller control it: https://github.com/cloudspannerecosystem/wrench/blob/9f346f26805724d5361d96ea8f3fb6b9cacfeb48/pkg/spanner/client.go#L134
    2. cmd/migrate.go also hardcodes the migration table name: https://github.com/cloudspannerecosystem/wrench/blob/9f346f26805724d5361d96ea8f3fb6b9cacfeb48/cmd/migrate.go#L36

    The code structure could be slightly cleaner if only one layer of the code was responsible for choosing the migration table name.

    I can think of a few options for refactoring to clarify this, but some have potential downsides of changing the API of exported methods of pkg/spanner Client . If it is important to not make breaking changes to pkg/spanner Client API (if users are using that as a library from their projects without using wrench command line tool) then it might be best not to do that.

    Suggested options:

    | option | description | code structure | breaking change to wrench tool? | breaking change to exported wrench pkg/spanner Client API? | | --- | --- | --- | --- | --- | | a | current structure | slightly unclear | NA, no change | NA, no change | | b | add tableName to TruncateAllTables | slightly clearer | no change | breaking change! | | c | add TruncateAllTablesV2 with tableName, deprecate TruncateAllTables | slightly clearer | no change | no change | | d | remove tableName from pkg/spanner Client, allow custom migration table name to be set using pkg/spanner Config | slightly clearer | no change | breaking change! |

    opened by fcostin 1
Releases(v1.3.0)
Owner
Cloud Spanner Ecosystem
A community contributed and supported ecosystem around Cloud Spanner.
Cloud Spanner Ecosystem
A CLI for generating ER diagrams for Ent schema

enter A CLI for generating ER (entity-relationship) diagrams for Ent schema using mermaid.js.

Ariel Mashraki 103 Dec 18, 2022
schema.org ontology command-line

schema.org ontology command-line

null 2 Oct 11, 2022
Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

Rclone ("rsync for cloud storage") is a command line program to sync files and directories to and from different cloud storage providers.

rclone 36.4k Jan 5, 2023
A component on PolarStack, a hybrid cloud Alibaba Cloud DBaaS product

What is PolarDB Stack Daemon? PolarDB Stack Daemon is a component on PolarStack, a hybrid cloud Alibaba Cloud DBaaS product, and is designed mainly fo

null 10 Nov 21, 2021
Rclone ("rsync for cloud storage") is a command-line program to sync files and directories to and from different cloud storage providers.

Website | Documentation | Download | Contributing | Changelog | Installation | Forum Rclone Rclone ("rsync for cloud storage") is a command-line progr

null 0 Nov 5, 2021
The Cloud Aviator: TUI client for cloud services (AWS, Vultr, Heroku, Render.com, ...)

=== T H E C L O U D A V I A T O R === ⠀⠀⠀⠀⠀⠀⠀⠀⢶⣦⣄⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀ ⠀⠀⠀⠀⠀⠀⠀⠀⠀⠙⢿⣷⣦⡀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀⠀

◤◢◤◢◤◢◤◢ 39 Dec 17, 2022
a work time management CLI tool for any platform

english |日本語 jobgosh | job management tool made with golang for shell a multi-platform work time management CLI tool to track and improve your day to

奥田 龍馬 10 May 16, 2022
tfacon is a CLI tool for connecting Test Management Platforms and Test Failure Analysis Classifier.

Test Failure Classifier Connector Description tfacon is a CLI tool for connecting Test Management Platforms and Test Failure Analysis Classifier. Test

Red Hat Quality Engineering 3 Jun 23, 2022
Google KMS backed Solana key management CLI tool

solana-kms solana-kms is a Google KMS backed Solana token management CLI utility. The main purpose of the tool is to ensure that the private key is ne

kubetrail.io 2 Jan 13, 2022
Command line tool for Google Cloud Datastore, written in Go

dsio dsio is a command line tool for Google Cloud Datastore. This tool is under development. Please use in your own risk. Features Bulk upsert entitie

Shinichi Nishimura 66 Feb 8, 2022
A CLI tool for GCP Cloud Build Triggers

cbtctl A CLI tool for GCP Cloud Build Triggers Usage Run a build trigger with substitutions $ cbtctl run --project <GCP Project Name> --trigger <Trigg

Yuki Ito 1 Nov 9, 2021
Coala-cli-tester: a tool for testing infrastructure of Keenetic Cloud

coala-cli-tester coala-cli-tester is a tool for testing infrastructure of Keenetic Cloud Build go build -ldflags="-s -w" -o ./coala-cli-tester ./*.go

Alexey Spiridonov 0 Dec 8, 2021
e2d is a command-line tool for deploying and managing etcd clusters, both in the cloud or on bare-metal

e2d is a command-line tool for deploying and managing etcd clusters, both in the cloud or on bare-metal. It also includes e2db, an ORM-like abstraction for working with etcd.

Chris Marshall 2 Aug 23, 2022
A personal knowledge management and sharing system for VSCode

Foam ?? This is an early stage project under rapid development. For updates join the Foam community Discord! ?? Foam is a personal knowledge managemen

Foam 13.3k Jan 9, 2023
SSH Lightweight management tools

tssh golang 实现的ssh 工具 安装 下载安装 下载地址 release windows用户请手动下载,暂时不提供一键安装模式( 主要是批处理脚本不会写 ) Mac一键安装 /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.

lrs 125 Sep 14, 2022
Go library and CLI utility for /etc/hosts management.

Etc Hosts Management Utility & Go Library /etc/hosts Management It is easy to open your /etc/hosts file in text editor and add or remove entries. Howe

txn2: Kubernetes apps and system utilities. 277 Nov 27, 2022
StudentMgmtSystem - Project for Student Management System and use of CURD Operation

StudentMgmtSystem Project for Student Management System and use of CURD Operatio

null 0 Jan 4, 2022
A Target Tracking , NoteTaking , CheckLists and Data Management GUI App for Bug Hunter's and Pentesters.

Screenshots Features • Installation • Usage • Features • Notes • Sandman A Target Tracking , NoteTaking , CheckLists and Data Management GUI App for B

Tarun Koyalwar 29 Dec 25, 2022
Reads from existing Cloud Providers (reverse Terraform) and generates your infrastructure as code on Terraform configuration

TerraCognita Imports your current Cloud infrastructure to an Infrastructure As Code Terraform configuration (HCL) or/and to a Terraform State. At Cycl

Cycloid 1.4k Dec 30, 2022