Run a real Postgres database locally on Linux, OSX or Windows as part of another Go application or test

Overview

Godoc Coverage Status Build Status Go Report Card

embedded-postgres

Run a real Postgres database locally on Linux, OSX or Windows as part of another Go application or test.

When testing this provides a higher level of confidence than using any in memory alternative. It also requires no other external dependencies outside of the Go build ecosystem.

Heavily inspired by Java projects zonkyio/embedded-postgres and opentable/otj-pg-embedded and reliant on the great work being done by zonkyio/embedded-postgres-binaries in order to fetch precompiled binaries from Maven.

Installation

embedded-postgres uses Go modules and as such can be referenced by release version for use as a library. Use the following to add the latest release to your project.

go get -u github.com/fergusstrange/embedded-postgres

How to use

This library aims to require as little configuration as possible, favouring overridable defaults

Configuration Default Value
Username postgres
Password postgres
Database postgres
Version 12.1.0
RuntimePath $USER_HOME/.embedded-postgres-go/extracted
Port 5432
StartTimeout 15 Seconds

A single Postgres instance can be created, started and stopped as follows

postgres := embeddedpostgres.NewDatabase()
err := postgres.Start()

// Do test logic

err := postgres.Stop()

or with created with custom configuration

logger := &bytes.Byffer{}
postgres := NewDatabase(DefaultConfig().
            Username("beer").
            Password("wine").
            Database("gin").
            Version(V12).
            RuntimePath("/tmp").
            Port(9876).
            StartTimeout(45 * time.Second).
            Logger(logger))
err := postgres.Start()

// Do test logic

err := postgres.Stop()

It should be noted that if postgres.Stop() is not called then the child Postgres process will not be released and the caller will block.

Examples

There are a number of realistic representations of how to use this library in examples.

Credits

Contributing

View the contributing guide.

Issues
  • Cannot find binary in archive

    Cannot find binary in archive

    Hey all, stumbled upon this repository and decided to give it a try.

    Unfortunately the following snippet does not seem to work me.

    pg := embeddedpostgres.NewDatabase()
    
    if err := pg.Start(); err != nil {
      t.Error(err)
      return
    }
    
    defer pg.Stop()
    

    This gives me the error error fetching postgres: cannot find binary in archive retrieved from https://repo1.maven.org/maven2/io/zonky/test/postgres/embedded-postgres-binaries-linux-amd64/12.8.0/embedded-postgres-binaries-linux-amd64-12.8.0.jar

    go version gives me: go version go1.17.1 linux/amd64 and I'm running Ubuntu 20.04.01 with kernel 5.8.0-23-generic.

    Am I doing something wrong?

    opened by survivorbat 16
  • Add a BinariesPath configuration option, for prefetching Postgres binaries before the test

    Add a BinariesPath configuration option, for prefetching Postgres binaries before the test

    Downloading and Un-archiving the postgres binaries is slow and flaky in some CI systems. For example - in my case, our CI system starts a fresh sandbox for every test it runs, and as such caching the downloaded postgres file is impossible. Even when it is possible to cache the downloaded file, it's still might be slow due to the time it takes to unarchive it.

    The change in this PR does the following: It gives an option to specify a binaries path, where the binaries will be unarchived to, and will not get deleted between tests. If the BinariesPath/bin directory exists, the system will assume files were pre-downloaded, and will not try to fetch/unarchive them.

    The PR is split into 4 commits:

    1. Refactor of the usage of the folders, in order to not "recalculate" the data/runtime directories over and over again (makes code simpler)
    2. Separates the variable holding the runtimePath from the one holding the binaryExtractLocation (as those can now be different).
    3. Adds the BinariesPath to the configuration, and adds the logic to reuse unarchived binaries (as well as adds new tests)
    4. Updates README (documentation)

    This change should be fully backward compatible if BinariesPath is not specified.

    This also helps with #35 (@mbfr).

    opened by mishas 14
  • Fixed CVE-2020-16845 by upgrading xz

    Fixed CVE-2020-16845 by upgrading xz

    Upgraded xz to fix this CVE https://github.com/ulikunitz/xz/security/advisories/GHSA-25xm-hr59-7c27

    https://deps.dev/go/github.com%2Ffergusstrange%2Fembedded-postgres

    opened by naveensrinivasan 10
  • unable to extract postgres archive: xz: data is truncated or corrupt

    unable to extract postgres archive: xz: data is truncated or corrupt

    When using embedded-postgres in a docker container we sometimes see the following error:

    Failed to start embedded postgres: unable to extract postgres archive: xz: data is truncated or corrupt Perhaps there's a timing error in the code that fetches the jar file from maven and extracts the embedded .tgz file containing the postgres binaries.

    opened by jfigus 9
  • Fix goroutine and connection leak on database connect

    Fix goroutine and connection leak on database connect

    Closes #58

    I also added a goroutine leak detector to prevent unclosed connections in the future and one exception for goroutine leak detector, since I did not find how to fix it.

    opened by EldarKurbanov 9
  • Support Intel and Arm MAC

    Support Intel and Arm MAC

    Note: this issue is created to isolate from the other issues that per affected on Linux and windows.

    Look into support for better logging on new Mac ARM, with rosetta emulation support by default.

    Longer term support for Mac ARM when the packages are available on Maven.

    opened by gedw99 8
  • cleanup deleted my project

    cleanup deleted my project

    The cleanup just deleted my whole project because I set RuntimePath to ./ and apparently the Stop method deletes the entire directory? Luckily version control, but I lost all my certs and an hour or so of work. Cool project, but this was a pretty big surprise and a major foot-gun.

    opened by erichanson 7
  • Unable to run in official docker golang:1.15

    Unable to run in official docker golang:1.15

    unable to init database using: /root/.embedded-postgres-go/extracted/bin/initdb -A password -U postgres -D /root/.embedded-postgres-go/extracted/data --pwfile=/root/.embedded-postgres-go/extracted/pwfile

    opened by thetruechar 5
  • Add locale config option

    Add locale config option

    This makes the locale of the database created configurable. E.g. for consistent testing where some things depend on the database locale, this is helpful. Or also on broken systems, where the default locale is an invalid string.

    I also bumped the versions to the latest point releases.

    opened by tachiniererin 5
  • Logger parametr does'twork correctly

    Logger parametr does'twork correctly

    Hey... I figured out that when I do something like ....... logger, _ := zap.NewProduction() defer logger.Sync() logger.Info("failed to fetch URL", // Structured context as strongly typed Field values. zap.String("url", url), zap.Int("attempt", 3), zap.Duration("backoff", time.Second), ) .... w := &zapio.Writer{Log: logger}

    ...... database := embeddedpostgres.NewDatabase(embeddedpostgres.DefaultConfig(). Version(embeddedpostgres.V12). Port(Port). RuntimePath(runtimePath). StartTimeout(CACHE_DB_STARTUP_DELLAY_SECONDS). Database(DbName). Password(DbUsername). Username(DbPasswd). Logger(w))

    the new io.writer for the instance is used (I see a new zap formatting applied for all messages) but by the end of the database startup it hangs and I need to kill the app.

    i tried the lorgus and standard go logger and all of them were hanging after the database started up. The reason to use this embedded Postgre database is to get all logging coming from one place otherwise we can easily run the postgres instance at the pod and make it "embedded" so I think logging not working correctly is the ISSUE.

    opened by divo77 4
  • Panic leads to stuck process/process not being closed.

    Panic leads to stuck process/process not being closed.

    Hey, awesome work so far, I tried the library, the issue I had is, if the test code is panicking the process will never be killed, even using recover did not work for me. The test were just stuck. And I had a postgres process blocking the configured port which needs to be killed manually.

    Any advice?

    What I tried from a setup point of view is the following, but the test still got stuck on panic and I had a process still running:

    func TestMain(m *testing.M) {
    	testPostgresInstance = embeddedpostgres.NewDatabase()
    	if err := testPostgresInstance.Start(); err != nil {
    		panic(err)
    	}
    	defer func() {
    		if err := testPostgresInstance.Stop(); err != nil {
    			panic(err)
    		}
    	}()
    	m.Run()
    	if r := recover(); r != nil {
    		if err := testPostgresInstance.Stop(); err != nil {
    			panic(err)
    		}
    	}
    }
    
    opened by dmeyer22 4
  • Add runAsUser functionality

    Add runAsUser functionality

    By supplying runAsUser it is possible to run initdb as non-root user (which otherwise fails) even if the calling process runs as root.

    In addition, flush logger on failures to get more detailed errors.

    opened by same-id 1
  • Add error info when 'unable to init db'

    Add error info when 'unable to init db'

    Currently, I get this error, without any useful information:

    panic: localpg: failed to start PgSQL: unable to init database using: /tmp/go_embedded_pgsql_binaries_3773798719/bin/initdb -A password -U username -D /tmp/go_embedded_pgsql_runtime_2105005878/data --pwfile=/tmp/go_embedded_pgsql_runtime_2105005878/pwfile --locale=en_US.UTF-8
    
    opened by dor-utila 0
  • Add CachePath as a configuration option

    Add CachePath as a configuration option

    This allows a configurable cache path to store postgres archives. Callers can centralize the location of all data stored by this library with this change.

    opened by kylecarbs 2
Releases(v1.17.0)
Owner
Fergus Strange
Fergus Strange
Ruby on Rails like test fixtures for Go. Write tests against a real database

testfixtures Warning: this package will wipe the database data before loading the fixtures! It is supposed to be used on a test database. Please, doub

null 826 Aug 1, 2022
Flugel Test Documentation for steps to run and test the automatio

Flugel Test Documentation Documentation for steps to run and test the automation #Test-01 1 - Local Test Using Terratest (End To End) 1- By runing " t

Suc0_Unix 0 Dec 30, 2021
This is a simple test application that sends fake video data from one pion instance to another

Pion test app for BWE This is a simple test application that sends fake video data from one pion instance to another. It is a modified version of the

Mathis Engelbart 2 Jun 8, 2022
Cloud Spanner load generator to load test your application and pre-warm the database before launch

GCSB GCSB Quickstart Create a test table Load data into table Run a load test Operations Load Single table load Multiple table load Loading into inter

Cloud Spanner Ecosystem 21 Jul 13, 2022
go-test-trace is like go test but it also generates distributed traces.

go-test-trace go-test-trace is like go test but it also generates distributed traces. Generated traces are exported in OTLP to a OpenTelemetry collect

JBD 373 Jun 24, 2022
Test-assignment - Test assignment with golang

test-assignment We have a two steam of data and we need to save it in the map: I

null 0 Jan 19, 2022
This repository includes consumer driven contract test for provider, unit test and counter api.

This repository includes consumer driven contract test for provider, unit test and counter api.

Ahmet Zümberoğlu 0 Feb 1, 2022
Sql mock driver for golang to test database interactions

Sql driver mock for Golang sqlmock is a mock library implementing sql/driver. Which has one and only purpose - to simulate any sql driver behavior in

DATA-DOG 4.6k Aug 6, 2022
Plow is a high-performance HTTP benchmarking tool with real-time web UI and terminal displaying

Plow is a HTTP(S) benchmarking tool, written in Golang. It uses excellent fasthttp instead of Go's default net/http due to its lightning fast performance.

ddc 3.2k Jul 29, 2022
Load generator for measuring overhead generated by EDRs and other logging tools on Linux

Simple load generator for stress-testing EDR software The purpose of this tool is to measure CPU overhead incurred by having active or passive securit

Hilko Bengen 6 Jun 14, 2022
manage your mocks / run mockgen more quickly / mocks up-to-date checking

gomockhandler If you find any bugs or have feature requests, please feel free to create an issue. gomockhandler is handler of golang/mock, as the name

Kensei Nakada 67 Aug 1, 2022
How we can run unit tests in parallel mode with failpoint injection taking effect and without injection race

This is a simple demo to show how we can run unit tests in parallel mode with failpoint injection taking effect and without injection race. The basic

amyangfei 1 Oct 31, 2021
Ritchie CLI is an open-source tool that allows to create, store and share any kind of automation, executing them through command lines, to run operations or start workflows ⚙️ 🖥 💡

Table of contents 1. About 2. Getting Started i. Installation ii. Initialize rit locally iii. Add your first formulas repository iv. Run the Hello Wor

ZUP IT INNOVATION 555 Jul 27, 2022
End to end functional test and automation framework

Declarative end to end functional testing (endly) This library is compatible with Go 1.12+ Please refer to CHANGELOG.md if you encounter breaking chan

Viant, Inc 214 Jul 23, 2022
Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻

Gnomock – tests without mocks ??️ Spin up entire dependency stack ?? Setup initial dependency state – easily! ?? Test against actual, close to product

Yury Fedorov 881 Aug 7, 2022
go-carpet - show test coverage in terminal for Go source files

go-carpet - show test coverage for Go source files To view the test coverage in the terminal, just run go-carpet. It works outside of the GOPATH direc

Sergey Mudrik 227 Aug 1, 2022
http integration test framework

go-hit hit is an http integration test framework written in golang. It is designed to be flexible as possible, but to keep a simple to use interface f

Tobias Salzmann 119 Jul 29, 2022
Go testing in the browser. Integrates with `go test`. Write behavioral tests in Go.

GoConvey is awesome Go testing Welcome to GoConvey, a yummy Go testing tool for gophers. Works with go test. Use it in the terminal or browser accordi

SmartyStreets 7.3k Aug 1, 2022
A Go test assertion library for verifying that two representations of JSON are semantically equal

jsonassert is a Go test assertion library for verifying that two representations of JSON are semantically equal. Usage Create a new *jsonassert.Assert

Roger Guldbrandsen 86 Jul 16, 2022