High quality cloud service emulators for local development stacks

Overview

emulators

High quality Google Cloud service emulators for local development stacks

Why?

At FullStory, our entire product and backend software stack runs in each engineer's local workstation. This high-quality local development experience keeps our engineers happy and productive, because they are able to:

  • build and test features locally
  • reproduce and fix bugs quickly and easily
  • run high-quality services in unit and integration tests

Our local development and testing story is simpler when our live code can rely on expected services to exist, and we don't have to write as many mocks.

Some of the ways we achieve this:

  • Our own backend services operate in a reasonable manner in a local environment.
  • Open source, third party services (such as Redis, Zookeeper, or Solr) run locally.
  • We emulate Google Cloud infrastructure.

What Google Cloud services do we emulate?

Service Persistence? Status Notes
Google Bigtable Yes Shipped, see below Fork of bigtable/bttest
Google Cloud Storage (GCS) Yes Shipped, see below Written from scratch
Google Pubsub No Considering persistence Vanilla pubsub/pstest
Google Cloud Functions n/a In consideration Thin wrapper that manages node processes.
Google Datastore Yes - Google's Datastore emulator (written in Java) works great

Google Bigtable Emulator

Our bigtable emulator is a fork of bigtable/bttest. A summary of the changes we made:

  • The core operates directly on Bigtable protobuf types, such as Table and Row, instead of bespoke types.
  • The storage layer is pluggable and operates on protos.
  • Leveldb is the default storage implementation, and runs either in-memory (transient for unit tests) or on disk (long running, persistence).

Installing

go get github.com/fullstorydev/emulators/bigtable
go install github.com/fullstorydev/emulators/bigtable/... # for the command-line `cbtemulator`

Running, out of process

Example, running on a specific port, with persistence:

> cbtemulator -port 8888 -dir var/bigtable
Writing to: var/bigtable
Cloud Bigtable emulator running on 127.0.0.1:8888

Usage:

  -dir string
    	if set, use persistence in the given directory
  -host string
    	the address to bind to on the local machine (default "localhost")
  -port int
    	the port number to bind to on the local machine (default 9000)

Running, in process

You can link bigtable emulator into existing Go binaries as a drop-in replacement for bigtable/bttest.

For unit tests:

	// start an in-memory leveldb BigTable test server (for unit tests)
	srv, err := bttest.NewServer("127.0.0.1:0", grpc.MaxRecvMsgSize(math.MaxInt32))
	if err != nil { 
		// ...
	}
	defer srv.Close()
	// bigtable.NewClient (via DefaultClientOptions) will look at this env var to figure out what host to talk to
	os.Setenv("BIGTABLE_EMULATOR_HOST", svr.Addr)

For on-disk persistence:

	// start an leveldb-backed BigTable test server
	srv, err := bttest.NewServerWithOptions(fmt.Sprintf("127.0.0.1:%d", *btport), bttest.Options{
		Storage: bttest.LeveldbDiskStorage{
			Root: bigtableStorageDir,
			ErrLog: func(err error, msg string) {
				// wire into logging
			},
		},
		GrpcOpts: []grpc.ServerOption{grpc.MaxRecvMsgSize(maxGrpcMessageSize)},
	})

Connecting to the Bigtable emulator from Go

	// assuming BIGTABLE_EMULATOR_HOST is already set...
	conn, err := grpc.Dial(os.Getenv("BIGTABLE_EMULATOR_HOST"), grpc.WithInsecure())
	if err != nil {
		// ...
	}
	defer conn.Close() // only if the life cycle is scoped to this call

	client, err := bigtable.NewClient(ctx, project, instance, option.WithGRPCConn(conn))
	if err != nil {
		// ...
	}
	tbl := client.Open("example")

Google Cloud Storage Emulator

Our storage emulator was written in house.

  • Supports basic file operations, iteration, attributes, copying, and some conditionals.
  • The storage layer is pluggable.
  • In memory btree (transient for unit tests) or disk-based storage (long running, persistence).

Installing

go get github.com/fullstorydev/emulators/storage
go install github.com/fullstorydev/emulators/storage/... # for the command-line `gcsemulator`

Running, out of process

Example, running on a specific port, with persistence:

> gcsemulator -port 8888 -dir var/storage
Writing to: var/storage
Cloud Storage emulator running on http://127.0.0.1:8888

Usage:

  -dir string
    	if set, use persistence in the given directory
  -host string
    	the address to bind to on the local machine (default "localhost")
  -port int
    	the port number to bind to on the local machine (default 9000)

For unit tests:

	// start an in-memory Storage test server (for unit tests)
	svr, err := gcsemu.NewServer("127.0.0.1:0", gcsemu.Options{})
	if err != nil {
		// ...
	}
	defer svr.Close()
	// gcsemu.NewClient will look at this env var to figure out what host/port to talk to
	os.Setenv("GCS_EMULATOR_HOST", svr.Addr)

For on-disk persistence:

	// start an on-disk Storage test server
	svr, err := gcsemu.NewServer(fmt.Sprintf("127.0.0.1:%d", *port), gcsemu.Options{
		Store: gcsemu.NewFileStore(*gcsDir),
	})

Connecting to the GCS emulator from Go

	// assuming GCS_EMULATOR_HOST is already set...
	client, err := gcsemu.NewClient(ctx)
	if err != nil {
		// ...
	}
	defer client.Close() // only if the life cycle is scoped to this call

NOTE

Do NOT use STORAGE_EMULATOR_HOST, as defined in cloud.google.com/go/storage. There are unresolved issues in the Go client implementation. STORAGE_EMULATOR_HOST is supported inconsistently, and even has some bugs that can cause data races when using the same *storage.Client for different types of access.

See:

Instead, use our gcsemu.NewClient(ctx) method which swaps out the entire HTTP transport.

You might also like...
Cloud cost estimates for Terraform in your CLI and pull requests 💰📉
Cloud cost estimates for Terraform in your CLI and pull requests 💰📉

Infracost shows cloud cost estimates for Terraform projects. It helps developers, devops and others to quickly see the cost breakdown and compare different options upfront.

Cloud-native way to provide elastic Jupyter Notebook services on Kubernetes
Cloud-native way to provide elastic Jupyter Notebook services on Kubernetes

elastic-jupyter-operator: Elastic Jupyter on Kubernetes Kubernetes 原生的弹性 Jupyter 即服务 介绍 为用户按需提供弹性的 Jupyter Notebook 服务。elastic-jupyter-operator 提供以下特性

Google Cloud Client Libraries for Go.
Google Cloud Client Libraries for Go.

Google Cloud Client Libraries for Go.

A Cloud Native Buildpack for Go

The Go Paketo Buildpack provides a set of collaborating buildpacks that enable the building of a Go-based application.

Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload.
Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload.

Fleex allows you to create multiple VPS on cloud providers and use them to distribute your workload. Run tools like masscan, puredns, ffuf, httpx or anything you need and get results quickly!

☁️🏃 Get up and running with Go on Google Cloud.

Get up and running with Go and gRPC on Google Cloud Platform, with this lightweight, opinionated, batteries-included service SDK.

Elkeid is a Cloud-Native Host-Based Intrusion Detection solution project to provide next-generation Threat Detection and Behavior Audition with modern architecture.
Elkeid is a Cloud-Native Host-Based Intrusion Detection solution project to provide next-generation Threat Detection and Behavior Audition with modern architecture.

Elkeid is a Cloud-Native Host-Based Intrusion Detection solution project to provide next-generation Threat Detection and Behavior Audition with modern architecture.

Sample apps and code written for Google Cloud in the Go programming language.
Sample apps and code written for Google Cloud in the Go programming language.

Google Cloud Platform Go Samples This repository holds sample code written in Go that demonstrates the Google Cloud Platform. Some samples have accomp

Use Google Cloud KMS as an io.Reader and rand.Source.

Google Cloud KMS Go io.Reader and rand.Source This package provides a struct that implements Go's io.Reader and math/rand.Source interfaces, using Goo

Comments
  • Resumable upload is broken using official go storage client

    Resumable upload is broken using official go storage client

    Description

    this issue is for the GCS emulator

    When doing a resumable upload a post request is sent to the emulator which returns the "real" address to a file in GCS in the Location header.

    Example POST to the emulator from the official go client: POST http://localhost:8080/upload/storage/v1/b/my-bucket/o The response contains has the Location header pointing to https://www.googleapis.com/storage/v1/b/my-bucket/o/my-object.ext?upload_id=4

    This location field is then used by the official storage library for further requests, essentially skipping the emulator for all subsequent upload chucks (hanging until context is cancelled in my case).

    Fix 1

    I've verified a fix by manually changing the location header value to http://localhost:8080/storage/v1/b/my-bucket/o/my-object.ext?upload_id=4 before it is used by the official client. I did so by overriding the value using the debugger in this file: google.golang.org/[email protected]/storage/v1/storage-gen.go:10295

    I suspect these fields come from storage/gcsemu/meta.go?

    Fix 2

    Setting the chunk size of the writer to >= file.size() also solves this issue, but then it isn't a resumable upload...

    It would be nice if you could update so that the location header points to the local emulator rather than the real official GCS url.

    I know you say that I shouldn't rely on STORAGE_EMULATOR_HOST, but with this (hopefully) minor fix the library works as expected for me at least...

    EDIT:

    I've verified that this quickfix solves the issue for me:

    diff --git a/storage/gcsemu/meta.go b/storage/gcsemu/meta.go
    index aee05f1..df8740e 100644
    --- a/storage/gcsemu/meta.go
    +++ b/storage/gcsemu/meta.go
    @@ -47,10 +47,10 @@ func scrubMeta(meta *storage.Object) {
     // Return the URL for a file.  Note that this will generally not be useful as an input an http request due to the way
     // that net/url does character substitution (see comments above).
     func bucketUrl(bucket string) string {
    -	return fmt.Sprintf("https://www.googleapis.com/storage/v1/b/%s", bucket)
    +	return fmt.Sprintf("http://localhost:8080/storage/v1/b/%s", bucket)
     }
     
     // Return the URL for a file.
     func objectUrl(bucket string, filepath string) string {
    -	return fmt.Sprintf("https://www.googleapis.com/storage/v1/b/%s/o/%s", bucket, filepath)
    +	return fmt.Sprintf("http://localhost:8080/storage/v1/b/%s/o/%s", bucket, filepath)
     }
    
    opened by Edholm 6
  • Pluggable storage layer interface can't be implemented

    Pluggable storage layer interface can't be implemented

    The storage layer is pluggable, but the interface contains private types, so it can't be implemented by any type outside of the gcsemu package. Neither of the two included storage implementations meet my needs, so I would like to be able to provide my own.

    What do you think about exporting gcsemu.httpBaseUrl?

    opened by iangudger 3
  • Bucket.Delete doesn't return an error

    Bucket.Delete doesn't return an error

    Hi, First of all, congratulations on great work on these Google Emulators.

    I am trying to use a storage emulator to test one golang application that uses GCS. But when I try test error on delete bucket that doesn't exist bucket. The emulator doesn't return any error to inform that Bucket doesn't exist.

    The real GCS API returns one error when I try to delete on the bucket that doesn't exist.

    Is this behavior expected by this emulator?

    opened by LeoCBS 1
Owner
Engineering at FullStory
FullStory APIs and Open-Source Projects
Engineering at FullStory
A local emulator for Cloud Bigtable with persistance to a sqlite3 backend.

Little Bigtable A local emulator for Cloud Bigtable with persistance to a sqlite3 backend. The Cloud SDK provided cbtemulator is in-memory and does no

Bitly 14 Sep 21, 2022
TurtleDex is a decentralized cloud storage platform that radically alters the landscape of cloud storage.

TurtleDex is a decentralized cloud storage platform that radically alters the landscape of cloud storage. By leveraging smart contracts, client-side e

TurtleDev 18 Feb 17, 2021
Cloudpods is a cloud-native open source unified multi/hybrid-cloud platform developed with Golang

Cloudpods is a cloud-native open source unified multi/hybrid-cloud platform developed with Golang, i.e. Cloudpods is a cloud on clouds. Cloudpods is able to manage not only on-premise KVM/baremetals, but also resources from many cloud accounts across many cloud providers. It hides the differences of underlying cloud providers and exposes one set of APIs that allow programatically interacting with these many clouds.

null 1 Jan 11, 2022
Contentrouter - Protect static content via Firebase Hosting with Cloud Run and Google Cloud Storage

contentrouter A Cloud Run service to gate static content stored in Google Cloud

G. Hussain Chinoy 0 Jan 2, 2022
a high-performance, POSIX-ish Amazon S3 file system written in Go

Goofys allows you to mount an S3 bucket as a filey system.

Ka-Hing Cheung 4.3k Sep 22, 2022
Lightweight Cloud Instance Contextualizer

Flamingo Flamingo is a lightweight contextualization tool that aims to handle initialization of cloud instances. It is meant to be a replacement for c

Tamer Tas 36 Jun 18, 2022
Go language interface to Swift / Openstack Object Storage / Rackspace cloud files (golang)

Swift This package provides an easy to use library for interfacing with Swift / Openstack Object Storage / Rackspace cloud files from the Go Language

Nick Craig-Wood 292 Jun 30, 2022
The extensible SQL interface to your favorite cloud APIs.

The extensible SQL interface to your favorite cloud APIs.

Turbot 2.4k Sep 22, 2022
Terraform provider for HashiCorp Cloud Platform.

HashiCorp Cloud Platform (HCP) Terraform Provider Requirements Terraform >= 0.12.x Go >= 1.14 Building The Provider Clone the repository Enter the rep

HashiCorp 27 Jul 4, 2022
The Cloud Posse Terraform Provider for various utilities (E.g. deep merging)

terraform-provider-utils Terraform provider to add additional missing functionality to Terraform This project is part of our comprehensive "SweetOps"

Cloud Posse 64 Sep 22, 2022