Record and replay your HTTP interactions for fast, deterministic and accurate tests

Overview

go-vcr

Build Status GoDoc Go Report Card codecov

go-vcr simplifies testing by recording your HTTP interactions and replaying them in future runs in order to provide fast, deterministic and accurate testing of your code.

go-vcr was inspired by the VCR library for Ruby.

Installation

Install go-vcr by executing the command below:

$ go get github.com/dnaeon/go-vcr/recorder

Usage

Here is a simple example of recording and replaying etcd HTTP interactions.

You can find other examples in the example directory of this repository as well.

package main

import (
	"log"
	"time"

	"github.com/dnaeon/go-vcr/recorder"

	"github.com/coreos/etcd/client"
	"golang.org/x/net/context"
)

func main() {
	// Start our recorder
	r, err := recorder.New("fixtures/etcd")
	if err != nil {
		log.Fatal(err)
	}
	defer r.Stop() // Make sure recorder is stopped once done with it

	// Create an etcd configuration using our transport
	cfg := client.Config{
		Endpoints:               []string{"http://127.0.0.1:2379"},
		HeaderTimeoutPerRequest: time.Second,
		Transport:               r, // Inject as transport!
	}

	// Create an etcd client using the above configuration
	c, err := client.New(cfg)
	if err != nil {
		log.Fatalf("Failed to create etcd client: %s", err)
	}

	// Get an example key from etcd
	etcdKey := "/foo"
	kapi := client.NewKeysAPI(c)
	resp, err := kapi.Get(context.Background(), etcdKey, nil)

	if err != nil {
		log.Fatalf("Failed to get etcd key %s: %s", etcdKey, err)
	}

	log.Printf("Successfully retrieved etcd key %s: %s", etcdKey, resp.Node.Value)
}

Custom Request Matching

During replay mode, You can customize the way incoming requests are matched against the recorded request/response pairs by defining a Matcher function. For example, the following matcher will match on method, URL and body:

r, err := recorder.New("fixtures/matchers")
if err != nil {
	log.Fatal(err)
}
defer r.Stop() // Make sure recorder is stopped once done with it

r.SetMatcher(func(r *http.Request, i cassette.Request) bool {
	if r.Body == nil {
		return cassette.DefaultMatcher(r, i)
	}
	var b bytes.Buffer
	if _, err := b.ReadFrom(r.Body); err != nil {
		return false
	}
	r.Body = ioutil.NopCloser(&b)
	return cassette.DefaultMatcher(r, i) && (b.String() == "" || b.String() == i.Body)
})

Protecting Sensitive Data

You often provide sensitive data, such as API credentials, when making requests against a service. By default, this data will be stored in the recorded data but you probably don't want this. Removing or replacing data before it is stored can be done by adding one or more Filters to your Recorder. Here is an example that removes the Authorization header from all requests:

r, err := recorder.New("fixtures/filters")
if err != nil {
	log.Fatal(err)
}
defer r.Stop() // Make sure recorder is stopped once done with it

// Add a filter which removes Authorization headers from all requests:
r.AddFilter(func(i *cassette.Interaction) error {
    delete(i.Request.Headers, "Authorization")
    return nil
})

Sensitive data in responses

Filters added using *Recorder.AddFilter are applied within VCR's custom http.Transport. This means that if you edit a response in such a filter then subsequent test code will see the edited response. This may not be desirable in all cases. For instance, if a response body contains an OAuth access token that is needed for subsequent requests, then redact the access token in SaveFilter will result in authorization failures.

Another way to edit recorded interactions is to use *Recorder.AddSaveFilter. Filters added with this method are applied just before interactions are saved when *Recorder.Stop is called.

r, err := recorder.New("fixtures/filters")
if err != nil {
	log.Fatal(err)
}
defer r.Stop() // Make sure recorder is stopped once done with it

// Your test code will continue to see the real access token and
// it is redacted before the recorded interactions are saved     
r.AddSaveFilter(func(i *cassette.Interaction) error {
    if strings.Contains(i.URL, "/oauth/token") {
        i.Response.Body = `{"access_token": "[REDACTED]"}`
    }

    return nil
})

Passing Through Requests

Sometimes you want to allow specific requests to pass through to the remote server without recording anything.

Globally, you can use ModeDisabled for this, but if you want to disable the recorder for individual requests, you can add Passthrough functions to the recorder. The function takes a pointer to the original request, and returns a boolean, indicating if the request should pass through to the remote server.

Here's an example to pass through requests to a specific endpoint:

// Pass through the request to the remote server if the path matches "/login".
r.AddPassthrough(func(req *http.Request) bool {
    return req.URL.Path == "/login"
})

License

go-vcr is Open Source and licensed under the BSD License

Issues
  • feat: add `*Recorder.AddSaveFilter` to redact interactions before saving

    feat: add `*Recorder.AddSaveFilter` to redact interactions before saving

    Closes #54

    Adds a new type of filter that is applied to interactions just before they are saved. These filters are added using *Recorder.AddSaveFilter.

    Why do we need this?

    (the following explanation assumes VCR is in recording mode)

    Imagine a work with an API that authenticates and authorizes requests using OAuth client credentials. The first step when interacting with this API is to acquire an access token and then make the intended API requests:

    > POST /oauth/token
    client_id=my_sample_app&client_secret=super-secret-credentials
    
    < {"access_token": "390278941074fe1a1acd"}
    
    
    > POST /blog/1234/comments
    > Authorization: Bearer 390278941074fe1a1acd
    
    {"message": "cool blog posts"}
    
    < {"comment_id": "8043891dde"}
    

    If we use an AddFilter to redact to scrub the access_token like so:

    rec.AddFilter(func(i *cassette.Interaction) error {
    	i.Response.Body = `{"access_token": "[REDACTED]"}`
    	return nil
    })
    

    Then the second request will be made like so:

    > POST /blog/1234/comments
    > Authorization: Bearer [REDACTED]
    
    {"message": "cool blog posts"}
    
    < {"comment_id": "8043891dde"}
    

    This is because the response is mutated in the recorder's RoundTripper and the mutation is used to build the http.Response before being passed on to the caller.

    So this means we cannot interfere with the response at the point where the current filters are added.

    Solution

    To get around this problem, we allow VCR users to register another set of filters that are applied when interactions are being saved. At that point, it is safe to mutate recorded interactions.

    opened by disintegrator 8
  • Add a new mode: ModeRecordingOrReplaying

    Add a new mode: ModeRecordingOrReplaying

    I'd like to have a new mode: ModeRecordingOrReplaying. It will do the recording if cassette returns ErrInteractionNotFound. Otherwise it does replaying. It is useful to me. How about others?

    opened by haibin 6
  • Strip Authentication string from headers

    Strip Authentication string from headers

    Hello!

    I was wondering, is there any way of modifying what is being dumped to the recorded response? For example, in the original ruby vcr gem there is an option to filter_sensitive_data, but for go-vcr package I can't figure out a way how to do the same.

    Currently, I'm doing this as a quick workaround, but I don't think that it's the best way to do that:

    if copiedReq.Header["Authorization"] != nil {
      copiedReq.Header["Authorization"] = []string{"<FILTERED>"}
    }
    
    // Add interaction to cassette
    interaction := &cassette.Interaction{
      Request: cassette.Request{
        Headers: copiedReq.Header,
        ...
      },
      ...
    }
    

    Could anyone suggest how to do that better? Maybe we can introduce configurable Interaction struct which will be an optional parameter of a New?

    opened by br4in3x 5
  • Update gopkg.in/yaml.v2 from v2.2.1 to v2.2.8

    Update gopkg.in/yaml.v2 from v2.2.1 to v2.2.8

    • Updated yaml.v2 version in go.mod from v2.2.1 to v2.2.8
    • Executed go mod vendor to update content of vendor directory
    • Executed make test to run included unittests
    opened by Blesmol 4
  • Tests fail if filter modifies URL

    Tests fail if filter modifies URL

    Adding this filter to my recorder makes tests to fail.

    		r.AddFilter(func(_ *cassette.Interaction) error {
    			u, _ := url.Parse(i.Request.URL)
    			q := u.Query()
    			q.Set("access_key", "no-op")
    			u.RawQuery = q.Encode()
    			i.Request.URL = u.String()
    			return nil
    		})
    

    The service I'm building this library for insists on using the access key as a URL param since they've been doing that since launch nearly 10 years ago.

    Since I'm snapshot testing the output of the requests, I don't want the access key getting logged.

    But, modifying the interaction causes subsequent tests to fail.

    What can I do to fix this?

    Thanks!

    opened by kayandra 4
  • honour request context even during replay

    honour request context even during replay

    The context.Context in http.Requests needs to be checked & honoured, even during a replay. Otherwise a change in tests or behaviour may be masked & not caught by tests.

    opened by retnuh 4
  • Provide higher level recorder utilties for recording all tests in a package

    Provide higher level recorder utilties for recording all tests in a package

    It could be helpful to provide a higher level functionality to managing a set of recorders in a given test package.

    • For example it'd be nice to inject a Recorder in a TestMain() so that all tests in a package making HTTP requests are recording/playing back requests.
    • Resulting in a httpfixtures/TestName.yml for each Test.* in the package automatically
    • Also we can allow for an env var to re-record tests when things change e.g. go-vcr.mode=record

    main_test.go

    
    
    func TestMain(m *testing.M) {
        var (
            code        = 0
            // recordMode can be specified via env var: "replay" (default) | "record" | "disabled"
            recorderMode = parseRecorderMode(os.Getenv("go-vcr.mode"))
        )
    
        testRecorder = recorder.NewTestRecorder(recorderMode, http.DefaultTransport)
    
        // inject recorder transport
        http.DefaultTransport = testRecorder.Transport()
    
        // Run package tests
        code = m.Run()
    
        // Stops the recorder and writes recorded HTTP files
        testRecorder.Stop()
        os.Exit(code)
    }
    
    func parseRecorderMode(s string) recorder.Mode {
        s = strings.ToLower(s)
        switch s {
        case "record", "recording":
            return recorder.ModeRecording
        case "replay", "replaying":
            return recorder.ModeReplaying
        case "disable", "disabled":
            return recorder.ModeDisabled
        default:
            return recorder.ModeReplaying
    }
    
    
    
    • The recorder.TestRecorder would manage a collection of recorder.Recorder mapped to each Test.* being run
      • We can use packages like runtime/debug to inspect stack trace for the running Test function
        • https://golang.org/pkg/runtime/debug/
    • I've found it necessary in test environments to provide additional HTTP transport to inject a X-TEST-SEQ and match the body to ensure POST requests that don't have unique request params (but result in different responses)
    // RoundTrip implements http round tripper
    func (l *TestRecorder) RoundTrip(req *http.Request) (*http.Response, error) {
    
        ctxt := l.GetContext()
        transport := ctxt.recorder.Transport
    
        // Don't append sequence numbers for GET requests as they should return the same information (POST etc may return new response data each time called e.g. /create)
        if req.Method == http.MethodGet {
            return transport.RoundTrip(req)
        }
    
        seq := ctxt.NextSequence()
        req.Header.Add("X-TEST-SEQ", strconv.Itoa(seq))
        return transport.RoundTrip(req)
    
     }
    
    httpRecorder.SetMatcher(func(r *http.Request, i cassette.Request) bool {
        if r.Method == http.MethodGet {
            return cassette.DefaultMatcher(r, i) && (reflect.DeepEqual(r.Header, i.Headers))
        }
    
        var b bytes.Buffer
        if _, err := b.ReadFrom(r.Body); err != nil {
            return false
        }
        r.Body = ioutil.NopCloser(&b)
        return cassette.DefaultMatcher(r, i) && (b.String() == "" || b.String() == i.Body) && (reflect.DeepEqual(r.Header, i.Headers))
    })
    
    opened by dougnukem 4
  • recorder.ModeReplaying not working as expected

    recorder.ModeReplaying not working as expected

    I was working on setting up testing for CI. I was hoping that if a cassette file was not present, the recorder would throw an error, which would prompt the developer to commit the missing file before their PR could be merged. This would let us run tests on CI while ensuring that no actual HTTP calls would occur.

    We'd set the mode through a utility function, but the pseudo code looks like this:

    import "github.com/dnaeon/go-vcr/v2/recorder"
    
    vcr, err := recorder.NewAsMode("foo", recorder.ModeReplaying, nil)
    if err != nil {
      log.Fatal(err)
    }
    // test using VCR
    vcr.Stop()
    

    No error occurred, and a new cassette file foo.yaml was created.

    It looks like the bug may be in: https://github.com/dnaeon/go-vcr/blob/a139137b0cf04933663cec479266a942f8a153ed/v2/recorder/recorder.go#L180-L186

    The mode is not ModeDisabled and no cassette file exists, so the mode is automatically reset to ModeRecording, then vcr.Stop() persists the cassette file, since the mode was changed to ModeRecording.

    opened by whitehat101 3
  • Upgrade gopkg.in/yaml.v2 dependency due to CVE-2019-11254

    Upgrade gopkg.in/yaml.v2 dependency due to CVE-2019-11254

    Hi,

    I would kindly like to ask whether you could upgrade your dependency to module gopkg.in/yaml.v2 from v2.2.1 to v2.2.8?

    There were improvements to the performance in specific cases, fixed by https://github.com/go-yaml/yaml/pull/555, and versions of yaml.v2 without that fix even have a CVE entry, although the description there is a little bit confusing, as it is mainly talking about the K8S API server...

    Still, updating from v2.2.1 to v2.2.8 should neither introduce compatibility issues nor other problems. I also did the update locally, and the test suite ran fine. I can also provide a PR that updates the go.mod file and contents of the vendor dir if you want.

    Thanks!

    opened by Blesmol 3
  • support response duration

    support response duration

    duration can be configured as a string that can be parsed by time.ParseDuration() (eg: '10ms`) to simulate the total network latency/tranfer and server processing time of the request.

    Signed-off-by: Xavier Coulon [email protected]

    opened by xcoulon 3
  • Avoid recursive CancelRequest. Fixes #23

    Avoid recursive CancelRequest. Fixes #23

    This code was posted originally on issue #23, but was not put on the project by anyone, so i'm sending it as a PR here.

    Because i'm testing go-vcr on a personal project, I need to have this method working, because for some reason when using go-vcr with a http.Client this method is called and causes the problem described on #23.

    Thanks.

    opened by fjorgemota 3
  • Verify HTTP request was issued

    Verify HTTP request was issued

    Is there a way to verify that each recorded request is reproduced during the playing phase?

    The use case:

    1. I write a test and record the http requests.
    2. Re-run the test and it passes
    3. I refactor the code and introduce a bug which prevents issuing a specific request.
    4. Re-run the test, it passes. It shouldn't.
    opened by diegosanchez 1
  • add filter after reading the cassette

    add filter after reading the cassette

    sometime we wants to modify the recorded values dynamically.

    for example some 3rd party library generates a requestID on the HTTP header and verify it when it got the response back.

    for example maybe a ReplyFilter can be added to give the caller a chance to modify the request or response

    opened by yulrizka 2
  • How to use go-vcr with

    How to use go-vcr with "github.com/satori/go.uuid" getting back Nil values for uuid

    When I use go-vcr with structs containing uuid.UUID github.com/satori/go.uuid I get back no values. Testing other structs without uuid.UUID types works as expected. Any advice is appreciated.

    Thank you!

    Testing with Ginkgo and Gomega (ommitted test code for brevity)

    Go version: go version go1.12 darwin/amd64

    type Test2 struct {
    	ID          uuid.UUID `json:"id"`
    	Name        string    `json:"name"`
    	Description string    `json:"description"`
    }
    
    func (s *UserService) Test2() *Test2 {
    	url := fmt.Sprintf("%v:%v/test", s.ServiceHost, s.ServicePort)
    	var client = &http.Client{Timeout: 10 * time.Second}
    	req, err := http.NewRequest("GET", url, nil)
    	req.Header.Add("Content-Type", "application/json")
    	req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", s.ServiceToken))
    	resp, err := client.Do(req)
    	if err != nil {
    		return nil
    	}
    	defer resp.Body.Close()
    
    	var t Test2
    	json.NewDecoder(resp.Body).Decode(&t)
    	return &t
    }
    

    Fixture:

    ---
    version: 1
    interactions:
    - request:
        body: ""
        form: {}
        headers:
          Authorization:
          - Bearer 123abc
          Content-Type:
          - application/json
        url: http://localhost:8080/test2
        method: GET
      response:
        body: |
          {"id":"5600966e-4ad6-4f79-8e01-8db13ba5c212","name":"Foo","description":"Bar"}
        headers:
          Content-Length:
          - "79"
          Content-Type:
          - application/json
          Date:
          - Fri, 19 Apr 2019 18:49:03 GMT
        status: 200 OK
        code: 200
        duration: ""
    
    

    Test code:

    	us := UserService{
    			ServiceHost:  "http://localhost",
    			ServicePort:  8080,
    			ServiceToken: "123abc",
    	}
    	url = fmt.Sprintf("%v:%v/test", us.ServiceHost, us.ServicePort)
    	req, err = http.NewRequest("GET", url, nil)
    	Expect(err).To(BeNil())
    	req.Header.Add("Content-Type", "application/json")
    	req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", us.ServiceToken))
    	resp, err = client.Do(req)
    	Expect(err).To(BeNil())
    	defer resp.Body.Close()
    	t := us.Test()
    	Expect(t.Description).To(Equal("Bar"))
    
    	r, err = recorder.New("fixtures/test2")
    	Expect(err).To(BeNil())
    	defer r.Stop()
    
    	client = &http.Client{
    	   Transport: r,
    	}
    
    	url = fmt.Sprintf("%v:%v/test2", us.ServiceHost, us.ServicePort)		
    	req, err = http.NewRequest("GET", url, nil)
    	Expect(err).To(BeNil())
    	req.Header.Add("Content-Type", "application/json")
    	req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", us.ServiceToken))
    	resp, err = client.Do(req)
    	Expect(err).To(BeNil())
    	defer resp.Body.Close()
    	t2 := us.Test2()
    
           iDString := "5600966e-4ad6-4f79-8e01-8db13ba5c212"
    	id, err := uuid.FromString(iDString)
    	Expect(err).To(BeNil())
    	Expect(t2.ID).To(Equal(id))
    

    Fails with

       Expected
              <uuid.UUID>: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
          to equal
              <uuid.UUID>: [86, 0, 150, 110, 74, 214, 79, 121, 142, 1, 141, 177, 59, 165, 194, 18]
    
    opened by lauraeci 1
  • Throw error on realHttpTransport connection

    Throw error on realHttpTransport connection

    Is there a way to throw an error when an HTTP connection is made on realHttpTransport during ModeReplay. I imagine this is up to our own implementation of realHttpTransport - is there a recipe we could include in the readme.md? I find this a super helpful feature of Ruby VCR.

    Thanks.

    opened by erichulburd 2
  • allowing matcher to see the order of requests

    allowing matcher to see the order of requests

    i just hit a scenario where having the order of the requests is more useful than complex matcher.

    An easy change to the matcher function adding the position in the array of the requests, allow you to check that the order of the requests is correct and use a simple matcher.

    If you are interested in this I can arrange a pull request quite easly.

    opened by spaghetty 3
Releases(v2.0.1)
  • v2.0.1(Jun 24, 2021)

    This release contains a breaking change. When upgrading to v2.0.0 make sure that you re-create your cassettes.

    A new flag of the recorder.Recorder (the SkipRequestLatency one) is supported which allows to skip latency simulation during replay.

    See https://github.com/dnaeon/go-vcr/pull/63 for more details.

    Source code(tar.gz)
    Source code(zip)
Owner
Marin Atanasov Nikolov
Marin Atanasov Nikolov
Rr-e2e-tests - Roadrunner end-to-end tests repository

RoadRunner end-to-end plugins tests License: The MIT License (MIT). Please see L

RoadRunner 1 Jan 21, 2022
Sql mock driver for golang to test database interactions

Sql driver mock for Golang sqlmock is a mock library implementing sql/driver. Which has one and only purpose - to simulate any sql driver behavior in

DATA-DOG 4.6k Aug 15, 2022
A Go implementation of Servirtium, a library that helps test interactions with APIs.

Servirtium is a server that serves as a man-in-the-middle: it processes incoming requests, forwards them to a destination API and writes the response into a Markdown file with a special format that is common across all of the implementations of the library.

Servirtium 6 Jun 16, 2022
Go-interactions - Easy slash commands for Arikawa

go-interactions A library that aims to make dealing with discord's slash command

null 7 May 26, 2022
Fortio load testing library, command line tool, advanced echo server and web UI in go (golang). Allows to specify a set query-per-second load and record latency histograms and other useful stats.

Fortio Fortio (Φορτίο) started as, and is, Istio's load testing tool and now graduated to be its own project. Fortio is also used by, among others, Me

Fortio (Φορτίο) 2.6k Aug 9, 2022
Extremely flexible golang deep comparison, extends the go testing package and tests HTTP APIs

go-testdeep Extremely flexible golang deep comparison, extends the go testing package. Latest news Synopsis Description Installation Functions Availab

Maxime Soulé 313 Aug 8, 2022
A simple and expressive HTTP server mocking library for end-to-end tests in Go.

mockhttp A simple and expressive HTTP server mocking library for end-to-end tests in Go. Installation go get -d github.com/americanas-go/mockhttp Exa

Americanas Go 6 Dec 19, 2021
A next-generation testing tool. Orion provides a powerful DSL to write and automate your acceptance tests

Orion is born to change the way we implement our acceptance tests. It takes advantage of HCL from Hashicorp t o provide a simple DSL to write the acceptance tests.

Wesovi Labs 44 Aug 8, 2022
Automatically update your Go tests

autogold - automatically update your Go tests autogold makes go test -update automatically update your Go tests (golden files and Go values in e.g. fo

Hexops 123 Jun 30, 2022
Terratest is a Go library that makes it easier to write automated tests for your infrastructure code.

Terratest is a Go library that makes it easier to write automated tests for your infrastructure code. It provides a variety of helper functions and patterns for common infrastructure testing tasks,

Gruntwork 6.3k Aug 15, 2022
Testing framework for Go. Allows writing self-documenting tests/specifications, and executes them concurrently and safely isolated. [UNMAINTAINED]

GoSpec GoSpec is a BDD-style testing framework for the Go programming language. It allows writing self-documenting tests/specs, and executes them in p

Esko Luontola 112 Apr 5, 2022
Robust framework for running complex workload scenarios in isolation, using Go; for integration, e2e tests, benchmarks and more! 💪

e2e Go Module providing robust framework for running complex workload scenarios in isolation, using Go and Docker. For integration, e2e tests, benchma

null 106 Aug 7, 2022
How we can run unit tests in parallel mode with failpoint injection taking effect and without injection race

This is a simple demo to show how we can run unit tests in parallel mode with failpoint injection taking effect and without injection race. The basic

amyangfei 1 Oct 31, 2021
Snapshot - snapshot provides a set of utility functions for creating and loading snapshot files for using snapshot tests.

Snapshot - snapshot provides a set of utility functions for creating and loading snapshot files for using snapshot tests.

Daniel J. Rollins 2 Jan 27, 2022
Package for comparing Go values in tests

Package for equality of Go values This package is intended to be a more powerful and safer alternative to reflect.DeepEqual for comparing whether two

Google 3k Aug 8, 2022
Go testing in the browser. Integrates with `go test`. Write behavioral tests in Go.

GoConvey is awesome Go testing Welcome to GoConvey, a yummy Go testing tool for gophers. Works with go test. Use it in the terminal or browser accordi

SmartyStreets 7.4k Aug 12, 2022
Ruby on Rails like test fixtures for Go. Write tests against a real database

testfixtures Warning: this package will wipe the database data before loading the fixtures! It is supposed to be used on a test database. Please, doub

null 830 Aug 11, 2022
A simple `fs.FS` implementation to be used inside tests.

testfs A simple fs.FS which is contained in a test (using testing.TB's TempDir()) and with a few helper methods. PS: This lib only works on Go 1.16+.

Carlos Alexandro Becker 31 Mar 3, 2022
Go testing in the browser. Integrates with `go test`. Write behavioral tests in Go.

GoConvey is awesome Go testing Welcome to GoConvey, a yummy Go testing tool for gophers. Works with go test. Use it in the terminal or browser accordi

SmartyStreets 7.4k Aug 7, 2022