Markdown based document-driven RESTful API testing.

Overview

Silk logo

silk Build Status Go Report Card

Markdown based document-driven web API testing.

Learn more

Video of Mat Ryer speaking about Silk

(VIDEO) Watch the talk about Silk (with slides) or read about Silk in this blog post.

Example Silk test file

Markdown API

Tests are made up of documents written in Markdown.

  • # Group - Top level headings represent groups of requests
  • ## GET /path - Second level headings represent a request
  • Code blocks with three back tics represent bodies
  • * Field: value - Lists describe headers and assertions
  • * ?param=value - Request parameters
  • --- seperators break requests from responses
  • Comments (starting with //) allow you to capture variables
  • Plain text is ignored to allow you to add documentation
  • Inline back tics are ignored and are available for formatting

Document structure

A document is made up of:

  • A request
  • --- seperator
  • Assertions

Requests

A request starts with ## and must have an HTTP method, and a path:

## METHOD /path

Examples include:

## GET /people

## POST /people/1/comments

## DELETE /people/1/comments/2

Request body (optional)

To specify a request body (for example for POST requests) use a codeblock using backtics (```):

```
{"name": "Silk", "release_year": 2016}
```

Request headers (optional)

You may specify request headers using lists (prefixed with *):

* Content-Type: "application/json"
* X-Custom-Header: "123"

Request parameters (optional)

Adding parameters to the path (like GET /path?q=something) can be tricky, especially when you consider escaping etc. To address this, Silk supports parameters like lists:

* ?param=value

The parameters will be correctly added to the URL path before the request is made.

Cookies

Setting cookies on a request can be done using the HTTP header pattern:

* Cookie: "key=value"

Assertions

Following the --- separator, you can specify assertions about the response. At a minimum, it is recommended that you assert the status code to ensure the request succeeded:

* Status: 200

You may also specify response headers in the same format as request headers:

* Content-Type: "application/json"
* X-MyServer-Version: "v1.0"

If any of the headers do not match, the test will fail.

Capturing data

Silk allows you to capture values at the point of asserting them and reuse them in future requests and assertions. To capture a value, include a comment on the line that mentions a {placeholder}:

* Data.UserID: /.*/ // The user's unique {id}.

The value from UserID (e.g. 123) will be stored in a variable called id, and you can refer to it later:

## GET /users/{id}

The above would be a request to GET /users/123.

  • Captured values are only available when assertions are successful

Environment variables

You can access environment variables inside Silk tests using the {$NAME} format, where NAME is the environment name.

Asserting cookies

To assert that a cookie is present in a response, make a regex assertion against the Set-Cookie HTTP header:

* Set-Cookie: /key=value/
  • All cookie strings are present in a single Set-Cookie seperated by a pipe character.

Validating data

You can optionally include a verbatim body using code blocks surrounded by three back tics. If the response body does not exactly match, the test will fail:

```
Hello world!
```

You can flag expected response bodies as json directly after the three back tics. This will assert that the actual response contains the same value for each expected key (recursively) allowing for differences in whitespace and ordering as well as being lenient towards additional (unexpected) keys in the response.

```json
{
    "id": 1,
    "release_year": 2016,
    "name": "Silk"
}
```

You can use the flag json(strict) to enforce that no additional fields may be present while still allowing for differences in whitespace and key order.

You may also make any number of regex assertions against the body using the Body object:

* Body: /Hello world/
* Body: /This should be found too/
* Body: /and this/

Alternatively, you can specify a list (using *) of data fields to assert accessible via the Data object:

* Status: 201
* Content-Type: "application/json"
* Data.name: "Silk"
* Data.release_year: 2016
* Data.tags[0]: "testing"
* Data.tags[1]: "markdown"
* Data[0].name: "Mat"
* Data[1].name: "David"
  • NOTE: Currenly this feature is only supported for JSON APIs.

Regex

Values may be regex, if they begin and end with a forward slash: /. The assertion will pass if the value (after being turned into a string) matches the regex.

* Status: /^2.{2}$/
* Content-Type: /application/json/

The above will assert that:

  • The status looks like 2xx, and
  • The Content-Type contains application/json

Command line

The silk command runs tests against an HTTP endpoint.

Usage:

silk -silk.url="{endpoint}" {testfiles...}
  • {endpoint} the endpoint URL (e.g. http://localhost:8080)
  • {testfiles} list of test files (e.g. ./testfiles/one.silk.md ./testfiles/two.silk.md)

Notes:

  • Omit trailing slash from endpoint
  • {testfiles} can include a pattern (e.g. /path/*.silk.md) as this is expended by most terminals to a list of matching files

Golang

Silk is written in Go and integrates seamlessly into existing testing tools and frameworks. Import the runner package and use RunGlob to match many test files:

package project_test

import (
  "testing"
  "github.com/matryer/silk/runner"
)

func TestAPIEndpoint(t *testing.T) {
  // start a server
  s := httptest.NewServer(yourHandler)
  defer s.Close()

  // run all test files
  runner.New(t, s.URL).RunGlob(filepath.Glob("../testfiles/failure/*.silk.md"))
}

Credit

  • Special thanks to @dahernan for his contributions and criticisms of Silk
  • Silk logo by Chris Ryer
Comments
  • make previous data available with special format

    make previous data available with special format

    If I create a thing:

       ## POST /item
    

    and it creates an ID for me:

         * Data.id = /.?/
    

    I might need to be able to use that ID in another request later:

       ## GET /item/{id}
    
    test please 
    opened by matryer 26
  • Problem with expected JSON

    Problem with expected JSON

    Hi, First of all I'd like to thank you for such a useful library. It's both simple to use and pretty powerful. I want to report an issue I've been facing using Silk. I've discovered it adding a property to the JSON I'm expecting in the response. Here is the JSON which works:

    {
        "is_complete": true,
        "user": {
            "id": 1,
            "name": "Juan Carlos",
            "email": "[email protected]",
            "phone": "541122334455",
            "gender": "male",
            "gender_of_interest": "women",
            "birthday": "1990-02-15T00:00:00Z",
            "height": 170,
            "job_title": "Software Engineer",
            "company": "Whim",
            "school": "UADE",
            "degree": {
                "id": 3,
                "description": "Bachelor's"
            },
            "descriptors": [
                {
                    "id": 5,
                    "description": "Sarcastic"
                },
                {
                    "id": 10,
                    "description": "Night owl"
                }
            ],
            "created_at": "2016-02-26T14:00:00Z",
            "updated_at": "2016-03-16T12:00:00Z"
        }
    }
    

    And here is the JSON which breaks my tests (notice the ethnicities array was added):

    {
        "is_complete": true,
        "user": {
            "id": 1,
            "name": "Juan Carlos",
            "email": "[email protected]",
            "phone": "541122334455",
            "gender": "male",
            "gender_of_interest": "women",
            "birthday": "1990-02-15T00:00:00Z",
            "height": 170,
            "job_title": "Software Engineer",
            "company": "Whim",
            "school": "UADE",
            "degree": {
                "id": 3,
                "description": "Bachelor's"
            },
            "descriptors": [
                {
                    "id": 5,
                    "description": "Sarcastic"
                },
                {
                    "id": 10,
                    "description": "Night owl"
                }
            ],
            "ethnicities": [
                {
                    "id": 1,
                    "description": "Asian"
                },
                {
                    "id": 3,
                    "description": "Latino"
                }
            ],
            "created_at": "2016-02-26T14:00:00Z",
            "updated_at": "2016-03-16T12:00:00Z"
        }
    }
    

    The error that I get is the following:

    [GIN] 2016/05/04 - 15:46:38 | 404 |       2.599µs | 127.0.0.1 |   iJS     /
    body expected:  
    
    {
        "is_complete": true,
        "user": {
            "id": 1,
            "name": "Juan Carlos",
            "email": "[email protected]",
            "phone": "541122334455",
            "gender": "male",
            "gender_of_interest": "women",
            "birthday": "1990-02-15T00:00:00Z",
            "height": 170,
            "job_title": "Software Engineer",
            "company": "Whim",
            "school": "UADE",
            "degree": {
                "id": 3,
                "description": "Bachelor's"
            },
            "descriptors": [
                {
                    "id": 5,
                    "description": "Sarcastic"
                },
                {
                    "id": 10,
                    "description": "Night owl"
                }
            ],
            "ethnicities": [
                {
                    "id": 1,
                    "description": "Asian"
                },
                {
                    "id": 3,
                    "description": "Latino"
                }
            ],
            "created_at": "2016-02-26T14:00:00Z",
            "updated_at": "2016-03-16T12:00:00Z"
        }
    }  
    
    
    actual:  
    
    404 page not found  
    
    --- FAIL: iJS zUxMiIsInR5c 
     users.md:22 - body doesn't match  
    --- FAIL: TestUsers (0.00s)
    FAIL
    coverage: 0.0% of statements
    

    Thanks in advance. JC

    opened by ivancevich 13
  • Improve handling of json bodies during validation

    Improve handling of json bodies during validation

    This uses the type annotation in the expected body to perform a deepEqual comparison on the deserialized JSON.

    This PR implements json (defaults to mode=same) and json(mode=subset) as discussed in #25.

    I did not have the time yet to add appropriate unit tests, but will happily do so before merging this PR.

    opened by felixLam 12
  • better body assertions: Specify kind of assertion to make

    better body assertions: Specify kind of assertion to make

    Proposal: better bodies in Silk

    Bodies in Silk could make use of the 'formatting' tag thing where not only the data type is specified, but also the kind of assertions that could take place:

    ``json(mode=exact)
    {
      "name": "Silk",
      "released": 2016
    }
    ``
    
    • Two back tics are used so as not to confuse renderer
    • This assumes that the text after the opening code block line is never rendered (needs validating)

    The types of bodies could be:

    • exact - the body must match verbatim
    • same - the JSON must essentially be equal but fields may be in different orders and formatting (whitespace) is ignored
    • subset - each mentioned field is asserted, additional fields in the body is ignored

    The syntax highlight language specified could also drive which ParseBody function is called.

    Cons

    • When rendered, you can't see the kind of assertion, but that might not be too much of a problem and could always be explained in the plain text surrounding the block.

    Alternatives

    Other options to consider:

    • the json language specifier might be enough to indicate the datatype, and we could use flags (when running the file) to indicate what type of assertion to make
    help wanted 
    opened by matryer 12
  • Markdown file length or assertions limit?

    Markdown file length or assertions limit?

    Is there any length limit for the .md file? Or any limitations in the number of asserts?

    I am having some test failures due to .md parsing. invalid request: net/http: invalid method "[email protected]" ** This is not what the .md files contains.

    I can reproduce it by replicating a simple test that pass in the same file for several times. When I try to run exactly the same test for 6-7 times.

    Any known issue?

    Thanks

    opened by rafavisa 9
  • add a format 'len(Data): int' to count elements in a slice

    add a format 'len(Data): int' to count elements in a slice

    Add a new format to count elements in a response slice.

    This would be useful when you expect an exact amount of data to retrieve.

    Usage:

    ## Get /foo/total
    Get 10 items
    === 
    len(Data): 10
    
    opened by craftgear 9
  • Basic usage information

    Basic usage information

    Hello, I just cloned the repo. But it is impossible to run anything at all from the files in the repo:

    silk -silk.url="{endpoint}" {testfiles...}

    as suggested in the README produces "command not found"

    Executing build.sh results in "main.go:8:2: cannot find package "github.com/matryer/silk/runner""

    What is the correct procedure for running the application? Could you adjust the documentation accordingly?

    opened by jamesbates 5
  • Postman export support

    Postman export support

    Hey found silk quite interesting! I'm looking for to give a try in my next microservice project.

    By the way while reading i was thinking in postman as far i know several people using for QA and test (not automated) purpose.

    So i was think if make sense to add support for that in this project, if is that something that you are willing for to have into core, or might be some a part, a command that we run pointing the markdown that generates that?

    I'm asking because i was thinking in give a try and implement that. :beers:

    question 
    opened by krolow 5
  • Blueprint compatible? Differences?

    Blueprint compatible? Differences?

    Just curious to what extent silk's syntax and conventions are similar and/or compatible with blueprint? I ask, because blueprint is a pretty established standard for documenting web APIs (along w/ Swagger and RAML, all of which are interconvertible) and there's already a lot of nice tooling around it (e.g., test2doc).

    If different, is it possible to convert silk-based api docs into blueprints or other api-spec formats?

    question 
    opened by joyrexus 3
  • ``clean()`` makes error messages confusing when you expect strings with facing and/or trailing spaces

    ``clean()`` makes error messages confusing when you expect strings with facing and/or trailing spaces

    When you have actual data strings with facing and/or trailing spaces, error messages are confusing.

    For example an actual data is " expected" then error message is:

    Data[9].title expected string: "expected"  actual string: "expected"  
    

    In this case actual string should be shown with a facing space.

    clean() function causes this. https://github.com/matryer/silk/blob/87124e93cc78cbee0f495245e5eace81f3ad7b08/parse/detail.go#L41

    Are these Trim callings really necessary?

    question 
    opened by craftgear 3
  • Missing package when building : `github.com/matryer/m`

    Missing package when building : `github.com/matryer/m`

    Hi Mat,

    i'm new to go so this may certainly be a problem between the chair and the keyboard, but when i try to go install github.com/matryer/silk from the freshly cloned repo, i get this error :

    silk/runner/run.go:17:2: cannot find package "github.com/matryer/m" in any of:
        /usr/local/Cellar/go/1.7.1/libexec/src/github.com/matryer/m (from $GOROOT)
        /Users/sgaide/dev/go/src/github.com/matryer/m (from $GOPATH)
    

    i can find no m package in your github repos either, am i missing something ?

    thanks for any help,

    Sebastien

    opened by sgaide 2
  • Dynamic request variables?

    Dynamic request variables?

    Hello.

    Couldn't find any information about this feature. Is it possible to define dynamic variables (i.e. random generated ones) in a request?

    Thanks.

    opened by importnil 0
  • x509: certificate signed by unknown authority

    x509: certificate signed by unknown authority

    When I'm running silk against certificate signed by untrusted root, I got the following error.

    > silk -silk.url="https://untrusted-root.badssl.com/" ./**/*.silk.md
    silk: running 1 file(s)...
    x509: certificate signed by unknown authority
    --- FAIL: silk (0.67s)
    FAIL
    

    I tried to find a way to skip certification checking in help but couldn't find it. (like option -k in curl)

     -k, --insecure      Allow connections to SSL sites without certs (H)
    

    I think the error was caused by http.DefaultTransport used in Runner.DoRequest. It seems to me that DefaultTransport does not allow a connection to insecure SSL sites.

    https://github.com/matryer/silk/blob/11c4f5465bf617bd7ff879e7e243c9180ca35528/runner/run.go#L56

    I found a way to skip it here https://groups.google.com/d/msg/golang-nuts/v5ShM8R7Tdc/I2wyTy1o118J

    I'm thinking of adding another option in silk to allow connections to SSL sites without certs. Any idea what the option should be? Is -k ok?

    opened by juacompe 0
  • remove unnecessary validation

    remove unnecessary validation

    Hi @matryer I've found out that the following JSON schema never gets successfully matched.

    {
        "name": "JC",
        "foo": {
            "bar": null,
            "baz": "qux"
        }
    }
    

    I've removed a condition which seems to be the problem, and also seems useless, since the first condition after recursion takes care of nil values. Let me know what you think. Thanks!

    opened by ivancevich 0
  • Body assertions based on JSON Schema definitions

    Body assertions based on JSON Schema definitions

    It would be nice to have the ability to assert response bodies based on embedded JSONSchema definitions in the same Markdown document or linking to the file system file definition.

    opened by tomas-fp 0
  • GraphQL API

    GraphQL API

    Would be nice if you can test a Graphql api backend using silk. Technically it would be possible but might look ugly most of the time. I was using cucumber for the tests but the idea of converting tests to documentation seems really inspiring.

    question 
    opened by pyrossh 1
Releases(0.5.1)
Owner
Mat Ryer
pace.dev and firesearch.dev - Gopher, developer, speaker, author (Go Programming Blueprints) - Xbar (BitBar reboot) xbarapp.com (now in beta)
Mat Ryer
A Go library help testing your RESTful API application

RESTit A Go micro-framework to help writing RESTful API integration test Package RESTit provides helps to those who want to write an integration test

RESTit 55 Oct 28, 2022
testcase is an opinionated behavior-driven-testing library

Table of Contents testcase Guide Official API Documentation Getting Started / Example Modules Summary DRY Modularization Stability Case Study About te

Adam Luzsi 104 Nov 10, 2022
siusiu (suite-suite harmonics) a suite used to manage the suite, designed to free penetration testing engineers from learning and using various security tools, reducing the time and effort spent by penetration testing engineers on installing tools, remembering how to use tools.

siusiu (suite-suite harmonics) a suite used to manage the suite, designed to free penetration testing engineers from learning and using various security tools, reducing the time and effort spent by penetration testing engineers on installing tools, remembering how to use tools.

Re 288 Nov 10, 2022
This repository includes consumer driven contract test for provider, unit test and counter api.

This repository includes consumer driven contract test for provider, unit test and counter api.

Ahmet Zümberoğlu 0 Feb 1, 2022
Learn Go with test-driven development

Learn Go with Tests Art by Denise Formats Gitbook EPUB or PDF Translations 中文 Português 日本語 한국어 Support me I am proud to offer this resource for free,

Chris James 18.8k Nov 27, 2022
Simple test driven approach in "GOLANG"

Testing in GOLANG Usage Only test go test -v Coverage go test -cover or go test -coverprofile=coverage.out go tool cover -html=coverage.out Benchmark

mahdi imani 0 Dec 5, 2021
Behaviour Driven Development tests generator for Golang

gherkingen It's a Behaviour Driven Development (BDD) tests generator for Golang. It accept a *.feature Cucumber/Gherkin file and generates a test boil

Maxim Krivchun 50 Sep 27, 2022
Gherkin-based BDD testing for go

gocuke ?? gocuke is a Gherkin-based BDD testing library for golang. Features tight integration with *testing.T (use any standard assertion library or

Regen Network 6 Jun 22, 2022
ESME is a go library that allows you to mock a RESTful service by defining the configuration in json format

ESME is a go library that allows you to mock a RESTful service by defining the configuration in json format. This service can then simply be consumed by any client to get the expected response.

Sumit Tokkar 3 Mar 2, 2021
Expressive end-to-end HTTP API testing made easy in Go

baloo Expressive and versatile end-to-end HTTP API testing made easy in Go (golang), built on top of gentleman HTTP client toolkit. Take a look to the

Tom 746 Nov 19, 2022
API testing framework inspired by frisby-js

frisby REST API testing framework inspired by frisby-js, written in Go Proposals I'm starting to work on frisby again with the following ideas: Read s

_Hofstadter 274 Sep 27, 2022
Testing API Handler written in Golang.

Gofight API Handler Testing for Golang Web framework. Support Framework Http Handler Golang package http provides HTTP client and server implementatio

Bo-Yi Wu 428 Nov 18, 2022
End-to-end HTTP and REST API testing for Go.

httpexpect Concise, declarative, and easy to use end-to-end HTTP and REST API testing for Go (golang). Basically, httpexpect is a set of chainable bui

Victor Gaydov 2k Nov 24, 2022
Hsuan-Fuzz: REST API Fuzzing by Coverage Level Guided Blackbox Testing

Hsuan-Fuzz: REST API Fuzzing by Coverage Level Guided Blackbox Testing Architecture Usage package main import ( restAPI "github.com/iasthc/hsuan-

Chung-Hsuan Tsai 21 Jul 13, 2022
mock server to aid testing the jaguar-java client API

stripe-mock stripe-mock is a mock HTTP server that responds like the real Stripe API. It can be used instead of Stripe's test mode to make test suites

knr.data 1 Dec 24, 2021
Terminal application used for API testing

Easily create, manage and execute http requests from the terminal.

PierreKieffer 64 Nov 9, 2022
Testy is a Go test running framework designed for Gametime's API testing needs.

template_library import "github.com/gametimesf/template_library" Overview Index Overview Package template_library is a template repository for buildin

Gametime United, Inc. 4 Jun 21, 2022
Fortio load testing library, command line tool, advanced echo server and web UI in go (golang). Allows to specify a set query-per-second load and record latency histograms and other useful stats.

Fortio Fortio (Φορτίο) started as, and is, Istio's load testing tool and now graduated to be its own project. Fortio is also used by, among others, Me

Fortio (Φορτίο) 2.7k Nov 17, 2022
:exclamation:Basic Assertion Library used along side native go testing, with building blocks for custom assertions

Package assert Package assert is a Basic Assertion library used along side native go testing Installation Use go get. go get github.com/go-playground/

Go Playgound 49 Nov 4, 2022