Mock all the services. Intuitive YAML DSL for HTTP, gRPC, Kafka, and AMQP mocks.

Overview

OpenMock

OpenMock is a Go service that can mock services in integration tests, staging environment, or anywhere. The goal is to simplify the process of writing mocks in various channels. Currently it supports the following channels:

  • HTTP
  • gRPC
  • Kafka
  • AMQP (e.g. RabbitMQ)

Usage

Use it with docker.

$ docker run -it -p 9999:9999 -v $(pwd)/demo_templates:/data/templates checkr/openmock 

More complete openmock instance (e.g. redis) with docker-compose.

$ docker-compose up

Test it.

$ curl localhost:9999/ping

Dependencies.

  • HTTP (native supported, thanks to https://echo.labstack.com/)
    • One can configure HTTP port, set env OPENMOCK_HTTP_PORT=80
  • GRPC (supported through through HTTP/2 interface)
    • One can configure GRPC port, set env OPENMOCK_GRPC_PORT=50051
  • Kafka (optional)
    • To enable mocking kafka, set env OPENMOCK_KAFKA_ENABLED=true.
    • One can also config the following kafka parameters, optionally with separate config for consumers and producers. For example OPENMOCK_KAFKA_SEED_BROKERS, OPENMOCK_KAFKA_PRODUCER_SEED_BROKERS, and OPENMOCK_KAFKA_CONSUMER_SEED_BROKERS
      • OPENMOCK_KAFKA_SEED_BROKERS
      • OPENMOCK_KAFKA_SASL_USERNAME
      • OPENMOCK_KAFKA_SASL_PASSWORD
      • OPENMOCK_KAFKA_TLS_ENABLED
  • AMQP (optional)
    • To enable mocking amqp, set env OPENMOCK_AMQP_ENABLED=true
    • One can also config OPENMOCK_AMQP_URL.
  • NPM (development only)
    • Used in Makefile during swagger admin API server generation

OpenMock Templates

Templates are YAML files that describe the behavior of OpenMock.

Templates Directory

You can put any number of .yaml or .yml files in a directory, and then point environment variable OPENMOCK_TEMPLATES_DIR to it. OpenMock will recursively (including subdirectories) load all the YAML files. For example:

# OPENMOCK_TEMPLATES_DIR=./demo_templates

./demo_templates
├── amqp.yaml
├── files
│   └── colors.json
├── http.yaml
├── jsonrpc.yaml
├── kafka.yaml
└── payload_from_file.yaml

Schema

OpenMock is configured a list of behaviors for it to follow. Each behavior is identified by a key, and a kind:

- key: respond-to-resource
  kind: Behavior

Expect

It represents the channel to listen on and condition for the actions of the behavior to be performed. Available channels are:

  • http
  • kafka
  • amqp
  • grpc

For example, under what condition and from what channel should we proceed with the actions.

- key: no-op
  kind: Behavior
  expect:
    # Condition checks if we need to do the actions or not
    # It only proceeds if it evaluates to "true"
    condition: '{{.HTTPHeader.Get "X-Token" | eq "t1234"}}'
    # Use one (and only one) of the following channels - [http, kafka, amqp]
    http:
      method: GET
      path: /ping
    kafka:
      topic: hello_kafka_in
    amqp:
      exchange: exchange_1
      routing_key: key_in
      queue: key_in

Actions

Actions are a series of functions to run. Availabe actions are:

  • publish_amqp
  • publish_kafka
  • redis
  • reply_http
  • send_http
  • reply_grpc
  • sleep
- key: every-op
  kind: Behavior
  expect:
    http:
      method: GET
      path: /ping
  actions:
    - publish_kafka:
        topic: hello_kafka_out
        payload: >
          {
            "kafka": "OK",
            "data": {}
          }
    - sleep:
        duration: 1s
    - reply_http:
        status_code: 200
        body: OK
        headers:
          Content-Type: text/html

The actions by default run in the order defined in the mock file; you can adjust this by adding an int 'order' value from lowest to highest number. The default value for 'order' is 0.

- key: every-op
  kind: Behavior
  expect:
    http:
      method: GET
      path: /ping
  actions:
    - publish_kafka:
        topic: hello_kafka_out
        payload: >
          {
            "kafka": "OK",
            "data": {}
          }
    - sleep:
        duration: 1s
      # sleep first
      order: -1000

Templates

Templates can be useful to assemble your payloads from parts

- key: dog
  kind: Template
  template: >
    <animal>dog</animal>

- key: cat
  kind: Template
  template: >
    <animal>cat</animal>

# $ curl 0:9999/fred
# <human>   <name>fred</name>   <pets>     <animal>dog</animal>      <animal>cat</animal>    </pets> </human>
- key: get-freds-pets
  kind: Behavior
  expect:
    http:
      method: GET
      path: /fred
  actions:
    - reply_http:
        status_code: 200
        body: >
          <human>
            <name>fred</name>
            <pets>
              {{template "dog"}}
              {{template "cat"}}
            </pets>
          </human>

Abstract Behaviors

Abstract Behaviors can be used to parameterize some data.

When an abstract behavior and a behavior extending it both have actions defined, all of them are run when the behavior matches. Actions will run from lowest to highest value of the 'order' field; if this is the same for two actions the action defined earlier in the abstract behavior runs first, followed by actions in the concrete behavior. Be aware that values with all digits will be interpreted into int type (YAML syntax), and it will fail the condition check given that some helper functions are returning string types. Pipe to toString before the comparison or alternatively put quotes around the values. See example in abstract_behaviors.yml.

- key: fruit-of-the-day
  kind: AbstractBehavior
  values:
    fruit: potato
  expect:
    condition: '{{.HTTPQueryString | contains .Values.day}}'
    http:
      method: GET
      path: /fruit-of-the-day
  actions:
    - reply_http:
        status_code: 200
        body: '{"fruit": "{{.Values.fruit}}"}'

# $ curl 0:9999/fruit-of-the-day?day=monday
# {"fruit": "apple"}
- key: monday-fruit
  kind: Behavior
  extend: fruit-of-the-day
  values:
    day: monday
    fruit: apple

# $ curl 0:9999/fruit-of-the-day?day=tuesday
# {"fruit": "potato"}
- key: tuesday-fruit
  kind: Behavior
  extend: fruit-of-the-day
  values:
    day: tuesday
  actions: 
    # sleep then reply_http
    - sleep:
         duration: 1s
      order: -1000

Dynamic templating

OpenMock leverages https://golang.org/pkg/text/template/ to write dynamic templates. Specifically, it supports a lot of Context and Helper Functions.

  • Usage of {{ expr }}. One can put {{ expr }} inside three types of places:
    • expect.condition
    • action.http.body, action.grpc.payload, action.kafka.payload, action.amqp.payload
    • action.http.body_from_file, action.grpc.payload_from_file, action.kafka.payload_from_file, action.amqp.payload_from_file ({{ expr }} will be in the file)
  • Use Context inside {{ expr }}.
    .HTTPHeader      # type: http.Header; example: {{.HTTPHeader.Get "X-Token"}}
    .HTTPBody        # type: string;      example: {{.HTTPBody}}
    .HTTPPath        # type: string;      example: {{.HTTPPath}}
    .HTTPQueryString # type: string;      example: {{.HTTPQueryString}}
    
    .GRPCHeader      # type: string;      example: {{.GRPCHeader}}
    .GRPCPayload     # type: string;      example: {{.GRPCPayload}}
    .GRPCService     # type: string;      example: {{.GRPCService}}
    .GRPCMethod      # type: string;      example: {{.GRPCMethod}}
    
    .KafkaTopic      # type: string;      example: {{.KafkaTopic}}
    .KafkaPayload    # type: string;      example: {{.KafkaPayload}}
    
    .AMQPExchange    # type: string;      example: {{.AMQPExchange}}
    .AMQPRoutingKey  # type: string;      example: {{.AMQPRoutingKey}}
    .AMQPQueue       # type: string;      example: {{.AMQPQueue}}
    .AMQPPayload     # type: string;      example: {{.AMQPPayload}}
  • Use helper functions inside {{ expr }}. We recommend pipeline format (|) of the functions.
    # Supported functions defined in ./template_helper.go
    
      - 
      - jsonPath    # doc: https://github.com/antchfx/xpath
      - gJsonPath   # doc: https://github.com/tidwall/gjson
      - xmlPath     # doc: https://github.com/antchfx/xpath
      - uuidv5      # uuid v5 sha1 hash
      - redisDo     # run redis commands. For example {{redisDo "RPUSH" "arr" "hi"}}
      - ...
    
    # Supported functions inherited from
    # https://github.com/Masterminds/sprig/blob/master/functions.go
    
      - replace
      - uuidv4
      - regexMatch
      - ...
    
    # Examples
    {{.HTTPHeader.Get "X-Token" | eq "t1234"}}
    {{.HTTPBody | jsonPath "user/first_name" | replace "A" "a" | uuidv5 }}
    {{.HTTPBody | gJsonPath "users.0.first_name" }}
    {{.HTTPBody | xmlPath "node1/node2/node3"}}

Admin Interface

Openmock also by default provides an API on port 9998 to control the running instance. See api documentation. You can serve the api documentation by getting go-swagger and running:

./swagger serve --host 0.0.0.0 --port 9997 docs/api_docs/bundle.yaml"

Command Line Interface

Openmock has a command-line interface to help with certain tasks interacting with openmock instances. This is invoked with the omctl command. This uses the cobra library to provide a discoverable CLI; run omctl for a list of commands / flags.

CLI: Directory

Push

Pushes a local openmock model from the file system to a remote instance.

# Adds templates from the ./demo_templates directory to the instance running on localhost.
omctl push --directory ./demo_templates --url http://localhost:9998

Examples

Example: Mock HTTP

# demo_templates/http.yaml

# $ curl 0:9999/ping
# OK
- key: ping
  kind: Behavior
  expect:
    http:
      method: GET
      path: /ping
  actions:
    - reply_http:
        status_code: 200
        body: OK
        headers:
          Content-Type: text/html

# $ curl 0:9999/token -H X-Token:t1234 -H Y-Token:t1234
# OK
- key: header-token-200
  kind: Behavior
  expect:
    condition: '{{.HTTPHeader.Get "X-Token" | eq "t1234" | and (.HTTPHeader.Get "Y-Token" | eq "t1234")}}'
    http:
      method: GET
      path: /token
  actions:
    - reply_http:
        status_code: 200
        body: OK

# $ curl 0:9999/token
# Invalid X-Token
- key: header-token-401
  kind: Behavior
  expect:
    condition: '{{.HTTPHeader.Get "X-Token" | ne "t1234"}}'
    http:
      method: GET
      path: /token
  actions:
    - reply_http:
        status_code: 401
        body: Invalid X-Token

Example: Mock GRPC

# demo_templates/grpc.yaml

- key: example_grpc
  expect:
    grpc:
      service: demo_protobuf.ExampleService
      method: ExampleMethod
  actions:
    - reply_grpc:
        payload_from_file: './files/example_grpc_response.json'

Example: Mock Kafka

# demo_templates/kafka.yaml

- key: test_kafka_1
  kind: Behavior
  expect:
    kafka:
      topic: hello_kafka_in
  actions:
    - publish_kafka:
        topic: hello_kafka_out
        payload: >
          {
            "kafka": "OK",
            "data": {}
          }

- key: test_kafka_2
  kind: Behavior
  expect:
    kafka:
      topic: hello_kafka_in_2
  actions:
    - publish_kafka:
        topic: hello_kafka_out
        payload_from_file: './files/colors.json' # the path is relative to OPENMOCK_TEMPLATES_DIR

If you started the example from docker-compose, you can test the above kafka mocks by using a kt docker container.

# Exec into the container
docker-compose exec kt bash

# Run some kt commands inside the container
# Notice that the container is within the docker-compose network, and it connects to "kafka:9092"

$ kt topic
$ echo '{"123":"hi"}' | kt produce -topic hello_kafka_in -literal
$ kt consume -topic hello_kafka_out -offsets all=newest:newest

Example: Mock AMQP (e.g. RabbitMQ)

# demo_templates/amqp.yaml

- key: test_amqp_1
  kind: Behavior
  expect:
    amqp:
      exchange: exchange_1
      routing_key: key_in
      queue: key_in
  actions:
    - publish_amqp:
        exchange: exchange_1
        routing_key: key_out
        payload: >
          {
            "amqp": "OK",
            "data": {}
          }

- key: test_amqp_2
  kind: Behavior
  expect:
    amqp:
      exchange: exchange_1
      routing_key: key_in
      queue: key_in
  actions:
    - publish_amqp:
        exchange: exchange_1
        routing_key: key_out
        payload_from_file: './files/colors.json'

Example: Use Redis for stateful things (by default, OpenMock uses an in-memory miniredis)

# demo_templates/redis.yaml

- key: hello_redis
  kind: Behavior
  expect:
    http:
      method: GET
      path: /test_redis
  actions:
    - redis:
      - '{{.HTTPHeader.Get "X-TOKEN" | redisDo "SET" "k1"}}'
      - '{{redisDo "RPUSH" "random" uuidv4}}'
      - '{{redisDo "RPUSH" "random" uuidv4}}'
      - '{{redisDo "RPUSH" "random" uuidv4}}'
    - reply_http:
        status_code: 200
        body: >
          {
            "k1": "{{redisDo "GET" "k1"}}",
            "randomStr": "{{redisDo "LRANGE" "random" 0 -1}}",
            "random": [
              {{ $arr := redisDo "LRANGE" "random" 0 -1 | splitList ";;" }}
              {{ range $i, $v := $arr }}
                {{if isLastIndex $i $arr}}
                  "{{$v}}"
                {{else}}
                  "{{$v}}",
                {{end}}
              {{end}}
            ]
          }

# To test
# curl localhost:9999/test_redis -H "X-TOKEN:t123"  | jq .

Example: Send Webhooks

# demo_templates/webhook.yaml

- key: webhooks
  kind: Behavior
  expect:
    http:
      method: GET
      path: /send_webhook_to_httpbin
  actions:
    - send_http:
        url: "https://httpbin.org/post"
        method: POST
        body: '{"hello": "world"}'
        headers:
          X-Token: t123
    - reply_http:
        status_code: 200
        body: 'webhooks sent'

# To test
# curl localhost:9999/send_webhook_to_httpbin

Example: Use data in templates

# demo_templates/http.yaml

- key: http-request-template
  kind: Template
  template: >
    { "http_path": "{{.HTTPPath}}", "http_headers": "{{.HTTPHeader}}" }

- key: color-template
  kind: Template
  template: >
    { "color": "{{.color}}" }

- key: teapot
  kind: AbstractBehavior
  expect:
    http:
      method: GET
      path: /teapot
  actions:
    - reply_http:
        status_code: 418
        body: >
          {
            "request-info": {{ template "http-request-template" . }},
            "teapot-info": {{ template "color-template" .Values }}
          }

# $ curl 0:9999/teapot
# {   "request-info": { "http_path": "/teapot", "http_headers": "map[Accept:[*/*] User-Agent:[curl/7.54.0]]" } ,   "teapot-info": { "color": "purple" }  }
- key: purple-teapot
  kind: Behavior
  extend: teapot
  values:
    color: purple

Advanced pipeline functions

To enable advanced mocks, for example, your own encoding/decoding of the kafka messages, one can develop by directly importing the github.com/checkr/openmock package, making a copy of the swagger-generated server main, and passing in a custom OpenMock.

For example: (see example)

package main

import (
  "github.com/checkr/openmock"
  "github.com/checkr/openmock/swagger_gen/restapi"
  "github.com/checkr/openmock/swagger_gen/restapi/operations"
  /// etc
)

func consumePipelineFunc(c openmock.Context, in []byte) (out []byte, error) {
  return decode(in), nil
}

func main() {
  // server set up copy & paste...
  
  // add our custom openmock functionality
  om := &openmock.OpenMock{}
  om.ParseEnv()
  om.KafkaConsumePipelineFunc = consumePipelineFunc

  server.ConfigureAPI(om)

  // rest of server set up copy & paste...
}

GRPC Configuration Notes

OpenMock uses the APIv2 protobuf module (google.golang.org/protobuf). If your project uses the APIv1 protobuf module, you can use https://github.com/golang/protobuf/releases/tag/v1.4.0 and convert your messages to be APIv2 compatible with the proto.MessageV2 method.

Please note that OpenMock expects the payload or payload_from_file for a reply_grpc action to be in the json form of your Response protobuf message. The request should be in the Request protobuf message format as it is parsed into json to support jsonPath and gJsonPath operations.

Example configuration by directly importing the github.com/checkr/openmock package into a wrapper project.

func main() {
  // server set up copy & paste...
  
  // add our custom openmock functionality
  om := &openmock.OpenMock{}
  om.GRPCServiceMap = map[string]openmock.GRPCService{
      "demo_protobuf.ExampleService": {
          "ExampleMethod": openmock.RequestResponsePair{
              Request:  proto.MessageV2(&demo_protobuf.ExampleRequest{}),
              Response: proto.MessageV2(&demo_protobuf.ExampleResponse{}),
          },
      },
  }
  om.ParseEnv()
  server.ConfigureAPI(om)

  // rest of server set up copy & paste...

Swagger

Swagger files / directories:

Makefile                    # contains build process for swagger generation
swagger/                    # directory containing swagger definition, split 
                            # up into a few files
   index.yaml               # all the model definitions are in here
   health.yaml              # method definitions relating to e.g. /health
swagger_gen/                # directory where generated swagger files go
  restapi/
    configure_open_mock.go  # this file contains code further customized from the 
                            # generated code to hook an implementation into the API
                            # the makefiles makes sure it is preserved when 
                            # generating the other files
docs/
  api_docs/
    bundle.yaml             # combined swagger spec file, generated by Makefile
pkg/
  admin/                    # code implementing the handlers for the swagger API

Install Swagger

brew tap go-swagger/go-swagger
brew install go-swagger

Generate

  • make gen - bundles the separate swagger files and generates swagger_gen
  • make build - builds the executables om and omctl

Run

OPENMOCK_REDIS_TYPE=redis OPENMOCK_REDIS_URL=<redis Url, e.g. redis://localhost:6379> OPENMOCK_TEMPLATES_DIR=./demo_templates ./om --port 9998

Issues
  • Initial GRPC implementation

    Initial GRPC implementation

    This PR adds GRPC mock support with the new action reply_grpc. The GRPC mocking configuration includes a mapping of response protobuf with the service and method (see grpc.go). We add GPRC endpoints by using a http/2 cleartext server to register the POST requests that GRPC sends. A GRPC POST request resolves to /service_name/method_name.

    For a mocked GRPC endpoint, we return a GRPC response based on the defined proto response type and the json payload or payload_from_file. We also set the proper length-prefixed message as defined in https://github.com/grpc/grpc/blob/master/doc/PROTOCOL-HTTP2.md:

    The repeated sequence of Length-Prefixed-Message items is delivered in DATA frames
    
    Length-Prefixed-Message → Compressed-Flag Message-Length Message
    Compressed-Flag → 0 / 1 # encoded as 1 byte unsigned integer
    Message-Length → {length of Message} # encoded as 4 byte unsigned integer (big endian)
    Message → *{binary octet}
    

    Currently, all GRPC responses return a trailing header grpc-status: 0 (success).

    opened by jpatdalton 9
  • EVER-1852 Perform http_requests async

    EVER-1852 Perform http_requests async

    In order to simulate webhooks we currently create actions like:

    actions:
         - publish_kafka:
            topic: delayed-webhook
            payload_from_file: ./delayed_webhook_data
         - publish_kafka:
            topic: other-delayed-webhook
            payload_from_file: ./other_delayed_webhook_data
         - reply_http:
            status_code: 200
    

    then create consumers for those webhooks like

    - key: delayed-webhook
      expect:
        kafka:
          topic: delayed-webhook
        - sleep:
            duration: 10s
        - send_http:
            url: https://checkrhq-dev.net/
            method: POST
    

    we have to do it this way since actions are executed synchronously. this can easily create confusion and introduce bugs when we are trying to simulate more complex workflows and webhook events by daisy-chaining kafka messages

    This PR makes it so that all send-http actions are performed in a goroutine and adds a new Sleep field to send-http actions. this allows us to make clearer m.Actions arrays in the templates, for example we can simulate webhooks like this:

    actions:
         - reply_http:
            status_code: 200
        - send_http:
              sleep: 10s
              url: https://checkrhq-dev.net/webhook_1
              method: POST
        - send_http:
              sleep: 20s
              url: https://checkrhq-dev.net/webhook_2
              method: POST
    

    TODO

    • [ ] tests
    • [ ] update demo templates

    I'll go ahead and complete these 2 TODOs if folks agree that this is a workable solution

    opened by dneilroth 8
  • Codify admin API in swagger and utilize go-swagger for server

    Codify admin API in swagger and utilize go-swagger for server

    Created an open API (https://swagger.io/docs/specification/about/) spec for the openmock admin API. This allows us to use swagger code generation tools to run our admin server, support JSON (for the API) as well as YAML, generate API documentation, and generate clients that can communicate to the OM instance.

    Design Decisions:

    • There's a pretty clear border in pkg/admin/crud.go and pkg/admin/convert.go where we convert back and forth from the models for the generated API and the original openmock models. We have a bunch of code written, particularly validation, that isn't easily replicated in swagger and seemed like we should leave that in place. I started down the path of replacing the whole OM model with the generated models, but it's a lot of work so maybe for another day.

    • we use the makefile to allow us to edit a few of the generated files; these might sometimes require manual merging if swagger-gen changed what it produces

    • the swagger-generated main.go replaces the original cmd/main.go

    opened by sesquipedalian-dev 4
  • Create helper script to post templates at a running instance

    Create helper script to post templates at a running instance

    • create a helper script that loads templates locally and posts them at a remote openmock instance
    • make some methods related to loading public so that the script can access them
    • when loading a template that uses 'BodyFromFile' or 'PayloadFromFile', delete the file reference after loading the file into the 'Body' or 'Payload' field. This should make sure a file reference loaded locally doesn't percolate to a remote instance where it would not be available
    opened by sesquipedalian-dev 4
  • Templating returns an INT when yaml value is all digits

    Templating returns an INT when yaml value is all digits

    We are getting the following error in our templated mocks:

    failed to render condition: {{
      (.HTTPBody | regexFindFirstSubmatch "<License>([^<>]+)</License>" | eq .Values.license_number)
    }}  err="template: :1:297: executing \"\" at <eq .Values.license_number>: error calling eq: incompatible types for comparison"
    echo: http: response.Write on hijacked connection from io.(*multiWriter).Write (multi.go:60)
    

    We are using an AbstractBehavior:

    - key: some-template
      kind: AbstractBehavior
      values:
        license_number: G1234567
      actions:
        - reply_http:
            status_code: 200
            body: OK
      expect:
        http:
          method: POST
          path: /some-path
        condition: '{{ (.HTTPBody | regexFindFirstSubmatch "<License>([^<>]+)</License>" | eq .Values.license_number) }}
    

    The mocks look like:

    - key: some-key
      extend: some-template
      values:
        license_number: G1234567
      actions:
        - redis:
    ...
    

    After some investigation, the problem turns out to be that one of the mocks have license_number of 000000000, and it was templated into a int instead of a string.

    Would it make sense to always return a string type?

    @WilsonGiese

    opened by michelleheh 3
  • Add linter

    Add linter

    Fixing all the linter issues

    e.g.

    handle_actions_test.go:24: File is not `gofmt`-ed with `-s` (gofmt)                                                            
                            Actions: []ActionDispatcher{ActionDispatcher{}},                                                       
    openmock.go:12:15: struct of size 248 bytes could be of size 224 bytes (maligned)                                              
    type OpenMock struct {                                                             
                  ^                                                                                                                
    handle_actions_test.go:51:25: Error return value of `unActingMock.DoActions` is not checked (errcheck)               
                    unActingMock.DoActions(Context{})                    
                                          ^                                                                                        
    handle_actions_test.go:69:23: Error return value of `actingMock.DoActions` is not checked (errcheck)
                    actingMock.DoActions(Context{})                                    
                                        ^                                                                                           load.go:207:38: Error return value of `(*github.com/checkr/openmock/vendor/github.com/fatih/structs.Field).Set` is not checked (
    errcheck)                                                                                                                      
                            baseStruct.Field(field.Name()).Set(field.Value())                           
                                                              ^                                                                    
    openmock.go:42:11: Error return value of `env.Parse` is not checked (errcheck)     
            env.Parse(om)                                                                                                           
                     ^                                                                                                              
    template_helper.go:124:9: Error return value of `h.Write` is not checked (errcheck)                           
            h.Write([]byte(data))                                                                                 
                   ^                                                                                                                handle_actions.go:39:1: receiver name should be a reflection of its identity; don't use generic names such as "this" or "self" (
    golint)                                                                                                                        
    func (self ActionSendHTTP) Perform(context Context) error {                                                                    
    ^                                                                                                                               handle_actions.go:65:1: receiver name should be a reflection of its identity; don't use generic names such as "this" or "self" (
    golint)                                                                                                                         
    func (self ActionReplyHTTP) Perform(context Context) error {                                             
    ^                                                                                                                               handle_actions.go:82:1: receiver name should be a reflection of its identity; don't use generic names such as "this" or "self" (
    golint)                                                                                               
    func (self ActionRedis) Perform(context Context) error {                                                                 
    ^                                                                                                                              
    kafka.go:55:24: U1000: func `(*kafkaClient).close` is unused (unused)                                       
    func (kc *kafkaClient) close() error {                                                                      
                           ^                                                                                                       
    kafka.go:113:4: S1000: should use for range instead of for { select {} } (gosimple)                                             
                            for {                                                                                                   
                            ^                                                                                                       
    load.go:120:5: SA4006: this value of `err` is never used (staticcheck)                                                          
            m, err := redis.StringMap(v, err)                                                    
               ^                                                                                                                   
    load.go:151:9: S1030: should use w.Bytes() instead of []byte(w.String()) (gosimple)                                             
            return []byte(w.String()), nil                                                                                          
                   ^                                                                                                               
    redis_test.go:32:3: SA4006: this value of `v` is never used (staticcheck)                                                      
                    v, err := redisHandleReply(om.redis.Do("rpush", "k1", "v1"))                                                    
                    ^                                                                                                               
    template_helper_test.go:12:2: S1021: should merge variable declaration with assignment on next line (gosimple)                  
            var tmpl string                                                                                                         
            ^                                                                            
    openmock.go:42:11: Error return value of `env.Parse` is not checked (errcheck)                               
            env.Parse(om)
                     ^                                                                                           
    template_helper.go:124:9: Error return value of `h.Write` is not checked (errcheck)                          
            h.Write([]byte(data))
                   ^                                                                                             
    handle_actions.go:39:1: receiver name should be a reflection of its identity; don't use generic names such as 
    "this" or "self" (golint)                                                                                    
    func (self ActionSendHTTP) Perform(context Context) error {                                                  
    ^                                                                                                            
    handle_actions.go:65:1: receiver name should be a reflection of its identity; don't use generic names such as "this" or "self" (golint)
    func (self ActionReplyHTTP) Perform(context Context) error {                                                 
    ^
    handle_actions.go:82:1: receiver name should be a reflection of its identity; don't use generic names such as "this" or "self" (golint)
    func (self ActionRedis) Perform(context Context) error {                                                     
    ^
    kafka.go:55:24: U1000: func `(*kafkaClient).close` is unused (unused)                                        
    func (kc *kafkaClient) close() error {                                                                       
                           ^                                                                                     
    kafka.go:113:4: S1000: should use for range instead of for { select {} } (gosimple)                          
                            for {                                                                                
                            ^
    load.go:120:5: SA4006: this value of `err` is never used (staticcheck)                                       
            m, err := redis.StringMap(v, err)                                                                    
               ^
    load.go:151:9: S1030: should use w.Bytes() instead of []byte(w.String()) (gosimple)                          
            return []byte(w.String()), nil
                   ^                                                                                             
    redis_test.go:32:3: SA4006: this value of `v` is never used (staticcheck)                                    
                    v, err := redisHandleReply(om.redis.Do("rpush", "k1", "v1"))                                 
                    ^
    template_helper_test.go:12:2: S1021: should merge variable declaration with assignment on next line (gosimple)        var tmpl string
            ^                                                            
    
    
    opened by zhouzhuojie 3
  • Add basicauth example

    Add basicauth example

    Tested locally. Sent a webhook to https://user1:[email protected]/basic-auth/user1/password1

    $ make build
    $ USER=user1 PASSWORD=password1 make run
    
    [http] 2019/08/21 11:26:57 HTTP Request: GET /basic-auth/user1/password1 HTTP/1.1          
    Host: httpbin.org                                                                          
                                                                                               
    [http] 2019/08/21 11:26:57 HTTP Response: HTTP/1.1 200 OK                                 
    Connection: close
    Access-Control-Allow-Credentials: true
    Access-Control-Allow-Origin: *
    Content-Type: application/json
    Date: Wed, 21 Aug 2019 18:27:00 GMT
    Referrer-Policy: no-referrer-when-downgrade
    Server: nginx
    X-Content-Type-Options: nosniff
    X-Frame-Options: DENY
    X-Xss-Protection: 1; mode=block
    
    {
      "authenticated": true,
      "user": "user1"
    }
    
    
    opened by zhouzhuojie 3
  • Implement post-key for manipulating the OM model via the HTTP admin interface

    Implement post-key for manipulating the OM model via the HTTP admin interface

    Current

    Openmock has an admin interface (HTTP on port 9998 by default) that allows user to POST new templates. Pseudo code:

    Unmarshal POST body as YAML into an open mock model (400 if fails) Foreach mock in loaded model S = marshal mock to YAML Redis hset redis_templates_storage S End Restart OM process

    When the OM (re)starts, it loads up YAML strings from the templates directory files, and from the redis key redis_templates_storage, and concatenates them into one big YAML string, which it then unmarshals into an OM model. The redis stuff is loaded with redis HGETALL redis_templates_storage

    The other functionality to note is that, when evaluating go templates in om (e.g. in conditions, http_reply bodies, etc), the redisDo commands lets mock users run arbitrary redis commands in the instance OM is using. This could potentially interfere with the template storage database.

    What we’re trying to do We would like to add a new endpoint to post sets of templates. This would allow remote services to functionally ‘delete’ the mocks that they’ve added, by doing HTTP DELETE at the same endpoint. This is easier to implement than deleting each key of the mock since the job that does the empty post won’t have to parse the mocks or know anything about them, just the unique key used.

    Proposed Add a new endpoint to the admin API to store sets of templates under a unique key (/api/v1/template_sets/:key). POSTing here causes om to save the mocks with HSET (redis_templates_storage_<post key>) …. DELETEing the same path that deletes the key redis_templates_storage_.
    When loading templates in load.go, first find all keys in the DB that start with redis_template_store, and concatenate the strings from all those hsets in the same way as we currently do with just redis_template_store. This should load in all the mocks that were posted as a template set in addition to the base ones.

    Testing Added unit testing for new functionality. Manual test script:

    1. post a template with set-key ABC
    2. post the same template with set-key DEF
    3. GET the templates, observe the posted template was added
    4. DELETE templates with set-key ABC
    5. GET templates, observe the posted template still present
    6. DELETE templates with set-key DEF
    7. GET templates, observe the posted template is gone now that we've eliminated all sources of it

    Work not directly related to immediate goal Should we take this opportunity to define the admin IF with goswagger? If we do, we could also make part of omctl update use the generated client? It seems like a fair amount of work to define OM’s model in swagger terms and use it throughout, maybe save it for hack week projects We should restrict the redisDo template helper so that any of the ‘args’ to the redis command are checked to see if they start with the string ‘redis_templates_storage’, and thow an error if so. This would prevent users from accidentally messing with OM’s template storage.

    Other solutions Considered storing any additional post keys in a separate redis set so that we don’t have to use the KEYs command to find keys while loading. KEYs is O(1) with a fairly low constant time on larger data stores, and we only have one redis instance for all of an IIE. We should see if performance becomes a problem here, and if so implement that solution. I don’t jump straight to it since it adds some complexity to the implementation in OM.

    opened by sesquipedalian-dev 2
  • Adds optional Conditions to Expect

    Adds optional Conditions to Expect

    Description

    This PR adds the ability to define a list of conditions in the template expectations that can be used instead of a single condition

    Currently it will preform an implicit AND on all conditions, and condition if it is also present.

    Motivation: Readability of template conditions

    A single condition works for simple expectations, but these conditions can get quite large, and become more difficult to read.

    For example:

    condition: '{{ and (and (.HTTPHeader.Get "SOAPAction" | eq "http://webconnect.abcd.com/webconnect/2019/01/WebService/Login") (.HTTPBody | regexFindFirstSubmatch "<Account>([^<>]+)</Account>" | eq "TEST_ACCOUNT")) (.HTTPBody | regexFindFirstSubmatch "<User>([^<>]+)</User>" | eq "JOHNDOE") }}''
    

    Could be

    conditions: 
        - '{{ .HTTPHeader.Get "SOAPAction" | eq "http://webconnect.abcd.com/webconnect/2019/01/WebService/Login" }}'
        - '{{ .HTTPBody | regexFindFirstSubmatch "<Account>([^<>]+)</Account>" | eq "TEST_ACCOUNT" }}'
        - '{{ .HTTPBody | regexFindFirstSubmatch "<User>([^<>]+)</User>" | eq "JOHNDOE" }}'
    
    
    
    enhancement 
    opened by WilsonGiese 2
  • Load body_from_file for ActionSendHTTP

    Load body_from_file for ActionSendHTTP

    Currently if you want to perform send_http events with arbitrary data from a file (vs a kafka,ampq, or http payload) there is no way to do so since we do not do not replace the ActionSendHTTP Body field with the data in the file path located in the BodyFromFile field. We already do this for all other models with a BodyFromFile field.

    Tested locally, looks good 👍

    test template looks like:

    - key: ping
      kind: Behavior
      expect:
        http:
          method: GET
          path: /ping
      actions:
        - reply_http:
            status_code: 200
            body: OK
            headers:
              Content-Type: text/xml
        - sleep:
            duration: 10s
        - send_http:
            url: http://localhost:9999/hello
            method: POST
            body_from_file: ../files/test.tpl
    
    - key: hello
      kind: Behavior
      expect:
        http:
          method: POST
          path: /hello
      actions:
        - reply_http:
            status_code: 200
            body: '{{ .HTTPBody }}'
    

    and test.tpl is:

    {
      "goodbye": "world"
    }
    

    running curl http://localhost:9999/ping we see the request to /ping return right away Screen Shot 2020-01-08 at 3 59 46 PM

    then the request to /hello return with the body from the test.tpl file after 10 seconds: Screen Shot 2020-01-08 at 3 59 52 PM

    opened by dneilroth 1
  • Add get/set data functions via Redis

    Add get/set data functions via Redis

    Add the ability to get/set via Redis.

    For example:

    - key: hello_redis                                                         
      expect:                                                                  
        http:                                                                  
          method: GET                                                          
          path: /test_redis                                                    
      actions:                                                                 
        - redis:                                                               
          - '{{.HTTPHeader.Get "X-TOKEN" | redisDo "SET" "k1"}}'               
          - '{{redisDo "RPUSH" "random" uuidv4}}'                              
          - '{{redisDo "RPUSH" "random" uuidv4}}'                              
          - '{{redisDo "RPUSH" "random" uuidv4}}'                              
        - reply_http:                                                          
            status_code: 200                                                   
            body: >                                                            
              {                                                                
                "k1": "{{redisDo "GET" "k1"}}",                                
                "randomStr": "{{redisDo "LRANGE" "random" 0 -1}}",             
                "random": [                                                    
                  {{ $arr := redisDo "LRANGE" "random" 0 -1 | splitList ";;" }}
                  {{ range $i, $v := $arr }}                                   
                    {{if isLastIndex $i $arr}}                                 
                      "{{$v}}"                                                 
                    {{else}}                                                   
                      "{{$v}}",                                                
                    {{end}}                                                    
                  {{end}}                                                      
                ]                                                              
              }                                                                
                                                                               
    
    [openmock] curl localhost:9999/test_redis -H "X-TOKEN:t123"  | jq .
    {
      "k1": "t123",
      "randomStr": "81a8ceac-0c0a-4f44-b6df-6a1a8f97e211;;94c69ba1-f399-48ad-a27e-7fe1720dbdbf;;3716ffcb-34e0-4280-a922-5cefd6ea7db4",
      "random": [
        "81a8ceac-0c0a-4f44-b6df-6a1a8f97e211",
        "94c69ba1-f399-48ad-a27e-7fe1720dbdbf",
        "3716ffcb-34e0-4280-a922-5cefd6ea7db4"
      ]
    }
    
    opened by zhouzhuojie 1
  • Usage example of GRPC is unclear

    Usage example of GRPC is unclear

    Hi!

    I launched openmock by running docker-compose and by setting OPENMOCK_GRPC_ENABLED and OPENMOCK_GRPC_PORT Then I made a GRPC request to demo_probuf.ExampleService.ExampleMethod Request timed out and I see following in logs:

         openmock         | {"error":"empty GRPCServiceMap","level":"error","msg":"failed to convert gRPC request body to JSON","time":"2021-02-18T06:42:30Z"}
         openmock         | {"grpc_host":"localhost:50051","grpc_method":"POST","grpc_path":"/demo_protobuf.ExampleService/ExampleMethod","grpc_req":"\u0000\u0000\u0000\u0000\u000e\u0008{\u0012\u0006fooBar\u001a\u0002{{","grpc_res":"{\"message\":\"Internal Server Error\"}\n","level":"info","msg":"","time":"2021-02-18T06:42:30Z"}
    

    I tried to find what that error means in source code and the only place where GRPCServiceMap is assigned is in /grpc_test.go Does that mean that currently in order to use GRPC mocking one has to write a Go wrapper project?

    Thanks!

    Sidenote: I'd like to say that it took me some time to find out that one has to enable GRPC by setting OPENMOCK_GRPC_ENABLED At first I just set OPENMOCK_GRPC_PORT and expected it to work 😅 since a Kafka configuration example which is a few lines below mentions OPENMOCK_KAFKA_ENABLED flag. Perhaps Readme.md should mention OPENMOCK_GRPC_ENABLED

    opened by tehnetaholic 0
  • Connection not closed properly

    Connection not closed properly

    We're using go's "net/http" client to send http requests to the stub. When sending multiple POST requests to the openmock http server the client fails with an EOF error.

    After investigating further we noticed that the this error occurs if the server closed the connection without noticing the client. https://stackoverflow.com/questions/28046100/golang-http-concurrent-requests-post-eof/34474535#34474535

    To reproduce this I created a simple stub:

    - key: mystub
      kind: Behavior
      expect:
        http:
          method: POST
          path: /
      actions:
        - reply_http:
            status_code: 200
            body: somepostresponse
    

    And a small go test

    import (
    	"fmt"
    	"io/ioutil"
    	"net/http"
    	"strconv"
    	"strings"
    	"testing"
    )
    
    func TestPost(t *testing.T) {
    	for i := 0; i < 20; i++ {
    		if err := sendRequest(i); err != nil {
    			panic(fmt.Sprintf("sending request %d failed because of %v", i, err))
    		}
    	}
    }
    
    func sendRequest(i int) error {
    	resp, err := http.Post("http://localhost:9091", "text", strings.NewReader("request "+strconv.Itoa(i)))
    	if err != nil {
    		panic(err)
    	}
    
    	defer resp.Body.Close()
    	respBody, err := ioutil.ReadAll(resp.Body)
    	if err != nil {
    		panic(err)
    	}
    
    	println(fmt.Sprintf("got response %s for request %d", string(respBody), i))
    	return nil
    }
    

    Our current workaround is to add the "connection: close" header to each stubbed request to signal the http client that it shouldn't reuse the connection, but is there any way this can be solved in the openmock server? If not, then this at least might be useful for the next person using go and running into this issue.

    Thanks!

    opened by pacbeckh 0
  • Admin endpoint for mocks creation should return 400 if the input is not valid

    Admin endpoint for mocks creation should return 400 if the input is not valid

    I think a few functions may need to bubble up the errors so that we can prevent invalid input early on. Incidents like an invalid yaml input could potentially crash the openmock server and may need manual interaction from the redis server.

    https://github.com/checkr/openmock/blob/68fadf576ac767f742206f76c0dd91d3d6f91b16/pkg/admin/convert.go#L62

    opened by zhouzhuojie 1
  • panic error when mocking http request > kafka topic

    panic error when mocking http request > kafka topic

    Hi there,

    When I try to mock kafka publising to a topic based on a an HTTP GET request, I get the following error: INFO[0012] try to publish to kafka err="<nil>" out_message= payload="{ \"kafka\": \"OK\", \"data\": {} " topic=hello_kafka_out echo: http: panic serving 172.17.0.1:60042: runtime error: invalid memory address or nil pointer dereference goroutine 6 [running]: net/http.(*conn).serve.func1(0xc000224000) /usr/local/go/src/net/http/server.go:1769 +0x139 panic(0xab7e00, 0x11c9e20) /usr/local/go/src/runtime/panic.go:522 +0x1b5 github.com/checkr/openmock.(*kafkaClient).sendMessage(0x0, 0xc0001dc220, 0xf, 0xc000288480, 0x112, 0x120, 0x0, 0x0) /go/src/github.com/checkr/openmock/kafka.go:43 +0x170 github.com/checkr/openmock.ActionPublishKafka.Perform(0xc0001dc220, 0xf, 0xc000158120, 0x112, 0x0, 0x0, 0xc9b8c0, 0xc00020a070, 0xc00020c4e0, 0x0, ...) /go/src/github.com/checkr/openmock/handle_actions.go:123 +0x1cd github.com/checkr/openmock.(*Mock).DoActions(0xc0000a1ee0, 0xc9b8c0, 0xc00020a070, 0xc00020c4e0, 0x0, 0x0, 0xc0001dc2b0, 0x5, 0x0, 0x0, ...) /go/src/github.com/checkr/openmock/handle_actions.go:28 +0x192 github.com/checkr/openmock.MocksArray.DoActions(0xc0001bd4e0, 0x2, 0x2, 0xc9b8c0, 0xc00020a070, 0xc00020c4e0, 0x0, 0x0, 0xc0001dc2b0, 0x5, ...) /go/src/github.com/checkr/openmock/handle_actions.go:14 +0x96 github.com/checkr/openmock.(*OpenMock).startHTTP.func2.1(0xc9b8c0, 0xc00020a070, 0x20, 0xafbe60) /go/src/github.com/checkr/openmock/http.go:42 +0x2c2 github.com/labstack/echo.(*Echo).Add.func1(0xc9b8c0, 0xc00020a070, 0x2, 0xc79180) /go/src/github.com/checkr/openmock/vendor/github.com/labstack/echo/echo.go:482 +0x87 github.com/labstack/echo/middleware.BodyDumpWithConfig.func1.1(0xc9b8c0, 0xc00020a070, 0x11f3300, 0x3) /go/src/github.com/checkr/openmock/vendor/github.com/labstack/echo/middleware/body_dump.go:81 +0x375 github.com/dafiti/echo-middleware.LogrusWithConfig.func1.1(0xc9b8c0, 0xc00020a070, 0x3, 0xc0000380a4) /go/src/github.com/checkr/openmock/vendor/github.com/dafiti/echo-middleware/logrus.go:97 +0x119 github.com/labstack/echo.(*Echo).ServeHTTP(0xc000208000, 0xc84fc0, 0xc00024c000, 0xc000226300) /go/src/github.com/checkr/openmock/vendor/github.com/labstack/echo/echo.go:594 +0x227 net/http.serverHandler.ServeHTTP(0xc0001fc000, 0xc84fc0, 0xc00024c000, 0xc000226300) /usr/local/go/src/net/http/server.go:2774 +0xa8 net/http.(*conn).serve(0xc000224000, 0xc86e00, 0xc000076240) /usr/local/go/src/net/http/server.go:1878 +0x851 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:2884 +0x2f4

    My template is as follows: ` - key: test_kafka_1 kind: Behavior expect: http: method: GET path: /ping kafka: topic: hello_kafka_in actions: - publish_kafka: topic: hello_kafka_out payload: > { "kafka": "OK", "data": {}

    - key: test_kafka_2 kind: Behavior expect: http: method: GET path: /ping kafka: topic: hello_kafka_in actions: - publish_kafka: topic: hello_kafka_out payload: > { "kafka": "OK", "data": {"base_tug":2.0,"avg_delta_tug":0.0,"ship_dim_pca_0":-0.021482626865567126,"ship_dim_pca_1":0.10497156370907207,"bs_ok":0,"hs_ok":0,"loc_score":0.002597402597402597,"ship_group":"Stukgoedschepen","wind_speed":4.0}, "nb_tug": 2.0 }`

    Did I do any wrong here?

    opened by openrory 0
Releases(v0.3.2)
Owner
Checkr
Background check API
Checkr
Example code to demonstrate how to mock external clients via context.Context

Mocking external client libraries using context.Context This code is paired with a blog post: Mocking external client libraries using context.Context

incident.io 2 Feb 22, 2022
Cloud governance reports from native services in a clear and readable digest

cloudig, or Cloudigest, is a simple CLI tool for creating reports from various cloud sources with user-provided comments. It is written in Go and curr

Optum 18 Feb 18, 2022
The Fabric Token SDK is a set of API and services that lets developers create token-based distributed application on Hyperledger Fabric.

The Fabric Token SDK is a set of API and services that let developers create token-based distributed application on Hyperledger Fabric.

null 52 Jun 7, 2022
The Bhojpur Ara is a software product used for automated resource assembly within Bhojpur.NET Platform ecosystem to enable delivery of applications and services.

Bhojpur Ara - Automated Resource Assembly The Bhojpur Ara is a service product used for automated resource assembly within the Bhojpur.NET Platform ec

Bhojpur Consulting 0 Apr 28, 2022
Nvote - Decentralized, vote-driven community similar to services like Reddit and HackerNews. Built on nostr

NVote Nvote is a decentralized, vote-driven community similar to services like R

Ronald Bell 18 Jun 20, 2022
Service that wrap up different movies-related APIs like IMDB and match it to streaming services

Service that wrap up different movies-related APIs like IMDB and match it to streaming services. That way you can check in which platforms you can find your favorite movies.

Xavier Graterol 2 Feb 10, 2022
Lambda stack to turn off and destroy all resources from your personal AWS Account to avoid billing surprises

AWS, Turn off my Account, please Lambda stack to turn off and destroy all resources from your personal AWS Account to avoid billing surprises Resource

Matheus Fidelis 63 Apr 10, 2022
The MinIO Admin Go Client SDK provides APIs to manage MinIO services

Golang Admin Client API Reference The MinIO Admin Golang Client SDK provides APIs to manage MinIO services. This quickstart guide will show you how to

High Performance, Kubernetes Native Object Storage 28 Jun 6, 2022
NKN-ESI (or nESI) is an NKN based Energy Services Interface (ESI).

nkn-esi NKN-ESI (or nESI) is an NKN based Energy Services Interface (ESI). An ESI supports a distributed marketplace for energy services on the electr

Elijah J. Passmore 2 Mar 18, 2022
Integrate AWS EKS Anywhere cluster with AWS Services

This article provides step-by-step instruction on integrating AWS EKS Anywhere with AWS Services so the applications running on customer data center can securely connect with these services.

Hari Ohm Prasath 2 Mar 6, 2022
Sample CDK projects developed by AWS Professional Services (ProServe)

AWS CDK Examples - ProServe This repository contains a set of CDK example projects which have been developed by AWS Professional Services (ProServe).

AWS Samples 5 Mar 17, 2022
CoreDNS plugin implementing K8s multi-cluster services DNS spec.

multicluster Name multicluster - implementation of Multicluster DNS Description This plugin implements the Kubernetes DNS-Based Multicluster Service D

CoreDNS 25 Jun 20, 2022
Simple tool to search tagged resources between all AWS resouces

Welcome to Cloud Inventory Tags ?? Simple tool to search tagged resources around all AWS Account Installation MacOS / OSX

Matheus Fidelis 16 Jan 26, 2022
Automate all the tasks you can do in NeteaseCloudMusic

Fuck163MusicTasks 自动完成网易云音乐人任务并领取云豆 说白了就是白嫖网易云年费黑胶 ✨ 特性 web/Android 双平台每日签到 音乐人每日签到(登录音乐人中心) 自动发布动态(音乐人每日任务) 自动回复粉丝评论(音乐人每日任务) 自动恢复粉丝私信(音乐人每日任务) 自动领取已

lz已经是条咸鱼了 124 Jun 20, 2022
AWS Tags Updater - Sync tags with all resources via sheet 🐏🐏

AWS Tags Updater - Sync tags with all resources via sheet ????

PePoDev 1 Mar 22, 2022
Go-http-client: An enhanced http client for Golang

go-http-client An enhanced http client for Golang Documentation on go.dev ?? This package provides you a http client package for your http requests. Y

Furkan Bozdag 40 May 23, 2022
Package anko provides a golang SDK to the Anko Investor Forecasts gRPC service.

github.com/anglo-korean/anko-go-sdk Package anko provides a golang SDK to the Anko Investor Forecasts gRPC service. import "github.com/anglo-korean/an

Anglo Korean 0 Jan 3, 2022
Testing ground for build-your-own golang/grpc demo app.

Getting started Prereqs You will need to install both Go and the protoc compiler (version 3): Go installation protoc installation Install the protobuf

Jordan Weiner 0 Dec 15, 2021