GoReplay is an open-source tool for capturing and replaying live HTTP traffic into a test environment in order to continuously test your system with real data. It can be used to increase confidence in code deployments, configuration changes and infrastructure changes.

Overview

GitHub release codebeat Go Report Card Join the chat at https://gitter.im/buger/gor Reviewed by Hound

Go Replay

https://goreplay.org/

GoReplay is an open-source network monitoring tool which can record your live traffic, and use it for shadowing, load testing, monitoring and detailed analysis.

About

As your application grows, the effort required to test it also grows exponentially. GoReplay offers you the simple idea of reusing your existing traffic for testing, which makes it incredibly powerful. Our state of art technique allows you to analyze and record your application traffic without affecting it. This eliminates the risks that come with putting a third party component in the critical path.

GoReplay increases your confidence in code deployments, configuration and infrastructure changes.

GoReplay offers unique approach for shadowing. Instead of being a proxy, GoReplay in background listen for traffic on your network interface, requiring no changes in your production infrastructure, rather then running GoReplay daemon on the same machine as your service.

Diagram

Check latest documentation.

Installation

Download latest binary from https://github.com/buger/goreplay/releases or compile by yourself.

Getting started

The most basic setup will be sudo ./gor --input-raw :8000 --output-stdout which acts like tcpdump. If you already have test environment you can start replaying: sudo ./gor --input-raw :8000 --output-http http://staging.env.

See the our documentation and Getting started page for more info.

Newsletter

Subscribe to our newsletter to stay informed about the latest features and changes to Gor project.

Want to Upgrade?

We have created a GoReplay PRO extension which provides additional features such as support for binary protocols like Thrift or ProtocolBuffers, saving and replaying from cloud storage, TCP sessions replication, etc. The PRO version also includes a commercial-friendly license, dedicated support, and it also allows you to support high-quality open source development.

Problems?

If you have a problem, please review the FAQ and Troubleshooting wiki pages. Searching the issues for your problem is also a good idea.

All bug-reports and suggestions should go through Github Issues or our Google Group (you can just send email to [email protected]). If you have a private question feel free to send email to [email protected].

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Added some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

Companies using Gor

  • GOV.UK - UK Government Digital Service
  • theguardian.com - Most popular online newspaper in the UK
  • TomTom - Global leader in navigation, traffic and map products, GPS Sport Watches and fleet management solutions.
  • 3SCALE - API infrastructure to manage your APIs for internal or external users
  • Optionlab - Optimize customer experience and drive engagement across multiple channels
  • TubeMogul - Software for Brand Advertising
  • Videology - Video advertising platform
  • ForeksMobile - One of the leading financial application development company in Turkey
  • Granify - AI backed SaaS solution that enables online retailers to maximise their sales
  • And many more!

If you are using Gor, we are happy to add you to the list and share your story, just write to: [email protected]

Author

Leonid Bugaev, @buger, https://leonsbox.com

Issues
  • How to merge/split  traffic from multiple endpoints

    How to merge/split traffic from multiple endpoints

    We have multiple endpoints for HA/load balancing reasons. To reconstruct production traffic recorded on all those nodes we would need to merge all recorded files in to one.

    I noticed in recording file timestamps are not in exact order but seems to be played in order. Is that right ?

    Will concatenating and sorting all records by timestamp do the job ?

    Also in the future we would like to split merged file to run on N clients to emulate real clients.

    We plan to split by sessions/source IP. Ideas ?

    I believe we are not first who tries to do this. I know google groups would be more appropriate , just do not see any activity there.

    opened by nikolai-derzhak-distillery 34
  • Middleware for custom request rewrite logic

    Middleware for custom request rewrite logic

    Middleware

    Middleware is a program that accepts request and response payload at STDIN and emits modified requests at STDOUT. You can implement any custom logic like stripping private data, advanced rewriting, support for oAuth and etc.

                       Original request      +--------------+
    +-------------+----------STDIN---------->+              |
    |  Gor input  |                          |  Middleware  |
    +-------------+----------STDIN---------->+              |
                       Original response     +------+---+---+
                                                    |   ^
    +-------------+    Modified request             v   |
    | Gor output  +<---------STDOUT-----------------+   |
    +-----+-------+                                     |
          |                                             |
          |            Replayed response                |
          +------------------STDIN----------------->----+
    

    Middleware can be written in any language, see examples/middleware folder for examples. Middleware program should accept the fact that all communication with Gor is asynchronous, there is no guarantee that original request and response messages will come one after each other. Your app should take care of the state if logic depends on original or replayed response, see examples/middleware/token_modifier.go as example.

    Simple bash echo middleware (returns same request) will look like this:

    while read line; do
      echo $line
    end
    

    Middleware can be enabled using --middleware option, by specifying path to executable file:

    gor --input-raw :80 --middleware "/opt/middleware_executable" --output-http "http://staging.server"
    

    Communication protocol

    All messages should be hex encoded, new line character specifieds the end of the message, eg. new message per line.

    Decoded payload consist of 2 parts: header and HTTP payload, separated by new line character.

    Example request payload:

    1 932079936fa4306fc308d67588178d17d823647c 1439818823587396305
    GET /a HTTP/1.1
    Host: 127.0.0.1
    
    

    Example response payload:

    2 8e091765ae902fef8a2b7d9dd960e9d52222bd8c 2782013
    HTTP/1.1 200 OK
    Date: Mon, 17 Aug 2015 13:40:23 GMT
    Content-Length: 0
    Content-Type: text/plain; charset=utf-8
    
    

    Header contains request meta information separated by spaces. First value is payload type, possible values: 1 - request, 2 - original response, 3 - replayed response. Next goes request id: unique among all requests (sha1 of time and Ack), but remain same for original and replayed response, so you can create associations between request and responses. Third argument varies depending on payload type: for request - start time, for responses - round-trip time.

    HTTP payload is unmodified HTTP requests/responses intercepted from network. You can read more about request format here, here and here. You can operate with payload as you want, add headers, change path, and etc. Basically you just editing a string, just ensure that it is RCF compliant.

    At the end modified (or untouched) request should be emitted back to STDOUT, keeping original header, and hex-encoded. If you want to filter request, just not send it. Emitting responses back is optional, and does not affect anything at the moment.

    Advanced example

    Imagine that you have auth system that randomly generate access tokens, which used later for accessing secure content. Since there is no pre-defined token value, naive approach without middleware (or if middleware use only request payloads) will fail, because replayed server have own tokens, not synced with origin. To fix this, our middleware should take in account responses of replayed and origin server, store originalToken -> replayedToken aliases and rewrite all requests using this token to use replayed alias. See examples/middleware/token_modifier.go and middleware_test.go#TestTokenMiddleware as example of described scheme.

    Latest precompiled binaries: https://www.dropbox.com/s/27chmbsqxrolvz4/gor_0.9.9_middleware_x64.tar.gz?dl=1 https://www.dropbox.com/s/vmeg5sexcleoo2e/gor_0.9.9_middleware_x86.tar.gz?dl=1

    opened by buger 28
  • Using same input and output port causes request duplication

    Using same input and output port causes request duplication

    Hello! Amazing work you did here. However, we found a small issue at our company.

    Using gor with this commandline: gor --output-http-timeout '1800s' --input-raw 'intf-ip-addr:8080' --output-http 'http://output-http-addr:8080|100%' --http-allow-url '/some/url' causes requests to be replayed multiple times on target machine (as checked on tomcat access-logs and tcpdump). Replaying request to another port (eg. 8081) fixes the issue. We confirmed the issue on gor versions 0.12.1 and 0.13.0, however, issue is not present in version 0.10.1. Machine runs binary release of gor on CentOS 6.6 2.6.32-504.12.2.el6.x86_64.

    opened by walczakp 26
  • --output-http-track-response  not returning response for replayed requests.

    --output-http-track-response not returning response for replayed requests.

    Hi here,

    I am trying to capture responses of the replayed requests , but not able to get back responses all time . On replaying a request, the program code is getting stuck on the net.conn.Read() call. https://github.com/buger/goreplay/blob/master/http_client.go#L199 .

    I see the response in tcpdump output, but not able to obtain the same from within the code using net.conn.Read()

    opened by jacoblukose 22
  • Errors: dial tcp: cannot assign requested address

    Errors: dial tcp: cannot assign requested address

    Hi, I'm seeing many errors from gor:

    2014/09/04 16:20:20 Request error: Get http://....: dial tcp x.x.x.x:8080: cannot assign requested address
    

    (x.x.x.x is an IP address and the actual http url removed).

    I use gor as:

    sudo ./gor --input-raw :80 --output-tcp y.y.y.y:28020
    

    and on y.y.y.y:

    ./gor --input-tcp :28020 --output-http http://x.x.x.x:8080
    

    The error seem to stem from high load in some sense. When request rate is low they are gone and everything works as expected. It's possible that the accepting http server (x.x.x.x:8080) is slow on high load. Could that be the reason for the errors on gor? Anyway, it's more of a hunch. What do these errors actually mean? And how to mitigate them? (even assuming that the accepting server is indeed slow)

    I tried, out of the blue to increase ulimit and set GOMAXPROCS to various numbers but these are just shots in the dark and as expected, they didn't help.

    What's your advise?

    (btw, sorry if there's a mailing list I didn't find it, so I'm posting this as a bug here, it doesn't mean I think it's an actual bug with gor of course).

    opened by rantav 21
  • Is the request id unique globally?

    Is the request id unique globally?

    As the document, I expect it is. image

    But after I captured traffic and analyze the .gor file, I saw that request id is not seemly unique. The following picture is my traffic file and I searched one request Id but 46 matched.

    image

    And I found that tcp_message.go#UUID() function calculates the request id by src address and dest address, which seems not to be unique.

    Could you please answer this question? Thanks!

    opened by ZmfCn 20
  • Gor performance issues

    Gor performance issues

    Don't know which would be the best way to debug GOR for performance testing. We've been working with it for almost a month now, and I've seen an increase on the load avg on a canary 300+ req/s aprox. It reach to a point where not only gor as a listener but the replay begin to grow instance loads.

    I am talking about m1.large instances on AWS 8GB ram, 2 cores Instance Store, yesterday someone mentioned that maybe it could be related to encoding characters on our requests (because we have japanese characters) but I think it doesn't make any sense.

    What you think @buger? any test that I can run on my systems to detect the bottleneck? and we are running 0.7.5.

    opened by rhoml 20
  • Limit listening side

    Limit listening side

    Hi,

    I have a volume problem where we get up to 20k qps on the server I need to sniff traffic from but i only need to forward a very small subset of them onto our test servers, ideally around 5-10 qps. I have tried limiting on the replay side but each time I start up I currently see the following error: "Error while Accept() accept tcp [::]:28020: too many open files"

    I believe if there is a way to limit as well on the sending side then this would not be an issue.

    Thanks

    Mark

    opened by pixie79 20
  • elasticsearch can't parse URI

    elasticsearch can't parse URI

    hi there,

    Based on your doc via https://github.com/buger/goreplay/wiki/Exporting-to-ElasticSearch

    The es7 and gor 1.2 have been installed in my centos7 OS and, I ran the command as below: (Is there any version limitation of elasticsearch?)

    /usr/local/bin/gor --input-raw :8080 --output-http http://localhost:8081 --output-http-elasticsearch http://localhost:9200/test123

    The test123 index has not been created. Also, the pattern of es URI must be scheme://host/index_name Otherwise, the exception will be thrown out: 2020/11/13 15:49:38 Can't initialize ElasticSearch plugin.Wrong ElasticSearch URL format. Expected to be: scheme://host/index_name

    It is appreciated that you could guide me on how to resolve this issue.

    And, another issue is about kafka with version 1.2, which https://github.com/buger/goreplay/issues/848 has been submitted. I am concerning whether this bug related with kafka will be fixed in next version (v1.3?) and when will the next version be released?

    Sorry for your inconvenience. BR Kimi

    needs info 
    opened by quzhixue-Kimi 18
  • Replaying requests for large files

    Replaying requests for large files

    I wanted to use gor for testing a HTTP server while mirroring real traffic. Unfortunately the traffic was for relatively big media files (from 100MB up to few GBs). In that case the buffer for reading the response body was too small and gor replayer was failing to replicate the load which real users are generating on the mirrored server.

    An other limitation was the fixed deadline for reading the response body. With very large files a timeout of 5s, 30s or whatever is not practical as very big files can take some time to be downloaded. On the other hand a relatively small timeout is required to stop stalled connections which do not move any data around anymore. Maybe because of a faulty HTTP server under test, network problems or something else.

    This pull request aims to remedy both of the problems. It does it by

    • Reading the "whole" body of the response. Or at least trying to do so. The reading is done in small chunks which are discarded instantly in order not to consume too much memory for large responses. An upper limit of the read data is still present to make sure a faulty server will not create an never ending chunked response or some other problem of the kind.
    • Instead of a fixed deadline, a rolling timeout is introduced while reading the response. The connection will timeout only if there is no network activity at all for the set amount of time.
    • The behaviour regarding small files is not changed in any way.
    • Fixes few crashes which were happening from time to time during prolonged tests.
    opened by ironsmile 18
  • Can't rewrite Host header

    Can't rewrite Host header

    This is on 0.7.5

    ~#  ./gor --input-raw :80  --output-http "http://my-staging-host"  --output-http-header "Host: my-staging-host"
    

    But.. sniffing the traffic

    ~# tcpdump -c 20 -s 1023 -A host my-staging-host and tcp port http
    ...
    Host: news-business.vlex.com
    User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
    ...
    Host: borme.vlex.es
    Connection: Keep-alive
    Accept: */*
    

    so it seems that Hostname is not being rewritten

    opened by angelf 18
  • Bump docker/build-push-action from 2 to 3

    Bump docker/build-push-action from 2 to 3

    Bumps docker/build-push-action from 2 to 3.

    Release notes

    Sourced from docker/build-push-action's releases.

    v3.0.0

    • Node 16 as default runtime by @​crazy-max (#564)
      • This requires a minimum Actions Runner version of v2.285.0, which is by default available in GHES 3.4 or later.
    • Standalone mode support by @​crazy-max (#601 #609)
    • chore: update dev dependencies and workflow by @​crazy-max (#571)
    • Bump @​actions/exec from 1.1.0 to 1.1.1 (#573)
    • Bump @​actions/github from 5.0.0 to 5.0.1 (#582)
    • Bump minimist from 1.2.5 to 1.2.6 (#584)
    • Bump semver from 7.3.5 to 7.3.7 (#595)
    • Bump csv-parse from 4.16.3 to 5.0.4 (#533)

    Full Changelog: https://github.com/docker/build-push-action/compare/v2.10.0...v3.0.0

    v2.10.0

    • Add imageid output and use metadata to set digest output (#569)
    • Add build-contexts input (#563)
    • Enhance outputs display (#559)

    v2.9.0

    • add-hosts input (#553 #555)
    • Fix git context subdir example and improve README (#552)
    • Add e2e tests for ACR (#548)
    • Add description on github-token option to README (#544)
    • Bump node-fetch from 2.6.1 to 2.6.7 (#549)

    v2.8.0

    • Allow specifying subdirectory with default git context (#531)
    • Add cgroup-parent, shm-size, ulimit inputs (#501)
    • Don't set outputs if empty or nil (#470)
    • docs: example to sanitize tags with metadata-action (#476)
    • docs: wrong syntax to sanitize repo slug (#475)
    • docs: test before pushing your image (#455)
    • readme: remove v1 section (#500)
    • ci: virtual env file system info (#510)
    • dev: update workflow (#499)
    • Bump @​actions/core from 1.5.0 to 1.6.0 (#160)
    • Bump ansi-regex from 5.0.0 to 5.0.1 (#469)
    • Bump tmpl from 1.0.4 to 1.0.5 (#465)
    • Bump csv-parse from 4.16.0 to 4.16.3 (#451 #459)

    v2.7.0

    • Add metadata output (#412)
    • Bump @​actions/core from 1.4.0 to 1.5.0 (#439)
    • Add note to sanitize tags (#426)
    • Cache backend API docs (#406)
    • Git context now supports subdir (#407)
    • Bump codecov/codecov-action from 1 to 2 (#415)

    v2.6.1

    • Small typo and ensure trimmed output (#400)

    ... (truncated)

    Commits
    • e551b19 Merge pull request #564 from crazy-max/node-16
    • 3554377 Merge pull request #609 from crazy-max/ci-fix-test
    • a62bc1b ci: fix standalone test
    • c208583 Merge pull request #601 from crazy-max/standalone-mode
    • fcd9124 Merge pull request #607 from docker/dependabot/github_actions/docker/metadata...
    • 0ebe720 Bump docker/metadata-action from 3 to 4
    • 38b4580 Standalone mode support
    • ba31738 Merge pull request #533 from docker/dependabot/npm_and_yarn/csv-parse-5.0.4
    • 43721d2 Update generated content
    • 5ea21bf Fix csv-parse implementation since major update
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 3
  • Bump docker/setup-buildx-action from 1 to 2

    Bump docker/setup-buildx-action from 1 to 2

    Bumps docker/setup-buildx-action from 1 to 2.

    Release notes

    Sourced from docker/setup-buildx-action's releases.

    v2.0.0

    • Node 16 as default runtime by @​crazy-max (#131)
      • This requires a minimum Actions Runner version of v2.285.0, which is by default available in GHES 3.4 or later.

    Full Changelog: https://github.com/docker/setup-buildx-action/compare/v1.7.0...v2.0.0

    v1.7.0

    • Standalone mode by @​crazy-max in (#119)
    • Update dev dependencies and workflow by @​crazy-max (#114 #130)
    • Bump tmpl from 1.0.4 to 1.0.5 (#108)
    • Bump ansi-regex from 5.0.0 to 5.0.1 (#109)
    • Bump @​actions/core from 1.5.0 to 1.6.0 (#110)
    • Bump actions/checkout from 2 to 3 (#126)
    • Bump @​actions/tool-cache from 1.7.1 to 1.7.2 (#128)
    • Bump @​actions/exec from 1.1.0 to 1.1.1 (#129)
    • Bump minimist from 1.2.5 to 1.2.6 (#132)
    • Bump codecov/codecov-action from 2 to 3 (#133)
    • Bump semver from 7.3.5 to 7.3.7 (#136)

    v1.6.0

    • Add config-inline input (#106)
    • Bump @​actions/core from 1.4.0 to 1.5.0 (#104)
    • Bump codecov/codecov-action from 1 to 2 (#101)

    v1.5.1

    • Explicit version spec for caching (#100)

    v1.5.0

    • Allow building buildx from source (#99)

    v1.4.1

    • Fix docker: invalid reference format (#97)

    v1.4.0

    • Update dev deps (#95)
    • Use built-in getExecOutput (#94)
    • Use core.getBooleanInput (#93)
    • Bump @​actions/exec from 1.0.4 to 1.1.0 (#85)
    • Bump y18n from 4.0.0 to 4.0.3 (#91)
    • Bump hosted-git-info from 2.8.8 to 2.8.9 (#89)
    • Bump ws from 7.3.1 to 7.5.0 (#90)
    • Bump @​actions/tool-cache from 1.6.1 to 1.7.1 (#82 #86)
    • Bump @​actions/core from 1.2.7 to 1.4.0 (#80 #87)

    v1.3.0

    • Display BuildKit version (#72)

    v1.2.0

    • Remove os limitation (#71)
    • Add test job for config input (#68)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 3
  • GOR docker --output-file doesn't record the traffic in actual file

    GOR docker --output-file doesn't record the traffic in actual file

    docker run --network=container:9dc9dc69a830 buger/goreplay --input-raw :4080 --output-file=gorrequests.gor

    I'm trying to record the traffic by using above command, but I don't see the file stored.

    opened by dvadiyala 0
  • k8:// line gives invalid path error

    k8:// line gives invalid path error

    I am trying to create a daemon set, I have a service deployed already in namespace: cms-test. My file looks like apiVersion: apps/v1 kind: DaemonSet metadata: name: goreplay-daemon spec: template: spec: hostNetwork: true serviceAccountName: goreplay containers: - name: goreplay image: buger/goreplay:latest command: - "--input-raw k8s://default/pod/nginx-deployment-6588ff5bdd-4cc7c" - "--output-file requests.log" But I am getting this error whenever I try to run it stat --input-raw k8s://cms-test/pod/cms-api-67f778dc5b-5wk2g: no such file or directory: unknown

    opened by Vaishali2896 0
  • user agent

    user agent

    If original request "user agent" is empty, the replayed one will be "Go-http-client/1.1" Is it intended behaviour? Anyway to keep original "user agent", even if it is empty?

    opened by ecc256 0
Releases(v2.0.0-rc2)
  • v2.0.0-rc2(Feb 7, 2022)

  • v2.0.0-rc1(Feb 7, 2022)

  • 1.3.3(Oct 6, 2021)

    What's Changed

    • Various improvements for packet capture, especially for POST requests https://github.com/buger/goreplay/pull/1007
    • Fixed replay of pcap files https://github.com/buger/goreplay/pull/1007
    • Fixed replay of files with malformed records https://github.com/buger/goreplay/pull/1015

    Full Changelog: https://github.com/buger/goreplay/compare/v1.3.2...1.3.3

    Source code(tar.gz)
    Source code(zip)
    gor-1.3.3_windows.zip(7.57 MB)
    gor-1.3.3.pkg(7.92 MB)
    gor_1.3.3_mac.tar.gz(7.92 MB)
    gor-1.3.3-1.x86_64.rpm(10.51 MB)
    gor_1.3.3_amd64.deb(10.55 MB)
    gor_1.3.3_x64.tar.gz(10.55 MB)
  • v1.3.2(Aug 19, 2021)

    • Fix promiscuous mode - if enabled no need to filter on host as we want to capture traffic not related to this host #994
    • Fix --input-raw-realip-header https://github.com/buger/goreplay/pull/996
    • Fixed false positive network interface lookups https://github.com/buger/goreplay/pull/1000

    Thanks @DimaGolomozy!

    Source code(tar.gz)
    Source code(zip)
    gor-1.3.2_windows.zip(7.57 MB)
    gor-1.3.2.pkg(10.93 MB)
    gor_1.3.2_mac.tar.gz(10.93 MB)
    gor-1.3.2-1.x86_64.rpm(10.51 MB)
    gor_1.3.2_amd64.deb(10.55 MB)
    gor_1.3.2_x64.tar.gz(10.55 MB)
  • v1.3.1(Aug 12, 2021)

  • v1.3.0(Aug 6, 2021)

    Highlights

    Added official Windows support https://github.com/buger/goreplay/wiki/Running-on-Windows

    Traffic capture engine fully re-constructed to be more scalable, efficient and bulletproof. You should see way lower packet loss, capture quality, and CPU/Memory usage.

    Deprecated own HTTP client in favour of native Go HTTP library https://github.com/buger/goreplay/pull/833

    Increased stability, and numerous bug fixes.

    Features

    • Added way to capture traffic from multipe ports with single input --input-raw :80,8080,3000 https://github.com/buger/goreplay/pull/935
    • Added --output-tcp-skip-verify flag to skip TCP output TLS verification https://github.com/buger/goreplay/pull/814 @tomerf
    • Added --output-tcp-workers to configure number of TCP output connections https://github.com/buger/goreplay/pull/819
    • Added new debugging endpoint when --http-pprof enabled: /debug/vars
    • Added new %i file name variable to inject unique instance ID to file name. Can be useful if you have logs from multiple machines and you need unique names.
    • Added --input-file-dry-run option to previewing how long it will take and how many requests in given file input https://github.com/buger/goreplay/commit/8e76559b492402ef1d2aee99e0106d085db7f935
    • Added --input-raw-max-wait option (in seconds), which allows to skip long pauses in recorded files https://github.com/buger/goreplay/commit/19ad90aa4bb0320c8ba623d5509a0690e8c3ec25
    • Added --input-file-read-depth option to pre-read and buffer requests (and sort them). By default, is 100 https://github.com/buger/goreplay/commit/625ed54f1ebcc006e7d04a27544807fbaff08508
    • Added --input-raw-timestamp-type go to enable application level timestamps, when network timestamps unreliable https://github.com/buger/goreplay/commit/8edb74e57276c712444ccfb89ed40dd0ca5c39d6

    Thank you, @urbanishimwe, @slimus, @arijitAD for jumping with me to this adventure!

    Huge kudos to all GoReplay community, especially to our contributors @DimaGolomozy @tomerf @swills @davidFR @two @jl2005 @rmasclef @othererik @ankitdobhal @eko @code4wt @wangfeng22 @lins05 @betty1126 @YuriYin @StanleyP

    Source code(tar.gz)
    Source code(zip)
    gor-1.3.0_windows.zip(7.57 MB)
    gor-1.3.0.pkg(10.93 MB)
    gor_1.3.0_mac.tar.gz(10.93 MB)
    gor-1.3.0-1.x86_64.rpm(10.51 MB)
    gor_1.3.0_amd64.deb(10.55 MB)
    gor_1.3.0_x64.tar.gz(10.55 MB)
  • v1.3.0_RC11(Aug 3, 2021)

    • Fix loopback interface issue on windows https://github.com/buger/goreplay/commit/c9274ac92a6f021240d82682002240cfceaecd5e
    • Fix buffer index number for sticky tcp output connection https://github.com/buger/goreplay/pull/981 @DimaGolomozy
    • Fix output file size limits https://github.com/buger/goreplay/pull/974 @lins05
    • Fixed usage of --input-raw-bpf-filter https://github.com/buger/goreplay/commit/2b993eda49b2a9e9d54ae6cbd616dbb6f6f28388
    • Fix BPF filter when listen on all interfaces https://github.com/buger/goreplay/commit/a727ade00608b526f0388e9d60c21705136be75c
    • Optimize packet capture thread to reduce context switching https://github.com/buger/goreplay/commit/214edb45f63ac38dc63c9f21017ddfab557f9d06
    • Added new debugging endpoint when --http-pprof enabled: /debug/vars
    • Added --input-file-dry-run option to previewing how long it will take and how many requests in given file input https://github.com/buger/goreplay/commit/8e76559b492402ef1d2aee99e0106d085db7f935
    • Added --input-raw-max-wait option (in seconds), which allows to skip long pauses in recorded files https://github.com/buger/goreplay/commit/19ad90aa4bb0320c8ba623d5509a0690e8c3ec25
    • Added --input-file-read-depth option to pre-read and buffer requests (and sort them). By default, is 100 https://github.com/buger/goreplay/commit/625ed54f1ebcc006e7d04a27544807fbaff08508
    • Added --input-raw-timestamp-type go to enable application level timestamps, when network timestamps unreliable https://github.com/buger/goreplay/commit/8edb74e57276c712444ccfb89ed40dd0ca5c39d6
    Source code(tar.gz)
    Source code(zip)
    gor-1.3.0_RC11_windows.zip(7.57 MB)
    gor-1.3.0_RC11.pkg(10.93 MB)
    gor_1.3.0_RC11_mac.tar.gz(10.93 MB)
    gor-1.3.0_RC11-1.x86_64.rpm(10.51 MB)
    gor_1.3.0_RC11_amd64.deb(10.55 MB)
    gor_1.3.0_RC11_x64.tar.gz(10.55 MB)
  • v1.3.0_RC6(Jun 30, 2021)

    • Significantly improved CPU and Memory usage.
    • Added af_packet packet capture engine for Linux https://github.com/buger/goreplay/pull/950
    • Skip incomplete or malformed messages from output (can be enabled back by setting --input-raw-allow-incomplete) https://github.com/buger/goreplay/pull/953
    Source code(tar.gz)
    Source code(zip)
    gor-v1.3.0_RC6_windows.zip(7.55 MB)
    gor-v1.3.0_RC6.pkg(10.89 MB)
    gor_v1.3.0_RC6_mac.tar.gz(10.89 MB)
    gor-v1.3.0_RC6-1.x86_64.rpm(10.48 MB)
    gor_v1.3.0_RC6_amd64.deb(10.53 MB)
    gor_v1.3.0_RC6_x64.tar.gz(10.53 MB)
  • v1.3.0_RC5(Jun 10, 2021)

    • Added a way to capture multiple ports using one input like this: --input-raw :80,8080,3000 #935
    • Fixed usage of "localhost" hostname when trying to limit scope of packet capture (e.g. --input-raw localhost:80), now it properly binds to loopback interface #943
    • Fixed detection of windows network interfaces #943
    • Fixed Makefile job to generate windows binaries
    • Fixed big memory allocation issue happening during HTTP message end check. Was also causing issues in chunked message detection.
    Source code(tar.gz)
    Source code(zip)
    gor-v1.3.0_RC5_windows.zip(7.53 MB)
    gor-v1.3.0_RC5.pkg(10.86 MB)
    gor_v1.3.0_RC5_mac.tar.gz(10.86 MB)
    gor-v1.3.0_RC5-1.x86_64.rpm(10.40 MB)
    gor_v1.3.0_RC5_amd64.deb(10.45 MB)
    gor_v1.3.0_RC5_x64.tar.gz(10.44 MB)
  • v1.3.0_RC4(Jun 9, 2021)

    Changes from previous RC

    • Big changes in capture engine to give more performance and stability #916
    • Fixed a bug when GoRepay were including outgoing HTTP request initialized within the app, and having the same port #941
    • Added (again) windows support #940
    • Fix Output HTTP connection leak #927
    Source code(tar.gz)
    Source code(zip)
    gor-1.3.0_RC4_windows.zip(7.52 MB)
    gor-1.3.0_RC4.pkg(10.85 MB)
    gor_1.3.0_RC4_mac.tar.gz(10.85 MB)
    gor-1.3.0_RC4-1.x86_64.rpm(10.40 MB)
    gor_1.3.0_RC4_amd64.deb(10.44 MB)
    gor_1.3.0_RC4_x64.tar.gz(10.44 MB)
  • v1.2.0(Sep 17, 2020)

    • Engine CPU consumption reduced by 60%
    • Implemented M'mapped AF_PACKET for Linux. To enable set --input-raw-engine af_packet. #807
    • Added Kafka TLS configuration flag #800
    • Fix wrong param value updated while --http-set-param #791
    • The engine can now capture more requests with large payload #797
    • Created TCP package #97
    • Fixed minor bugs in HTTP proto parser
    Source code(tar.gz)
    Source code(zip)
    gor-v1.2.0.pkg(11.29 MB)
    gor_v1.2.0_mac.tar.gz(11.29 MB)
    gor-v1.2.0-1.x86_64.rpm(10.88 MB)
    gor_v1.2.0_amd64.deb(10.92 MB)
    gor_v1.2.0_x64.tar.gz(10.92 MB)
  • v1.1.0(Jul 4, 2020)

    • Fix usage of multiple --input-raw plugins, e.g. --input-raw :80 --input-raw :81 --input-raw :82 https://github.com/buger/goreplay/pull/700
    • Allow to set byte size limits in human friedly format: --output-file-size-limit, --output-file-max-size-limit, --copy-buffer-size and --input-raw-buffer-size can now parse inputs from differents bases and data units like: 10mb, 10kb, 100gb, 18t https://github.com/buger/goreplay/pull/754
    • Fix calculation of the output file size when using gzip compression https://github.com/buger/goreplay/pull/777
    • Added --output-tcp-sticky so request/response with same ID sent to the same connection.
    • Better handling of malformed middleware output https://github.com/buger/goreplay/pull/737
    • Fixed race when reading or writing using files https://github.com/buger/goreplay/pull/756 https://github.com/buger/goreplay/pull/764
    • Improved errors verbosity https://github.com/buger/goreplay/issues/703
    • Updated dependencies, including libpcap which should bring more stability and performance
    • Move to Go modules
    • PRO code moved to the main repository under separate license
    Source code(tar.gz)
    Source code(zip)
    goreplay-1.1.0.pkg(11.27 MB)
    gor_1.1.0_mac.tar.gz(11.27 MB)
    goreplay-1.1.0-1.x86_64.rpm(10.81 MB)
    goreplay_1.1.0_amd64.deb(10.86 MB)
    gor_1.1.0_x64.tar.gz(10.86 MB)
  • v1.0.0(Mar 30, 2019)

    Long awaited major release of GoReplay!

    • Added option to specify custom BPF filter (e.g. tcpdump syntax). Can be useful in case of non standard network interfaces like tunneling or SPAN port. Example: --input-raw-bpf-filter 'dst port 80’ https://github.com/buger/goreplay/pull/478
    • Support for reading directly from pcap file, using --input-raw-engine pcap_file --input-raw ./recording.pcap_
    • Added official Docker image docker pull buger/goreplay. Should be run with --network host argument.
    • Added RPM and DEB packages
    • Added support for HTTP proxies. Just set HTTP_PROXY env variable.
    • Added HTTP basic auth filter: --http-basic-auth-filter "^customer[0-9].*. You specify regexp which match userid:passwd string. [https://github.com/buger/goreplay/pull/475]
    • Added option to limit size of output file: when limit is reached, it will exit. Can be used for safety reasons. Example: --output-file-max-size-limit 5gb.
    • Added way to configure output HTTO request queue, which used to hold requests, if all workers are busy. Example: --output-http-queue-len 5000. Default is 1000.
    • Control frequency of --output-http-stats reports using --output-http-stats-msoption. Example: --output-http-stats-ms 5000 (every 5 seconds).
    • Configurable way to set minimum number of HTTP workers, by setting --output-http-workers-min. Can be used in conjunction with --output-http-workers which will act as a max worker count.
    • Added way to dynamically profile GoReplay performance using Golang pprof tools. Example: --http-pprof :8181. It starts web server on given address, and expose special /debug/pprof endpoint with list of reports.
    • Added —input-raw-buffer-size which controls size of the OS buffer (in bytes) which holds packets until they dispatched. Default value depends by system: in Linux around 2MB. If you see big package drop, increase this value.
    • In addition to buffer size, added option to turn pcap immediate mode when packets delivered without buffering. Can help reduce packet drop. Example: --input-raw-immediate-mode.
    • Added way to use standard Golang HTTP client, by adding --output-http-compatibility-mode
    • Snaplen (max number of bytes being read for each packet) now dynamically set based on interface MTU + max header size. In most situations it should reduce package drop, because each packet will consume less space in the buffer. However in some virtualized environments like OpenShift, packet size can be significantly bigger then MTU, so you can disable optimization by setting --input-raw-override-snaplen.
    • Added support for ElasticSearch basic HTTP auth
    • Numerous fixes to improve quality of HTTP packets parsing
    Source code(tar.gz)
    Source code(zip)
    gor_1.0.0_mac.tar.gz(7.08 MB)
    gor_1.0.0_x64.tar.gz(5.45 MB)
    gor-1.0.0-1.x86_64.rpm(7.05 MB)
    gor_1.0.0_amd64.deb(7.08 MB)
  • v1.0-rc2(Feb 23, 2019)

  • v0.16.1(Jun 30, 2017)

    • Add support for TLS connections between --input-tcp and --output-tcp. --input-tcp-secure --input-tcp-certificate ./cert.pem --input-tcp-certificate-key ./key.pem --output-tcp-secure #457
    • Add basic support for old server, like HTTP 0.9 #468 #463 #467
    • GoReplay now exit once finished replaying with --input-file #456
    • Fix prettifier issue, when there is problems with gzip encoding payload #470
    Source code(tar.gz)
    Source code(zip)
    gor_0.16.1_x64.tar.gz(4.61 MB)
    gor_0.16.1_mac.tar.gz(4.34 MB)
  • v0.16.0.2(May 1, 2017)

    Finally a new big release, a lot of changes on all fronts. New features, usability stability, fixes.

    Thank you, everyone, who made it happen!

    It would not be possible without this people @smostovoy @exklamationmark @manjeshnilange @kudos @oivoodoo @sattvik @ylegat @nrwiersma @SophisticaSean

    CHANGELOG

    Major

    • [PRO] Added S3 input and output https://github.com/buger/goreplay/wiki/%5BPRO%5D-Using-S3-for-storing-and-replaying-traffic
    • Added new NodeJS middleware framework https://github.com/buger/goreplay/tree/master/middleware
    • Add Kafka input and output https://github.com/buger/goreplay/wiki/Streaming-from-and-to-Apache-Kafka
    • Improve accuracy of replay by using timestamp of when TCP packet was received by network interface (previously used time of package capture by GoReplay)
    • Using —prettify-http option, you can automatically decode Gzip encoded responses, and de-construct chunked bodies.
    • Now you can enable HTTP response tracking by providing —output-http-track-respose. Previously was available only if middleware is turned on.

    Minor

    • Add new option --input-raw-expire to configure TCP message expiration
    • Filter response if its request was filtered
    • Allow space inside filters syntax: `--http-disallow-header 'Host: www.vertaa.fi' now valid syntax
    • File name pattern now support request id %r, will log each request to separate file, and %t for payload type (0 - request, 1 - response, 2 - replayed response).
    • Add basic SNI support to support HTTP replay to hosts that require SNI, such as Amazon API Gateway. API Gateway)
    • Fix --output-file-size-limit option
    • Do not add port to Host header #383
    • Improve malformed TCP packet handling to avoid panics
    • Fix HTTP timeout for 204 (No content) responses
    • Fix handling of HTTP error codes like 400 or 304.
    • Fixed replay of HTTP PATCH requests
    Source code(tar.gz)
    Source code(zip)
    gor_0.16.0_mac.tar.gz(4.34 MB)
    gor_0.16.0_x64.tar.gz(4.60 MB)
  • v0.15.1(Aug 31, 2016)

  • v0.15.1-beta1(Aug 23, 2016)

  • v0.15.0(Aug 10, 2016)

    • [PRO] Added support for working with binary protocols (thrift/protocol buffers)
    • [PRO] Recording and replaying keep alive TCP sessions
    • Replaying request in proper order when using multiple files #300
    • Validate bodies when Content-Length or Transfer-Encoding found #317
    • Return of ElasticSearch support! #331 #333
    • Added --exit-after option to specify duration after which Gor will exit #336
    • Updated response output file format, now third value is timestamp and forth is latency
    • Properly read body when no Content-Length but there is Connection: close d34c27c
    • Remove --input-http option (too confusing and never really worked) f0acd31
    • Force Go DNS resolver (C one caused crashes for multiple people) 07fa6d9
    • Fix interception on loopback interface when non-local IP used 1ed8691
    • Fix intercepting traffic from virtual interfaces 51860e1
    • Fix 100-continue header when it places not in the end #314
    • Fix relative file names when using --output-file #304
    • Properly cleanup used resources on exit #305
    • Fix connection timeout when using --output-http-timeout c4271ff
    Source code(tar.gz)
    Source code(zip)
    gor_v0.15.0_x64.tar.gz(4.16 MB)
    gor_v0.15.0_mac.tar.gz(3.76 MB)
    gor.exe(7.50 MB)
  • v0.14.1(Jun 10, 2016)

  • v0.14.0(Jun 9, 2016)

    • Windows support!
    • Allow date variables in --output-file names: --output-file %Y-%m-%d-%H.log will create new file each hour #290
    • Support for gzip compressed files, both for input and output #290
    • Replaying from multiple files: --input-file now support file name patterns #290
    • Looping files for replaying indefinitely #290
    • New default --output-file behaviour for writing files in multiple chunks #293
    • Faster buffered file output #290
    • Support for lower-case HTTP headers
    • Added built-in file server for the tutorial: gor file-server :8000
    • Fix bug when gor catch its own --output-http traffic if it replayed to the same port as origin #295
    • --input-raw-realip-header option to injecting header with user real IP #296
    Source code(tar.gz)
    Source code(zip)
    gor.exe(7.13 MB)
    gor_v0.14.0_mac.tar.gz(3.61 MB)
    gor_v0.14.0_x64.tar.gz(4.03 MB)
  • v0.13.0(May 25, 2016)

    • --input-raw now do not track responses by default, you can enable them using --input-raw-track-response #279
    • --output-dummy now renamed to --output-stdout and does not require arguments #282
    • HTTP client response buffer now configurable via --output-http-response-buffer #249
    • Fixed recovery when --output-tcp endpoint goes down https://github.com/buger/gor/commit/0ccace8cad988dfb15ceb725ff3085b5f7c386fd
    • Improved HTTP capture and replaying for requests with multi-packet headers and POST requests #277 #281
    • Fixed libpcap crashing when listening on multiple interfaces https://github.com/buger/gor/commit/98868e81c500579428049944ce47ee50d0ec107b
    • Fixed --input-tcp crashing for payloads > 64kb https://github.com/buger/gor/commit/d40d6c6116c93f5f0f8da768296738cf915821ad
    • Fixed raw socket engine IPv6 support https://github.com/buger/gor/commit/98fe7f19b21eaaffcd0d733a57668f63de5921d7
    Source code(tar.gz)
    Source code(zip)
    gor_0.13.0_mac.tar.gz(3.53 MB)
    gor_0.13.0_x64.tar.gz(3.95 MB)
  • v0.12.1(May 13, 2016)

  • v0.12.0(May 11, 2016)

  • v0.11.2(May 3, 2016)

  • v0.11.1(Apr 29, 2016)

  • v0.11.0(Apr 29, 2016)

    New site and announce of Pro version https://gortool.com

    Release details:

    • License switched to LGPL, and added option for commercial friendly option
    • Default traffic interception engine now libpcap, which significantly increased capture quality and speed
    • Multiple fixes improving stability, speed and quality
    • Go 1.6.1
    Source code(tar.gz)
    Source code(zip)
    gor_0.11.1_x64.tar.gz(2.93 MB)
  • v0.10.1(Sep 3, 2015)

  • 0.10.0(Aug 24, 2015)

Owner
Leonid Bugaev
👉 👉 👉 👉 👉 👉 👉 👉 👉 👉 👉
Leonid Bugaev
HTTP API traffic recording and replay middleware based on GoReplay, can be used for migration and refactoring testing

gorc HTTP API traffic recording and replay middleware based on GoReplay, can be used for migration and refactoring testing. English | 中文 Requirements

Jioby 2 Feb 13, 2022
A LoRaWAN nodes' and network simulator that works with a real LoRaWAN environment (such as Chirpstack) and equipped with a web interface for real-time interaction.

LWN Simulator A LoRaWAN nodes' simulator to simulate a LoRaWAN Network. Table of Contents General Info Requirements Installation General Info LWN Simu

ARSLab 23 May 2, 2022
A tool for capturing newly issued x.509 from Certificate Transparency logs & performing periodic revocation checking.

ct-logster This repository contains the tools for collecting newly issued x509 certificates from Certificate Transparency logs, as well as performing

Adam Halim 4 May 4, 2022
Courier Order Provider is a service that receives signals from core server in order to emit this orders to courier groups.

Courier Order Provider Courier Order Provider is a service that receives signals(messages) from core server in order to emit this orders to courier gr

null 0 Nov 4, 2021
Limit-order-book - Limit order books keep records of orders for a given symbol to be traded

Limit Order Book Limit order books keep records of orders for a given symbol to

Michael Bowler 0 Jan 17, 2022
Traefik config validator: a CLI tool to (syntactically) validate your Traefik configuration filesTraefik config validator: a CLI tool to (syntactically) validate your Traefik configuration files

Traefik Config Validator Note This is currently pre-release software. traefik-config-validator is a CLI tool to (syntactically) validate your Traefik

Thomas Klinger 0 Dec 16, 2021
Apache Traffic Control is an Open Source implementation of a Content Delivery Network

Apache Traffic Control Apache Traffic Control is an Open Source implementation of a Content Delivery Network. Documentation Intro CDN Basics Traffic C

The Apache Software Foundation 764 May 10, 2022
Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time

Basenine Schema-free, document-oriented streaming database that optimized for monitoring network traffic in real-time. Featured Aspects Has the fastes

UP9 34 May 5, 2022
Dummy - HTTP server for testing cluster deployments

dummy HTTP server for testing cluster deployments. Very small container image fo

Sam Leavens 0 Feb 17, 2022
Tapestry is an underlying distributed object location and retrieval system (DOLR) which can be used to store and locate objects. This distributed system provides an interface for storing and retrieving key-value pairs.

Tapestry This project implements Tapestry, an underlying distributed object location and retrieval system (DOLR) which can be used to store and locate

Han Cai 1 Mar 16, 2022
A Realtime API Gateway used with NATS to build REST, real time, and RPC APIs, where all your clients are synchronized seamlessly.

Realtime API Gateway Synchronize Your Clients Visit Resgate.io for guides, live demos, and resources. Resgate is a Go project implementing a realtime

Resgate.io - Synchronize Your Clients 567 Apr 13, 2022
Anaximander is an ISP probing tool implementing several reduction techniques to cut down the number of probes launched in order to map an Autonomous System

Anaximander is an ISP probing tool implementing several reduction techniques to cut down the number of probes launched in order to map an Autonomous System, while still keeping high discovery levels.

null 1 Mar 31, 2022
Simple application in Golang that retrieves your ip and updates your DNS entries automatically each time your IP changes.

DNS-Updater Simple application in Golang that retrieves your ip and updates your DNS entries automatically each time your IP changes. Motivation Havin

42_atomys 9 Mar 10, 2022
Swiss Army knife Proxy tool for HTTP/HTTPS traffic capture, manipulation, and replay on the go.

Features • Installation • Usage • Running Proxify • Installing SSL Certificate • Applications of Proxify • Join Discord Swiss Army Knife Proxy for rap

ProjectDiscovery 1.6k May 3, 2022
[WIP] gg is a portable tool to redirect the traffic of a given program to your modern proxy without installing any other programs.

gg gg (go-graft), was inspired by graftcp. go-graft is a pure golang implementation with more useful features. TODO: Use system DNS as the fallback. R

mzz 171 May 7, 2022
Hybridnet is an open source container networking solution, integrated with Kubernetes and used officially by following well-known PaaS platforms

Hybridnet What is Hybridnet? Hybridnet is an open source container networking solution, integrated with Kubernetes and used officially by following we

Alibaba 115 May 10, 2022
A minimal analytics package to start collecting traffic data without client dependencies.

go-web-analytics A minimal analytics package to start collecting traffic data without client dependencies. Logging incoming requests import "github.co

Jake Kalstad 0 Nov 23, 2021
Transfer 10Gbps http traffic over 1Gbps networks :)

httpteleport Teleports 10Gbps http traffic over 1Gbps networks. Built on top of fastrpc. Use cases httpteleport may significantly reduce inter-server

Aliaksandr Valialkin 445 Apr 26, 2022