HTTP load testing tool and library. It's over 9000!

Overview

Vegeta Build Status Go Report Card PkgGoDev Gitter Donate

Vegeta is a versatile HTTP load testing tool built out of a need to drill HTTP services with a constant request rate. It can be used both as a command line utility and a library.

Vegeta

Install

Pre-compiled executables

Get them here.

Homebrew on Mac OS X

You can install Vegeta using the Homebrew package manager on Mac OS X:

$ brew update && brew install vegeta

Source

You need go installed and GOBIN in your PATH. Once that is done, run the command:

$ go get -u github.com/tsenart/vegeta

Versioning

Both the library and the CLI are versioned with SemVer v2.0.0.

After v8.0.0, the two components are versioned separately to better isolate breaking changes to each.

CLI releases are tagged with cli/vMAJOR.MINOR.PATCH and published on the Github releases page. As for the library, new versions are tagged with both lib/vMAJOR.MINOR.PATCH and vMAJOR.MINOR.PATCH. The latter tag is required for compatibility with go mod.

Contributing

See CONTRIBUTING.md.

Usage manual

Usage: vegeta [global flags] <command> [command flags]

global flags:
  -cpus int
    	Number of CPUs to use (defaults to the number of CPUs you have)
  -profile string
    	Enable profiling of [cpu, heap]
  -version
    	Print version and exit

attack command:
  -body string
    	Requests body file
  -cert string
    	TLS client PEM encoded certificate file
  -chunked
    	Send body with chunked transfer encoding
  -connections int
    	Max open idle connections per target host (default 10000)
  -duration duration
    	Duration of the test [0 = forever]
  -format string
    	Targets format [http, json] (default "http")
  -h2c
    	Send HTTP/2 requests without TLS encryption
  -header value
    	Request header
  -http2
    	Send HTTP/2 requests when supported by the server (default true)
  -insecure
    	Ignore invalid server TLS certificates
  -keepalive
    	Use persistent connections (default true)
  -key string
    	TLS client PEM encoded private key file
  -laddr value
    	Local IP address (default 0.0.0.0)
  -lazy
    	Read targets lazily
  -max-body value
    	Maximum number of bytes to capture from response bodies. [-1 = no limit] (default -1)
  -max-workers uint
    	Maximum number of workers (default 18446744073709551615)
  -name string
    	Attack name
  -output string
    	Output file (default "stdout")
  -proxy-header value
    	Proxy CONNECT header
  -rate value
    	Number of requests per time unit [0 = infinity] (default 50/1s)
  -redirects int
    	Number of redirects to follow. -1 will not follow but marks as success (default 10)
  -resolvers value
    	List of addresses (ip:port) to use for DNS resolution. Disables use of local system DNS. (comma separated list)
  -root-certs value
    	TLS root certificate files (comma separated list)
  -targets string
    	Targets file (default "stdin")
  -timeout duration
    	Requests timeout (default 30s)
  -unix-socket string
    	Connect over a unix socket. This overrides the host address in target URLs
  -workers uint
    	Initial number of workers (default 10)

encode command:
  -output string
    	Output file (default "stdout")
  -to string
    	Output encoding [csv, gob, json] (default "json")

plot command:
  -output string
    	Output file (default "stdout")
  -threshold int
    	Threshold of data points above which series are downsampled. (default 4000)
  -title string
    	Title and header of the resulting HTML page (default "Vegeta Plot")

report command:
  -buckets string
    	Histogram buckets, e.g.: "[0,1ms,10ms]"
  -every duration
    	Report interval
  -output string
    	Output file (default "stdout")
  -type string
    	Report type to generate [text, json, hist[buckets], hdrplot] (default "text")

examples:
  echo "GET http://localhost/" | vegeta attack -duration=5s | tee results.bin | vegeta report
  vegeta report -type=json results.bin > metrics.json
  cat results.bin | vegeta plot > plot.html
  cat results.bin | vegeta report -type="hist[0,100ms,200ms,300ms]"

-cpus

Specifies the number of CPUs to be used internally. It defaults to the amount of CPUs available in the system.

-profile

Specifies which profiler to enable during execution. Both cpu and heap profiles are supported. It defaults to none.

-version

Prints the version and exits.

attack command

-body

Specifies the file whose content will be set as the body of every request unless overridden per attack target, see -targets.

-cert

Specifies the PEM encoded TLS client certificate file to be used with HTTPS requests. If -key isn't specified, it will be set to the value of this flag.

-chunked

Specifies whether to send request bodies with the chunked transfer encoding.

-connections

Specifies the maximum number of idle open connections per target host.

-duration

Specifies the amount of time to issue request to the targets. The internal concurrency structure's setup has this value as a variable. The actual run time of the test can be longer than specified due to the responses delay. Use 0 for an infinite attack.

-format

Specifies the targets format to decode.

json format

The JSON format makes integration with programs that produce targets dynamically easier. Each target is one JSON object in its own line. The method and url fields are required. If present, the body field must be base64 encoded. The generated JSON Schema defines the format in detail.

jq -ncM '{method: "GET", url: "http://goku", body: "Punch!" | @base64, header: {"Content-Type": ["text/plain"]}}' |
  vegeta attack -format=json -rate=100 | vegeta encode
http format

The http format almost resembles the plain-text HTTP message format defined in RFC 2616 but it doesn't support in-line HTTP bodies, only references to files that are loaded and used as request bodies (as exemplified below).

Although targets in this format can be produced by other programs, it was originally meant to be used by people writing targets by hand for simple use cases.

Here are a few examples of valid targets files in the http format:

Simple targets
GET http://goku:9090/path/to/dragon?item=ball
GET http://user:[email protected]:9090/path/to
HEAD http://goku:9090/path/to/success
Targets with custom headers
GET http://user:[email protected]:9090/path/to
X-Account-ID: 8675309

DELETE http://goku:9090/path/to/remove
Confirmation-Token: 90215
Authorization: Token DEADBEEF
Targets with custom bodies
POST http://goku:9090/things
@/path/to/newthing.json

PATCH http://goku:9090/thing/71988591
@/path/to/thing-71988591.json
Targets with custom bodies and headers
POST http://goku:9090/things
X-Account-ID: 99
@/path/to/newthing.json
Add comments

Lines starting with # are ignored.

# get a dragon ball
GET http://goku:9090/path/to/dragon?item=ball
# specify a test accout
X-Account-ID: 99

-h2c

Specifies that HTTP2 requests are to be sent over TCP without TLS encryption.

-header

Specifies a request header to be used in all targets defined, see -targets. You can specify as many as needed by repeating the flag.

-http2

Specifies whether to enable HTTP/2 requests to servers which support it.

-insecure

Specifies whether to ignore invalid server TLS certificates.

-keepalive

Specifies whether to reuse TCP connections between HTTP requests.

-key

Specifies the PEM encoded TLS client certificate private key file to be used with HTTPS requests.

-laddr

Specifies the local IP address to be used.

-lazy

Specifies whether to read the input targets lazily instead of eagerly. This allows streaming targets into the attack command and reduces memory footprint. The trade-off is one of added latency in each hit against the targets.

-max-body

Specifies the maximum number of bytes to capture from the body of each response. Remaining unread bytes will be fully read but discarded. Set to -1 for no limit. It knows how to intepret values like these:

  • "10 MB" -> 10MB
  • "10240 g" -> 10TB
  • "2000" -> 2000B
  • "1tB" -> 1TB
  • "5 peta" -> 5PB
  • "28 kilobytes" -> 28KB
  • "1 gigabyte" -> 1GB

-name

Specifies the name of the attack to be recorded in responses.

-output

Specifies the output file to which the binary results will be written to. Made to be piped to the report command input. Defaults to stdout.

-rate

Specifies the request rate per time unit to issue against the targets. The actual request rate can vary slightly due to things like garbage collection, but overall it should stay very close to the specified. If no time unit is provided, 1s is used.

A -rate of 0 or infinity means vegeta will send requests as fast as possible. Use together with -max-workers to model a fixed set of concurrent users sending requests serially (i.e. waiting for a response before sending the next request).

Setting -max-workers to a very high number while setting -rate=0 can result in vegeta consuming too many resources and crashing. Use with care.

-redirects

Specifies the max number of redirects followed on each request. The default is 10. When the value is -1, redirects are not followed but the response is marked as successful.

-resolvers

Specifies custom DNS resolver addresses to use for name resolution instead of the ones configured by the operating system. Works only on non Windows systems.

-root-certs

Specifies the trusted TLS root CAs certificate files as a comma separated list. If unspecified, the default system CAs certificates will be used.

-targets

Specifies the file from which to read targets, defaulting to stdin. See the -format section to learn about the different target formats.

-timeout

Specifies the timeout for each request. The default is 0 which disables timeouts.

-workers

Specifies the initial number of workers used in the attack. The actual number of workers will increase if necessary in order to sustain the requested rate, unless it'd go beyond -max-workers.

-max-workers

Specifies the maximum number of workers used in the attack. It can be used to control the concurrency level used by an attack.

report command

Usage: vegeta report [options] [<file>...]

Outputs a report of attack results.

Arguments:
  <file>  A file with vegeta attack results encoded with one of
          the supported encodings (gob | json | csv) [default: stdin]

Options:
  --type    Which report type to generate (text | json | hist[buckets] | hdrplot).
            [default: text]

  --buckets Histogram buckets, e.g.: '[0,1ms,10ms]'

  --every   Write the report to --output at every given interval (e.g 100ms)
            The default of 0 means the report will only be written after
            all results have been processed. [default: 0]

  --output  Output file [default: stdout]

Examples:
  echo "GET http://:80" | vegeta attack -rate=10/s > results.gob
  echo "GET http://:80" | vegeta attack -rate=100/s | vegeta encode > results.json
  vegeta report results.*

report -type=text

Requests      [total, rate, throughput] 1200, 120.00, 65.87
Duration      [total, attack, wait]     10.094965987s, 9.949883921s, 145.082066ms
Latencies     [min, mean, 50, 95, 99, max]  90.438129ms, 113.172398ms, 108.272568ms, 140.18235ms, 247.771566ms, 264.815246ms
Bytes In      [total, mean]             3714690, 3095.57
Bytes Out     [total, mean]             0, 0.00
Success       [ratio]                   55.42%
Status Codes  [code:count]              0:535  200:665
Error Set:
Get http://localhost:6060: dial tcp 127.0.0.1:6060: connection refused
Get http://localhost:6060: read tcp 127.0.0.1:6060: connection reset by peer
Get http://localhost:6060: dial tcp 127.0.0.1:6060: connection reset by peer
Get http://localhost:6060: write tcp 127.0.0.1:6060: broken pipe
Get http://localhost:6060: net/http: transport closed before response was received
Get http://localhost:6060: http: can't write HTTP request on broken connection

The Requests row shows:

  • The total number of issued requests.
  • The real request rate sustained during the attack period.
  • The throughput of successful requests over the total period.

The Duration row shows:

  • The attack time taken issuing all requests (total - wait)
  • The wait time waiting for the response to the last issued request (total - attack)
  • The total time taken in the attack (attack + wait)

Latency is the amount of time taken for a response to a request to be read (including the -max-body bytes from the response body).

  • min is the minimum latency of all requests in an attack.
  • mean is the arithmetic mean / average of the latencies of all requests in an attack.
  • 50, 90, 95, 99 are the 50th, 90th, 95th and 99th percentiles, respectively, of the latencies of all requests in an attack. To understand more about why these are useful, I recommend this article from @tylertreat.
  • max is the maximum latency of all requests in an attack.

The Bytes In and Bytes Out rows shows:

  • The total number of bytes sent (out) or received (in) with the request or response bodies.
  • The mean number of bytes sent (out) or received (in) with the request or response bodies.

The Success ratio shows the percentage of requests whose responses didn't error and had status codes between 200 and 400 (non-inclusive).

The Status Codes row shows a histogram of status codes. 0 status codes mean a request failed to be sent.

The Error Set shows a unique set of errors returned by all issued requests. These include requests that got non-successful response status code.

report -type=json

All duration like fields are in nanoseconds.

{
  "latencies": {
    "total": 237119463,
    "mean": 2371194,
    "50th": 2854306,
    "90th": 3228223,
    "95th": 3478629,
    "99th": 3530000,
    "max": 3660505,
    "min": 1949582
  },
  "buckets": {
    "0": 9952,
    "1000000": 40,
    "2000000": 6,
    "3000000": 0,
    "4000000": 0,
    "5000000": 2
  },
  "bytes_in": {
    "total": 606700,
    "mean": 6067
  },
  "bytes_out": {
    "total": 0,
    "mean": 0
  },
  "earliest": "2015-09-19T14:45:50.645818631+02:00",
  "latest": "2015-09-19T14:45:51.635818575+02:00",
  "end": "2015-09-19T14:45:51.639325797+02:00",
  "duration": 989999944,
  "wait": 3507222,
  "requests": 100,
  "rate": 101.01010672380401,
  "throughput": 101.00012489812,
  "success": 1,
  "status_codes": {
    "200": 100
  },
  "errors": []
}

In the buckets field, each key is a nanosecond value representing the lower bound of a bucket. The upper bound is implied by the next higher bucket. Upper bounds are non-inclusive. The highest bucket is the overflow bucket; it has no upper bound. The values are counts of how many requests fell into that particular bucket. If the -buckets parameter is not present, the buckets field is omitted.

report -type=hist

Computes and prints a text based histogram for the given buckets. Each bucket upper bound is non-inclusive.

cat results.bin | vegeta report -type='hist[0,2ms,4ms,6ms]'
Bucket         #     %       Histogram
[0,     2ms]   6007  32.65%  ########################
[2ms,   4ms]   5505  29.92%  ######################
[4ms,   6ms]   2117  11.51%  ########
[6ms,   +Inf]  4771  25.93%  ###################

report -type=hdrplot

Writes out results in a format plottable by https://hdrhistogram.github.io/HdrHistogram/plotFiles.html.

Value(ms)  Percentile  TotalCount  1/(1-Percentile)
0.076715   0.000000    0           1.000000
0.439370   0.100000    200         1.111111
0.480836   0.200000    400         1.250000
0.495559   0.300000    599         1.428571
0.505101   0.400000    799         1.666667
0.513059   0.500000    999         2.000000
0.516664   0.550000    1099        2.222222
0.520455   0.600000    1199        2.500000
0.525008   0.650000    1299        2.857143
0.530174   0.700000    1399        3.333333
0.534891   0.750000    1499        4.000000
0.537572   0.775000    1548        4.444444
0.540340   0.800000    1598        5.000000
0.543763   0.825000    1648        5.714286
0.547164   0.850000    1698        6.666667
0.551432   0.875000    1748        8.000000
0.553444   0.887500    1773        8.888889
0.555774   0.900000    1798        10.000000
0.558454   0.912500    1823        11.428571
0.562123   0.925000    1848        13.333333
0.565563   0.937500    1873        16.000000
0.567831   0.943750    1886        17.777778
0.570617   0.950000    1898        20.000000
0.574522   0.956250    1911        22.857143
0.579046   0.962500    1923        26.666667
0.584426   0.968750    1936        32.000000
0.586695   0.971875    1942        35.555556
0.590451   0.975000    1948        40.000000
0.597543   0.978125    1954        45.714286
0.605637   0.981250    1961        53.333333
0.613564   0.984375    1967        64.000000
0.620393   0.985938    1970        71.113640
0.629121   0.987500    1973        80.000000
0.638060   0.989062    1976        91.424392
0.648085   0.990625    1979        106.666667
0.659689   0.992188    1982        128.008193
0.665870   0.992969    1984        142.227279
0.672985   0.993750    1986        160.000000
0.680101   0.994531    1987        182.848784
0.687810   0.995313    1989        213.356091
0.695729   0.996094    1990        256.016385
0.730641   0.996484    1991        284.414107
0.785516   0.996875    1992        320.000000
0.840392   0.997266    1993        365.764448
1.009646   0.997656    1993        426.621160
1.347020   0.998047    1994        512.032770
1.515276   0.998242    1994        568.828214
1.683532   0.998437    1995        639.795266
1.887487   0.998633    1995        731.528895
2.106249   0.998828    1996        853.242321
2.325011   0.999023    1996        1023.541453
2.434952   0.999121    1996        1137.656428
2.544894   0.999219    1996        1280.409731
2.589510   0.999316    1997        1461.988304
2.605192   0.999414    1997        1706.484642
2.620873   0.999512    1997        2049.180328
2.628713   0.999561    1997        2277.904328
2.636394   0.999609    1997        2557.544757
2.644234   0.999658    1997        2923.976608
2.652075   0.999707    1997        3412.969283
2.658916   0.999756    1998        4098.360656
2.658916   0.999780    1998        4545.454545
2.658916   0.999805    1998        5128.205128
2.658916   0.999829    1998        5847.953216
2.658916   0.999854    1998        6849.315068
2.658916   0.999878    1998        8196.721311
2.658916   0.999890    1998        9090.909091
2.658916   0.999902    1998        10204.081633
2.658916   0.999915    1998        11764.705882
2.658916   0.999927    1998        13698.630137
2.658916   0.999939    1998        16393.442623
2.658916   0.999945    1998        18181.818182
2.658916   0.999951    1998        20408.163265
2.658916   0.999957    1998        23255.813953
2.658916   0.999963    1998        27027.027027
2.658916   0.999969    1998        32258.064516
2.658916   0.999973    1998        37037.037037
2.658916   0.999976    1998        41666.666667
2.658916   0.999979    1998        47619.047619
2.658916   0.999982    1998        55555.555556
2.658916   0.999985    1998        66666.666667
2.658916   0.999986    1998        71428.571429
2.658916   0.999988    1998        83333.333333
2.658916   0.999989    1998        90909.090909
2.658916   0.999991    1998        111111.111111
2.658916   0.999992    1998        125000.000000
2.658916   0.999993    1998        142857.142858
2.658916   0.999994    1998        166666.666668
2.658916   0.999995    1998        199999.999999
2.658916   0.999996    1998        250000.000000
2.658916   0.999997    1998        333333.333336
2.658916   0.999998    1998        500000.000013
2.658916   0.999999    1998        999999.999971
2.658916   1.000000    1998        10000000.000000

encode command

Usage: vegeta encode [options] [<file>...]

Encodes vegeta attack results from one encoding to another.
The supported encodings are Gob (binary), CSV and JSON.
Each input file may have a different encoding which is detected
automatically.

The CSV encoder doesn't write a header. The columns written by it are:

  1. Unix timestamp in nanoseconds since epoch
  2. HTTP status code
  3. Request latency in nanoseconds
  4. Bytes out
  5. Bytes in
  6. Error
  7. Base64 encoded response body
  8. Attack name
  9. Sequence number of request

Arguments:
  <file>  A file with vegeta attack results encoded with one of
          the supported encodings (gob | json | csv) [default: stdin]

Options:
  --to      Output encoding (gob | json | csv) [default: json]
  --output  Output file [default: stdout]

Examples:
  echo "GET http://:80" | vegeta attack -rate=1/s > results.gob
  cat results.gob | vegeta encode | jq -c 'del(.body)' | vegeta encode -to gob

plot command

Plot

Usage: vegeta plot [options] [<file>...]

Outputs an HTML time series plot of request latencies over time.
The X axis represents elapsed time in seconds from the beginning
of the earliest attack in all input files. The Y axis represents
request latency in milliseconds.

Click and drag to select a region to zoom into. Double click to zoom out.
Choose a different number on the bottom left corner input field
to change the moving average window size (in data points).

Arguments:
  <file>  A file output by running vegeta attack [default: stdin]

Options:
  --title      Title and header of the resulting HTML page.
               [default: Vegeta Plot]
  --threshold  Threshold of data points to downsample series to.
               Series with less than --threshold number of data
               points are not downsampled. [default: 4000]

Examples:
  echo "GET http://:80" | vegeta attack -name=50qps -rate=50 -duration=5s > results.50qps.bin
  cat results.50qps.bin | vegeta plot > plot.50qps.html
  echo "GET http://:80" | vegeta attack -name=100qps -rate=100 -duration=5s > results.100qps.bin
  vegeta plot results.50qps.bin results.100qps.bin > plot.html

Usage: Generated targets

Apart from accepting a static list of targets, Vegeta can be used together with another program that generates them in a streaming fashion. Here's an example of that using the jq utility that generates targets with an incrementing id in their body.

jq -ncM 'while(true; .+1) | {method: "POST", url: "http://:6060", body: {id: .} | @base64 }' | \
  vegeta attack -rate=50/s -lazy -format=json -duration=30s | \
  tee results.bin | \
  vegeta report

Usage: Distributed attacks

Whenever your load test can't be conducted due to Vegeta hitting machine limits such as open files, memory, CPU or network bandwidth, it's a good idea to use Vegeta in a distributed manner.

In a hypothetical scenario where the desired attack rate is 60k requests per second, let's assume we have 3 machines with vegeta installed.

Make sure open file descriptor and process limits are set to a high number for your user on each machine using the ulimit command.

We're ready to start the attack. All we need to do is to divide the intended rate by the number of machines, and use that number on each attack. Here we'll use pdsh for orchestration.

$ PDSH_RCMD_TYPE=ssh pdsh -b -w '10.0.1.1,10.0.2.1,10.0.3.1' \
    'echo "GET http://target/" | vegeta attack -rate=20000 -duration=60s > result.bin'

After the previous command finishes, we can gather the result files to use on our report.

$ for machine in 10.0.1.1 10.0.2.1 10.0.3.1; do
    scp $machine:~/result.bin $machine.bin &
  done

The report command accepts multiple result files. It'll read and sort them by timestamp before generating reports.

vegeta report *.bin

Usage: Real-time Analysis

If you are a happy user of iTerm, you can integrate vegeta with jplot using jaggr to plot a vegeta report in real-time in the comfort of your terminal:

echo 'GET http://localhost:8080' | \
    vegeta attack -rate 5000 -duration 10m | vegeta encode | \
    jaggr @count=rps \
          hist\[100,200,300,400,500\]:code \
          p25,p50,p95:latency \
          sum:bytes_in \
          sum:bytes_out | \
    jplot rps+code.hist.100+code.hist.200+code.hist.300+code.hist.400+code.hist.500 \
          latency.p95+latency.p50+latency.p25 \
          bytes_in.sum+bytes_out.sum

Usage (Library)

The library versioning follows SemVer v2.0.0. Since lib/v9.0.0, the library and cli are versioned separately to better isolate breaking changes to each component.

See Versioning for more details on git tag naming schemes and compatibility with go mod.

package main

import (
  "fmt"
  "time"

  vegeta "github.com/tsenart/vegeta/v12/lib"
)

func main() {
  rate := vegeta.Rate{Freq: 100, Per: time.Second}
  duration := 4 * time.Second
  targeter := vegeta.NewStaticTargeter(vegeta.Target{
    Method: "GET",
    URL:    "http://localhost:9100/",
  })
  attacker := vegeta.NewAttacker()

  var metrics vegeta.Metrics
  for res := range attacker.Attack(targeter, rate, duration, "Big Bang!") {
    metrics.Add(res)
  }
  metrics.Close()

  fmt.Printf("99th percentile: %s\n", metrics.Latencies.P99)
}

Limitations

There will be an upper bound of the supported rate which varies on the machine being used. You could be CPU bound (unlikely), memory bound (more likely) or have system resource limits being reached which ought to be tuned for the process execution. The important limits for us are file descriptors and processes. On a UNIX system you can get and set the current soft-limit values for a user.

$ ulimit -n # file descriptors
2560
$ ulimit -u # processes / threads
709

Just pass a new number as the argument to change it.

License

See LICENSE.

Donate

If you use and love Vegeta, please consider sending some Satoshi to 1MDmKC51ve7Upxt75KoNM6x1qdXHFK6iW2. In case you want to be mentioned as a sponsor, let me know!

Donate Bitcoin

Comments
  • Proposal: ramp-up request rate

    Proposal: ramp-up request rate

    In this AWS document: http://aws.amazon.com/articles/1636185810492479, it states: "We recommend that you increase the load at a rate of no more than 50 percent every five minutes. Both step patterns and linear patterns for load generation should work well with Elastic Load Balancing"

    I thus propose a ramp-up with an the interface @tsenart has proposed on the project Gitter chat:

    vegeta attack -plan="10:20s 100:20s 1000:20s"
    
    opened by ghost 44
  • Continuous fuzzing on Fuzzit

    Continuous fuzzing on Fuzzit

    Integrates with Fuzzit for automated bug discovery. Initially discussed in #429.

    I've tried to fuzz everything that does some parsing. The main target AttackerHTTP decodes fuzz into a valid HTTP response. It stubs the targeter and communicates through a socket file.

    Other targets are:

    • AttackerTCP - Fuzz response handling with unstructured byte stream.
    • HTTPTargeter - Fuzz decoding of a target list in HTTP format.
    • JSONTargeter - Fuzz decoding of a target list in JSON format.
    • ResultsFormatDetection - Fuzz result list format detection.
    • GobDecoder - Fuzz decoding of a result list in gob format.
    • CSVDecoder - Fuzz decoding of a result list in CSV format.
    • JSONDecoder - Fuzz decoding of a result list in JSON format.

    The format detection fuzzer caught one crash in a local run. I'm attaching the data here. The unsuffixed file contains the crashing input. Seems to have run out of memory, maybe a memory leak somewhere.

    crashers.FormatDetection.zip

    There's a vegeta org on Fuzzit all preconfigured and ready to go. If you sign in to set up an account I can ask them to add you to the org. There's a little setup to do:

    • In Fuzzit settings grab an API Key.
    • In repo settings in Travis paste it to envvar FUZZIT_API_KEY.

    Including the "fuzz target parser" issue in the close list because I think this patch fulfills it.

    Closes #429. Closes #140.

    Background

    Enables automated fuzzing. Proposed in #429.

    Checklist

    • [x] Git commit messages conform to community standards.
    • [x] Each Git commit represents meaningful milestones or atomic units of work.
    • [x] Changed or added code is covered by appropriate tests.
    opened by bookmoons 18
  • Add 90th percentile to the results

    Add 90th percentile to the results

    Quite a few tools offer 90th percentile statistic, and that's one metric we do monitor. e.g. jmeter, wrk2, etc. I don't see why vegeta shouldn't include it. Right now 95th % is the closest thing to it, but still that's a higher number.

    I assume I got it reasonable but please review the quantile ratios setup and the expected latencies for each quantile with this addition. Which leads to the below question:

    And on a related note, any reason to use the specific quantiles package instead of writing code from scratch or using another statistical/percentile/quantile library? I came across these in a quick search, not sure how they compare against the current library used.

    https://github.com/montanaflynn/stats/blob/master/percentile.go https://github.com/bmizerany/perks/tree/master/quantile

    I'm not a statistician, so can't say much on this but the current quantile library used at least is a bit confusing to the novice user since we are free to define the ratio's for each percentile - one could make mistakes. Contrast that with percentile libraries for python, node.js/javascript, etc. and it's more straightforward: import library and call a percentile method against an array of values, specifying the desired percentile, no mucking with some percentile/quantile ratio value that maps it to the percentile accuracy.

    opened by daluu 17
  • Support for per-target headers and body contents

    Support for per-target headers and body contents

    Be able to specify headers and/or body contents per target, in addition to specifying them globally.

    A few notes:

    1. If a body file isn't found we skip the target
    2. If we find any per-target headers we use none of the global headers. I didn't have a use case for this, but if it's useful to merge the global and per-target headers it's straightforward and I'd be happy to do so.
    3. Yes, we're reading the full body for all the targets into memory. You can probably use the -lazy option if this is a big problem.
    opened by cwinters 17
  • Include histogram in JSON output

    Include histogram in JSON output

    Background

    See #394

    Interesting cases to test:

    # Main use case
    % vegeta report -type=json -hist='[0,1ms,2ms,3ms,4ms,5ms]' vegeta.bin
    {"latencies": ..., "hist": [0,0,0,56,44,400], ...}
    
    # If -hist is not given, "hist" is omitted
    % vegeta report -type=json vegeta.bin
    
    # -type=hist supports -hist too (for consistency)
    % vegeta report -type=hist -hist='[0,1ms,2ms,3ms,4ms,5ms]' vegeta.bin
    
    # Legacy -type=hist invocation continues to work unchanged
    % vegeta report -type=hist'[0,1ms,2ms,3ms,4ms,5ms]' vegeta.bin
    
    # Help should mention the new parameter
    % vegeta --help
    

    Checklist

    • [ ] Git commit messages conform to community standards.
    • [ ] Each Git commit represents meaningful milestones or atomic units of work.
    • [ ] Changed or added code is covered by appropriate tests.
    opened by fxkr 16
  • attack: Add -resolvers flag to command

    attack: Add -resolvers flag to command

    This adds the ability to override the host operating system resolver by allowing this to be specified on the command line (comma separated list).

    In the event more than one is specified, the resolvers are tried in order in a round robin manner.

    Background

    In some situations it is desirable to override the default host resolver and use a separate host for DNS resolution. This adds the option to specify the dns server on the command line.

    Checklist

    • [x] Git commit messages conform to community standards.
    • [x] Each Git commit represents meaningful milestones or atomic units of work.
    • [x] Changed or added code is covered by appropriate tests.
    opened by nathanejohnson 16
  • JSON targets format

    JSON targets format

    Background

    The original targets format closely resembles plain HTTP/1.x messages. It was meant to be used by humans writing those targets directly on their terminals. With its lack of support for in-line body definitions, this ill-defined legacy format made Vegeta hard to integrate with other programs generating dynamic targets on the fly with distinct request bodies.

    Extending the legacy format to support arbitrary in-line request bodies would require further deviation from the HTTP/1.x message format since, to the best of my knowledge, there is no official plain text framing format that would allow the boundaries between multiple targets with arbitrary request bodies to be easily recognized (and hence, parsed).

    It could be done, certainly, but for the use case of integrating Vegeta with other programs generating targets, I think a simpler alternative JSON based format could be better.

    This PR proposes just such a format and is intended to gather feedback from interested parties.

    It's currently a work-in-progress, but the CLI interface is already specified and documented.

    Please provide feedback: @coxx, @1u0, @mbrookes1304, @cwinters, @xla.

    Related Issues and PRs
    • https://github.com/tsenart/vegeta/pull/148
    • https://github.com/tsenart/vegeta/pull/260
    • https://github.com/tsenart/vegeta/issues/298

    Checklist

    • [x] Git commit messages conform to community standards.
    • [x] Each Git commit represents meaningful milestones or atomic units of work.
    • [x] Changed or added code is covered by appropriate tests.
    opened by tsenart 14
  • Doubt in rate and workers

    Doubt in rate and workers

    One of the doubts I have is the workers flag. For example, I have a rate of 1000 req/s and 1 worker... this mean I'll have one thread sending the 1000 requests? What about 10 workers? Means that I have 10 threads sending 1000 requests in total right? Not each one.

    Is possible to configure Vegeta to have 10 workers sending 1000 req/s each? Not in total.

    opened by bmpandrade 14
  • v12.8.0 cannot be imported as a Go module

    v12.8.0 cannot be imported as a Go module

    Version and Runtime

    v12.8.0

    Expected Behaviour

    Vegeta can be imported as a Go module

    Actual Behaviour

    github.com/tsenart/vegeta v12.8.0+incompatible in go.mod:

    go: github.com/tsenart/[email protected]+incompatible/go.mod: verifying module: github.com/tsenart/[email protected]+incompatible/go.mod: reading https://sum.golang.org/lookup/github.com/tsenart/[email protected]+incompatible: 410 Gone
            server response: not found: github.com/tsenart/[email protected]+incompatible: invalid version: +incompatible suffix not allowed: module contains a go.mod file, so semantic import versioning is required
    

    github.com/tsenart/vegeta v12.8.0 in go.mod:

    go: errors parsing go.mod:
    …/go.mod:8: require github.com/tsenart/vegeta: version "v12.8.0" invalid: module contains a go.mod file, so major version must be compatible: should be v0 or v1, not v12
    

    Steps to Reproduce

    1. git clone https://github.com/mxey/vegeta-go-mod
    2. cd vegeta-go-mod
    3. go build

    Additional Context

    opened by mxey 13
  • ability to set headers and request bodies in targets

    ability to set headers and request bodies in targets

    Continuing with my curl theme :smile: , would be great to be able to specify request headers and request bodies for POST / PUT

    WTB for stress testing apis that require things like Accept headers to select an api version, as well as setting content type for POSTing and PUTing

    opened by sclasen 13
  • Add support for building snaps

    Add support for building snaps

    Tomas, I read your guidelines, so hopefully this PR will meet them.

    I would like to ask you to add snap support for vegeta.

    Snaps are cross-distro software package, with automatic updates. You can make the app once, publish it as snap and not worry about any distro-specific compatibility - this is handled by the snap service on any supported distro - and this is a big family with Ubuntu, Debian, Fedora, SUSE, Manjaro, and many others.

    So this might be a nice opportunity, especially if you're looking to simplify deployment and reach a wider audience.

    On the technical side, I'd like to outline what I did, so if you want to accept this PR, you can do the same and land your snap in the Snap Store (snapcraft.io/store).

    1. Snaps are built using snapcraft, a command line tool.
    2. You can build on various systems, but the simplest and eaiser is to use Ubuntu 18.10 LTS.
    3. The process includes getting snapcraft, building with snapcraft using a snapcraft.yaml file, and finally publishing in the store.

    Install and run snapcraft

    snap install snapcraft --classic --beta git clone https://github.com/igorljubuncic/vegeta.git cd vegeta git checkout add-snapcraft snapcraft

    This command will generate a .snap file, something like vegeta_1.0_amd64.snap.

    Test on your system

    This snap can be installed and tested locally with:

    snap install vegeta_1.0_amd64.snap --dangerous

    You need the --dangerous flag because the snap isn't in the store and yet and has no digital signature.

    Run snap and test behavior

    snap run vegeta

    Register developer account

    Create an account: https://snapcraft.io/account.

    Login & register on the command line

    snapcraft login snapcraft register

    Upload snap to the store

    This is done by pushing the created snap to the store, and you need to choose a channel. We have multiple channels, i.e. risk levels, similar to the traditional deployment phases (alpha, beta, etc.).

    We use a slightly different categorization, with 'edge' for the riskiest, and this is always the best channel to choose when uploading a new snap or new, untested version. You can then promote to 'beta' and 'candidate' and finally to 'stable' once you're happy with how your snap is working.

    snapcraft push vegeta_1.0_amd64.snap --release edge

    Test on a clean machine

    This will install from the store - previously, we tested a local install:

    snap install vegeta --edge

    I hope this covers all the steps, and this is clear and simple. If not, you can always reach out and I can help with the details. It would be really nice to have your app (and not just because of the cool avatar) in the Snap Store. Have a good one.

    opened by igorljubuncic 12
  • Invalid example

    Invalid example

    Version and Runtime

    ~ $ vegeta -version
    Version: 12.8.4
    Commit: 
    Runtime: go1.17.2 darwin/arm64
    Date: 2020-10-11T10:37:33Z
    

    Expected Behaviour

    An example has been given that is not valid

    Actual Behaviour

    Returns

    encode: can't detect encoding of "results.1.bin"
    

    Steps to Reproduce

    #!/bin/bash                                                                                                                                                                       
                                                                                                                                                                                      
    echo "GET https://www.stage.bbc.co.uk/" | vegeta attack -duration=5s -rate=10/s > results.gob                                                                                     
    echo "GET https://www.stage.bbc.co.uk/" | vegeta attack -duration=5s -rate=100/s | vegeta encode > results.json                                                                   
    vegeta report results.* 
    

    The actual example given starts here: https://github.com/tsenart/vegeta/blob/d73edf2bc2663d83848da2a97a8401a7ed1440bc/encode.go#L50

    opened by kazimir-malevich 0
  • Vegeta load test all requests in queue and experience high latency on the client side

    Vegeta load test all requests in queue and experience high latency on the client side

    It seems in my vegeta load test, all the requests are waiting in queue. And there's no CPU usage or scale up/down action until we change vegeta request timeout to above 60sec. All the requests show status 200 on server side, and I'm curious about why the requests are in queue. On the contrary, when I ran load test with same rps rate on locust for the same deployment, my result shows low latency.

    rps_30_vegeta_timeout_30s_results Results_30rps_Vegeta_timout_30s

    rps_30_vegeta_timeout_90s_results Results_30rps_Vegeta_timout_90s

    user_30_locust

    opened by Samuel-Xu-1996 0
  • Vegeta real-time not working

    Vegeta real-time not working

    Version and Runtime

    Version: 12.8.4
    Commit: 
    Runtime: go1.17.2 darwin/arm64
    Date: 2020-10-11T10:37:33Z
    

    Expected Behaviour

    I'm trying to use real-time analysis as in readme, but not working as expected. I'm using iTerm on OSx (M1 chip), installed jaggr and jplot with brew, and receiving a broken pipe error.

    Steps to Reproduce

    The full initial command I tried:

    echo 'GET http://localhost:3000/tablet-ipad/tablet-lenovo-tab-p11-za940394br-64gb-11-android-13-mp' | \
          vegeta attack -rate 60/1m -duration 5s | vegeta encode | \
          jaggr @count=rps \
                hist\[100,200,300,400,500\]:code \
                p25,p50,p95:latency \
                sum:bytes_in \
                sum:bytes_out | \
          jplot rps+code.hist.100+code.hist.200+code.hist.300+code.hist.400+code.hist.500 \
                latency.p95+latency.p50+latency.p25 \
                bytes_in.sum+bytes_out.sum
    

    And get:

    [1]    13551 done                echo  |
           13552 broken pipe         vegeta attack -rate 60/1m -duration 5s |
           13553 broken pipe         vegeta encode |
           13554 segmentation fault  jaggr @count=rps hist\[100,200,300,400,500\]:code p25,p50,p95:latency   |
           13555 segmentation fault  jplot  latency.p95+latency.p50+latency.p25 bytes_in.sum+bytes_out.sum
    

    Then I tried just the first part (with jaggr only):

    echo 'GET http://localhost:3000/tablet-ipad/tablet-lenovo-tab-p11-za940394br-64gb-11-android-13-mp' | \
          vegeta attack -rate 60/1m -duration 5s | vegeta encode | \
          jaggr @count=rps \
                hist\[100,200,300,400,500\]:code \
                p25,p50,p95:latency \
                sum:bytes_in \
                sum:bytes_out
    

    With same result:

    [1]    13572 done                echo  |
           13573 broken pipe         vegeta attack -rate 60/1m -duration 5s |
           13574 broken pipe         vegeta encode |
           13575 segmentation fault  jaggr @count=rps hist\[100,200,300,400,500\]:code p25,p50,p95:latency
    

    Then i runned only the vegeta command saving the output to a json file output.txt (renamed to txt because github do not support json):

    echo 'GET http://localhost:3000/tablet-ipad/tablet-lenovo-tab-p11-za940394br-64gb-11-android-13-mp' | \
          vegeta attack -rate 60/1m -duration 5s | vegeta encode >> output.json
    

    I realized this file do not put commas, I don't know if this is expected.

    PS: I tried replacing the url http://localhost:3000 to https://google.com and I get the same errors.

    Hope someone can help!

    opened by wac2007 1
  • How does vegeta adjust connection creation?

    How does vegeta adjust connection creation?

    I did some stress testing in Kubernets via vegeta

    1. test in cluster with internal K8s service endpoint, there are only two connections and the test can finished
    watch -d ss -s
    Total: 3
    TCP:   110 (estab 2, closed 108, orphaned 1, timewait 2)
    
    Transport Total     IP        IPv6
    RAW	  0         0         0
    UDP	  0         0         0
    TCP       2         2         0
    INET	  2         2         0
    FRAG	  0         0         0
    
    1. Using same rate, test with ingress endpoint through nginx controller, there are so many connections vegeta created and the test cannot finished
    watch -d ss -s
    Total: 13175
    TCP:   24640 (estab 10833, closed 11470, orphaned 0, timewait 0)
    
    Transport Total     IP        IPv6
    RAW	  0         0         0
    UDP	  0         0         0
    TCP	  13170     13170     0
    INET	  13170     13170     0
    FRAG	  0         0         0
    

    I guess testing with ingress endpoint, will through SNAT->LB->Ingress controller->SVC, the latency increased, for guaranteeing the request rate, vegeta need to create more tcp connections to send the request. Is my understanding right? Even if I set the -max-connections at 10, the vegeta also create connections more then it. Is that mean for guaranteeing the request rate, vegeta will create connections as closer as the -max-connections?

    opened by fred214 0
  • Add a new report type - expoHist

    Add a new report type - expoHist

    Provide an alternative way generating buckets in histogram

    Background

    • Provide a new way generating buckets, an attribute in the command 'report': exponential sequence.
    • Append in README.md with explanation about usage of '--type expoHist'.

    This PR manages to improve the distortion of which executing command 'report -type=hist[]' causes, the distortion that, if arithmetic x-axis was applied, the result set having approximate latency might be cohered into single bucket while trivial cases, including P99 or P999, might be dispersed into trivial buckets. However, Labeling x-axis in form of exponential sequence separates the high-density result set while at the same time cohering scattered cases, making graph more readable.

    Example case below both comes with 200 requests, with 20 QPS and lasting 10 seconds, presenting through exponential-sequence bucket on the left side and arithmetic bucket on the other side.

    Common phenomenon in which large amount of request data are under stress testing usually causes distortion. In practice, for example, the amount of requests goes above 100,000 if test comes with 10,000 QPS lasting 10 seconds. Then, the outcome of graph would be presented with a normal distribution( or right/left-skewed distribution), meaning that 65%, even 90%, above of cases fall in the certain range, therefore causing distortion on graph. Further, P99, the other result set worth to be observed as well, could be underestimated if the amount of them, usually, turns to be scarce.

    Therefore, in order to prevent such readability-relative issues, an alternative method printing histogram comes into play: exponential sequence.

    Checklist

    • [x] Git commit messages conform to community standards.
    • [x] Each Git commit represents meaningful milestones or atomic units of work.
    • [x] Changed or added code is covered by appropriate tests.
    opened by wu0h961738 0
Releases(v12.8.4)
Owner
Tomás Senart
Lucid ramblings in time and space.
Tomás Senart
Cloud Spanner load generator to load test your application and pre-warm the database before launch

GCSB GCSB Quickstart Create a test table Load data into table Run a load test Operations Load Single table load Multiple table load Loading into inter

Cloud Spanner Ecosystem 21 Jul 13, 2022
Check-load - Simple cross-platform load average check

Sensu load average check Table of Contents Overview Usage examples Configuration

KOHMURA Jin 0 Jun 16, 2022
Ddosify - High-performance load testing tool

Ddosify - High-performance load testing tool Features ✔️ Protocol Agnostic - Currently supporting HTTP, HTTPS, HTTP/2. Other protocols are on the way.

Ddosify 4.9k Sep 30, 2022
Hive-fleet: a distributed, scalable load-testing tool built in go that leverages Google Cloud Functions

hive-fleet hive-fleet is a distributed, scalable load-testing tool, built on top

Cristian Glavan 5 Jan 27, 2022
Simple Golang Load testing app built on top of vegeta

LOVE AND WAR : Give Your App Love By Unleashing War Simple load testing app to test your http services Installation Build docker image: docker build -

Raymond Gitonga 1 Oct 26, 2021
Trade Matching / Transaction System Load Testing Solution

Load Generation System for Trade Matching Systems Operation Users select one of the following options from thew Test Management Portal: Generate a new

Traiano Welcome 1 Feb 25, 2022
A yaml data-driven testing format together with golang testing library

Specimen Yaml-based data-driven testing Specimen is a yaml data format for data-driven testing. This enforces separation between feature being tested

Design it, Run it 0 Jan 31, 2022
HTTP mock for Golang: record and replay HTTP/HTTPS interactions for offline testing

govcr A Word Of Warning I'm in the process of partly rewriting govcr to offer better support for cassette mutations. This is necessary because when I

Seb C 131 Sep 27, 2022
siusiu (suite-suite harmonics) a suite used to manage the suite, designed to free penetration testing engineers from learning and using various security tools, reducing the time and effort spent by penetration testing engineers on installing tools, remembering how to use tools.

siusiu (suite-suite harmonics) a suite used to manage the suite, designed to free penetration testing engineers from learning and using various security tools, reducing the time and effort spent by penetration testing engineers on installing tools, remembering how to use tools.

Re 277 Sep 30, 2022
HTTP load generator, ApacheBench (ab) replacement, formerly known as rakyll/boom

hey is a tiny program that sends some load to a web application. hey was originally called boom and was influenced from Tarek Ziade's tool at tarekzia

Jaana Dogan 14.3k Sep 21, 2022
Ditto is a CLI testing tool that helps you verify if multiple HTTP endpoints have the same outputs.

Ditto is a CLI testing tool that helps you verify if multiple HTTP endpoints have the same outputs.

Cristopher 1 Nov 24, 2021
Client tool for testing HTTP server timeouts

HTTP timeout test client While testing Go HTTP server timeouts I wrote this little tool to help me test. It allows for slowing down header write and b

Adam Pritchard 8 Sep 21, 2022
Tesuto - a little library for testing against HTTP services

tesuto import "github.com/guregu/tesuto" tesuto is a little library for testing

Greg 2 Jan 18, 2022
Extremely flexible golang deep comparison, extends the go testing package and tests HTTP APIs

go-testdeep Extremely flexible golang deep comparison, extends the go testing package. Latest news Synopsis Description Installation Functions Availab

Maxime Soulé 316 Sep 17, 2022
End-to-end HTTP and REST API testing for Go.

httpexpect Concise, declarative, and easy to use end-to-end HTTP and REST API testing for Go (golang). Basically, httpexpect is a set of chainable bui

Victor Gaydov 2k Oct 1, 2022
HTTP traffic mocking and testing made easy in Go ༼ʘ̚ل͜ʘ̚༽

gock Versatile HTTP mocking made easy in Go that works with any net/http based stdlib implementation. Heavily inspired by nock. There is also its Pyth

Tom 1.7k Sep 23, 2022
This is a DNS over TLS stress test tool

This is a DNS over TLS stress test tool

Cody Chen 3 Apr 17, 2022
Expressive end-to-end HTTP API testing made easy in Go

baloo Expressive and versatile end-to-end HTTP API testing made easy in Go (golang), built on top of gentleman HTTP client toolkit. Take a look to the

Tom 741 Sep 28, 2022
Golang HTTP client testing framework

flute Golang HTTP client testing framework Presentation https://speakerdeck.com/szksh/flute-golang-http-client-testing-framework Overview flute is the

Shunsuke Suzuki 17 Sep 27, 2022