A CLI to turn Go's benchmark output into pictures

Overview

benchdraw

CircleCI GoDoc codecov

benchdraw allows you to make easy to read picture plots from data in Go's benchmark format, implemented in pure Go.

Benchdraw does not try to be as configurable or good looking as gnuplot. It only intends to produce good enough pictures for the most common cases that users can generate with minimal effort.

Install

go get github.com/cep21/benchdraw

Usage

First generate some benchmark data by running your benchmarks and sending them to a file. Here is what my makefile looks like:

bench:
    go test -v -benchmem -run=^$$ -bench=. ./... > benchmark.txt

Then, run benchdraw against benchmark.txt to create pictures. benchdraw expects that you name your benchmarks as described by https://github.com/golang/proposal/blob/master/design/14313-benchmark-format.md

Importantly, that you use key=value format to group your sub benchmarks and that / divides the key space. For example

BenchmarkDecode/text=digits/level=speed/size=1e4-8   	     100	    154125 ns/op	  64.88 MB/s	   40418 B/op	       7 allocs/op
BenchmarkDecode/text=digits/level=speed/size=1e5-8   	      10	   1367632 ns/op	  73.12 MB/s	   41356 B/op	      14 allocs/op
BenchmarkDecode/text=digits/level=speed/size=1e6-8   	       1	  13879794 ns/op	  72.05 MB/s	   52056 B/op	      94 allocs/op
BenchmarkDecode/text=digits/level=default/size=1e4-8 	     100	    147551 ns/op	  67.77 MB/s	   40418 B/op	       8 allocs/op
BenchmarkDecode/text=digits/level=default/size=1e5-8 	      10	   1197672 ns/op	  83.50 MB/s	   41508 B/op	      13 allocs/op

There are example pictures inside examples and example benchmark results inside testdata. Run make draw_examples to see all the examples drawn.

Simple example

Here we filter the benchmarks to just the ones named "BenchmarkTdigest_Add" and plot the tag "source" as our X dimension. The default Y dimension is ns/op.

# Sample line from simpleres.txt
# BenchmarkTdigest_Add/source=linear/digest=caio-8 	 1299153	       932 ns/op	      33 B/op	       0 allocs/op
#
./benchdraw --filter="BenchmarkTdigest_Add" --x=source < ./testdata/simpleres.txt > ./examples/out0.svg

firts example

Reading from a file

You can run the same simple example another way, passing directly the input and output file names

./benchdraw --filter="BenchmarkTdigest_Add" --x=source --group="digest" --input=./testdata/simpleres.txt --output=./examples/out1.svg

firts example

Plot another metric

You can set the "y" value to plot. Here I set it to allocs/op. Notice how the table at the top right "digits/twain" bleeds into the bar graph. For this case, it may be better to use a line output (see below).

# Sample line from decodeexample.txt
# BenchmarkDecode/text=digits/level=best/size=1e5-8    	      10	   1185527 ns/op	  84.35 MB/s	   41508 B/op	      13 allocs/op
#
./benchdraw --filter="BenchmarkDecode/level=best" --x=size --y="allocs/op" --input=./testdata/decodeexample.txt --output=./examples/sample_allocs.svg

line output

Line output

Bar graphs are the default, but you can also output line charts. This can help if the table at the top gets in the way.

./benchdraw --filter="BenchmarkDecode/level=best" --x=size --plot=line --y="allocs/op" --input=./testdata/decodeexample.txt --output=./examples/sample_line.svg

line output

Here is another example line output

# Sample line from benchresult.txt
# BenchmarkCorrectness/size=1000000/source=rand/digest=caio/quant=0.000000-8                  	1000000000	         0.0649 ns/op	       100 %correct	       0 B/op	       0 allocs/op
    ./benchdraw --filter="BenchmarkCorrectness/size=1000000/quant=0.000000" --x=source --plot=line --y=%correct --v=4 --input=./testdata/benchresult.txt --output=./examples/sample_line3.svg

line output

Custom metrics

You can also plot benchmark results of custom metrics. Here I plot the custom metric %correct.

	./benchdraw --filter="BenchmarkCorrectness/size=1000000/digest=segmentio" --x=quant --y=%correct --v=4 --input=./testdata/benchresult.txt --output=./examples/segmentio_correct.svg

line output

Grouping

Sometimes your benchmarks have too many dimensions to read easily

	./benchdraw --filter="BenchmarkCorrectness/size=1000000" --x=quant --y=%correct --v=4 --input=./testdata/benchresult.txt --output=./examples/too_many.svg

line output

You can group those bars. By default, grouping aggregates with a mean(average) function. Here I try to show, on average, how correct two different implementations of the tdigest algorithm are as quantiles increase.

./benchdraw --filter="BenchmarkCorrectness/size=1000000" --x=quant --y=%correct --group="digest" --v=4 --input=./testdata/benchresult.txt --output=./examples/grouped.svg

line output

Using benchmark key/value tags

You can use the benchmark format's support for tagged data to chart changes over time. Here is an example file.

commit: 7cd9055
BenchmarkDecode/text=digits/level=speed/size=1e4-8   	     100	    154125 ns/op	  64.88 MB/s	   40418 B/op	       7 allocs/op
commit: 3ab3ace
BenchmarkDecode/text=digits/level=speed/size=1e4-8   	     100	    154125 ns/op	  64.88 MB/s	   40418 B/op	       7 allocs/op
commit: 92ae1af
BenchmarkDecode/text=digits/level=speed/size=1e4-8   	     100	    167185 ns/op	  64.88 MB/s	   40418 B/op	       7 allocs/op
commit: 920af9b
BenchmarkDecode/text=digits/level=speed/size=1e4-8   	     100	    168129 ns/op	  64.88 MB/s	   40418 B/op	       7 allocs/op
commit: a1b93a0
BenchmarkDecode/text=digits/level=speed/size=1e4-8   	     100	    140125 ns/op	  64.88 MB/s	   40418 B/op	       7 allocs/op

Notice how each benchmark run contains the tag "commit". We can use commit as our x axis

./benchdraw --filter="BenchmarkDecode" --x=commit --plot=line --input=./testdata/encodeovertime.txt --output=./examples/comits.svg

line output

Parameter explanations

x (required)

A x parameter should be a tag or dimension of your benchmark and will get distributed on the X axis of your image.

y

A y parameter should be a unit of one of your benchmark runs The default is "ns/op".

filter

A filter limits which benchmarks we consider. It is in a similar format to the expected benchmark output. Each / segment is a filter. If the filter has =, then it is an exact match. If the filter has just a word, then it's an existence match for that word. For example BenchmarkDecode/text=digits matches

  • BenchmarkDecode/name=bob/text=digits
  • BenchmarkDecode/text=digits.

Does not match

  • BenchmarkDecode
  • BenchmarkDecode/text=sawyer

Design Rational

The tool will never be as powerful as gnuplot. My hope was to capture the most common cases.

Contributing

Contributions welcome! Submit a pull request on github and make sure your code passes make lint test. For large changes, I strongly recommend creating an issue on GitHub first to confirm your change will be accepted before writing a lot of code. GitHub issues are also recommended, at your discretion, for smaller changes or questions.

License

This library is licensed under the Apache 2.0 License.

Issues
  • Unable to go get

    Unable to go get

    opened by simar7 1
  • Panics when running

    Panics when running

    I get a panic when running with the following benchmark.

    [email protected]:~/go/src/github.com/gopherguides/learn/_training/advanced/benchmarking/src/fib_new (master % u= origin/master)
    $ cat benchmark.txt
    goos: darwin
    goarch: amd64
    pkg: github.com/gopherguides/learn/_training/advanced/benchmarking/src/fib_new
    BenchmarkFib-8          92689851                12.8 ns/op             0 B/op          0 allocs/op
    PASS
    ok      github.com/gopherguides/learn/_training/advanced/benchmarking/src/fib_new       1.211s
    
    [email protected]:~/go/src/github.com/gopherguides/learn/_training/advanced/benchmarking/src/fib_new (master % u= origin/master)
    $ benchdraw --x=source -filter=BencharkFib < ./benchmark.txt > ./out0.svg
    panic: runtime error: index out of range [0] with length 0
    
    goroutine 1 [running]:
    gonum.org/v1/plot.(*Plot).NominalX(0xc00004b000, 0x0, 0x0, 0x0)
            /Users/corylanou/go/src/gonum.org/v1/plot/plot.go:393 +0x206
    github.com/cep21/benchdraw/internal.(*Plotter).createPlot(0x18269c0, 0x0, 0xc0000a60a0, 0x1, 0x7ffeefbff555, 0xb, 0x7ffeefbff546, 0x6, 0x137abb9, 0x5, ...)
            /Users/corylanou/go/src/github.com/cep21/benchdraw/internal/plotter.go:78 +0x169
    github.com/cep21/benchdraw/internal.(*Plotter).Plot(0x18269c0, 0x0, 0xc0000a60a0, 0x13eda80, 0xc00000e010, 0x137a3d7, 0x3, 0x1, 0x7ffeefbff555, 0xb, ...)
            /Users/corylanou/go/src/github.com/cep21/benchdraw/internal/plotter.go:40 +0x103
    main.(*Application).run(0x18269c0, 0x0, 0x0)
            /Users/corylanou/go/src/github.com/cep21/benchdraw/main.go:196 +0xdc7
    main.(*Application).main(0x18269c0)
            /Users/corylanou/go/src/github.com/cep21/benchdraw/main.go:131 +0x2f
    main.main()
            /Users/corylanou/go/src/github.com/cep21/benchdraw/main.go:217 +0x2d
    
    bug help wanted 
    opened by corylanou 5
Releases(v0.1.0)
Owner
Jack Lindamood
Jack Lindamood
webify - Turn functions and commands into web services

webify is a very basic CGI server which forwards all requests to a single script. A design goal is to be as zero-config as possible.

Michael Alexander 901 May 8, 2022
Turn .mp3 files in current directory to a podcast feed just one command.

dir2cast Turn .mp3 files in current directory to a podcast feed just one command. Then you can subscribe to it with your favorite podcast client, down

王子亭 3 Apr 24, 2022
A simple way of sending messages from the CLI output to your Discord channel with webhook.

discat A simple way of sending messages from the CLI output to your Discord channel with webhook. Actually, this is a fork version of slackcat that I

dw1 28 Apr 11, 2022
CLI to output stargazer ⭐️ histogram for a GitHub repository

bestgo bestgo is a CLI that pulls live data from https://api.bestofgo.dev (UI coming soon). This is an application that scrapes GitHub data for Go rep

Michael Fridman 7 Apr 15, 2022
A CLI tool for running Go commands with colorized output

Goli Goli is a CLI Tool for running Go commands with colorized output. Note: Goli is still a WIP. It has very basic commands and limitations. Feel fre

Arthur Diniz 14 May 3, 2022
Soren L. Hansen 1.5k May 9, 2022
CLI tool (hcron) and Go library (cron) to convert CRON expression into human readable description.

cron cron is a Go library that parses a cron expression and outputs a human readable description of the cron schedule. For example, given the expressi

Quy Le 69 Apr 11, 2022
✏️ CLI tool to split a file into smaller sub-files

filesplit CLI tool to split a file into smaller sub-files Build $ go build Usage filesplit [mode] [-F, --file] [-N, --number] Examples # Split foo.txt

Martin Sit 5 Apr 20, 2022
Cli tool to translate text from any language into german

GERMAN A cli tool for converting text into German. Build Locally $> go build $> go install Dependencies To execute successfully, a free tier DEEPL API

Kieran O'Sullivan 3 Jan 24, 2022
Utility CLI to convert Spring Boot Yaml configuration into external configuration

boot-config-export Utility CLI to convert Spring Boot Yaml configuration into external configuration (as environment variables). The variables are tra

Lucian 0 Nov 17, 2021
A CLI tool which loads data from yaml files into the Google Cloud Spanner tables

splanter A CLI tool which loads data from yaml files into the Google Cloud Spanner tables (mainly for the development).

Yuki Ito 4 Nov 29, 2021
Utilities to prettify console output of tables, lists, progress-bars, text, etc.

go-pretty Utilities to prettify console output of tables, lists, progress-bars, text, etc. Table Pretty-print tables into ASCII/Unicode strings.

Naveen Mahalingam 1.3k May 9, 2022
🎨 Terminal color rendering library, support 8/16 colors, 256 colors, RGB color rendering output, support Print/Sprintf methods, compatible with Windows.

?? Terminal color rendering library, support 8/16 colors, 256 colors, RGB color rendering output, support Print/Sprintf methods, compatible with Windows. GO CLI 控制台颜色渲染工具库,支持16色,256色,RGB色彩渲染输出,使用类似于 Print/Sprintf,兼容并支持 Windows 环境的色彩渲染

Gookit 1.1k May 7, 2022
Go simple progress bar writing to output

?? progress-go Go simple progress bar writing to output ?? ABOUT Contributors: Rafał Lorenz Want to contribute ? Feel free to send pull requests! Have

Rafał Lorenz 52 Feb 24, 2022
Read from standard input and output a Haags translation of the given input.

haags Read from standard input and output a Haags translation of the given input. Building make && sudo make install You may also run go build on syst

Kees van Voorthuizen 3 Sep 13, 2021
Fetches the output for an AWS SSM command for every target

AWSCommander Fetches the output for an AWS SSM command for every target. Optionally outputs as HTML. Examples Get command from Tokio Japan as text AWS

Michael Bradley 0 Nov 24, 2021
OS Command Execution and Gets the output

OS Command Execution and Gets the output https://stackoverflow.com/questions/24095661/os-exec-sudo-command-in-go/24095983#24095983 https://stackoverfl

MD MOSTAIN BILLAH 4 Dec 24, 2021
Chalk is a Go Package which can be used for making terminal output more vibrant with text colors, text styles and background colors.

Chalk Chalk is a Go Package which can be used for making terminal output more vibrant with text colors, text styles and background colors. Documentati

null 2 Jan 2, 2022
Project-2 - Create a project that calls service created above, pass text and prints JSON output returned from the service

Project Assignment Steps to run the project: First Download the repo present her

null 0 Jan 27, 2022