Continuous Benchmark for Go Project

Overview

GitHub release Go Report Card MIT License

Abstract

cob compares benchmarks between the latest commit (HEAD) and the previous commit (HEAD{@1}). The program will fail if the change in score is worse than the threshold. This tools is suitable for CI/CD to detect a regression of a performance automatically.

cob runs go test -bench before and after commit internally, so it depends on go command.

CAUTION: Note that git reset is executed when you run cob. You should commit all changes before running cob.

Table of Contents

Continuous Integration (CI)

See cob-example for details.

GitHub Actions

name: Bench
on: [push, pull_request]
jobs:
  test:
    name: Bench
    runs-on: ubuntu-latest
    steps:

    - name: Set up Go 1.13
      uses: actions/[email protected]
      with:
        go-version: 1.13
      id: go

    - name: Check out code into the Go module directory
      uses: actions/[email protected]

    - name: Install cob
      run: curl -sfL https://raw.githubusercontent.com/knqyf263/cob/master/install.sh | sudo sh -s -- -b /usr/local/bin

    - name: Run Benchmark
      run: cob

Travis CI

dist: bionic
language: go
go:
  - 1.13.x

before_script:
  - curl -sfL https://raw.githubusercontent.com/knqyf263/cob/master/install.sh | sudo sh -s -- -b /usr/local/bin

script:
  - cob

CircleCI

version: 2
jobs:
  bench:
    docker:
      - image: circleci/golang:1.13
    steps:
      - checkout
      - run:
          name: Install cob
          command: curl -sfL https://raw.githubusercontent.com/knqyf263/cob/master/install.sh | sudo sh -s -- -b /usr/local/bin
      - run:
          name: Run cob
          command: cob
workflows:
  version: 2
  build-workflow:
    jobs:
      - bench

Example

Override a command to measure benchmarks

To measure benchmarks by make bench, you can use -bench-cmd and -bench-args options.

$ cob -bench-cmd make -bench-args bench

Run only those benchmarks matching a regular expression

$ cob -bench-args "test -bench Append -benchmem ./..."
Result
2020/01/12 17:32:30 Run Benchmark: 4363944cbed3da7a8245cbcdc8d8240b8976eb24 HEAD{@1}
2020/01/12 17:32:32 Run Benchmark: 599a5523729d4d99a331b9d3f71dde9e1e6daef0 HEAD

Result
======

+-----------------------------+----------+---------------+-------------------+
|            Name             |  Commit  |    NsPerOp    | AllocedBytesPerOp |
+-----------------------------+----------+---------------+-------------------+
| BenchmarkAppend_Allocate-16 |   HEAD   |  179.00 ns/op |      117 B/op     |
+                             +----------+---------------+-------------------+
|                             | [email protected]{1} |  115.00 ns/op |      23 B/op      |
+-----------------------------+----------+---------------+-------------------+

Comparison
==========

+-----------------------------+---------+-------------------+
|            Name             | NsPerOp | AllocedBytesPerOp |
+-----------------------------+---------+-------------------+
| BenchmarkAppend_Allocate-16 | 55.65%  |      408.70%      |
+-----------------------------+---------+-------------------+

Show only benchmarks with worse score

$ cob -only-degression
Result
2020/01/12 17:48:35 Run Benchmark: 4363944cbed3da7a8245cbcdc8d8240b8976eb24 HEAD{@1}
2020/01/12 17:48:38 Run Benchmark: 599a5523729d4d99a331b9d3f71dde9e1e6daef0 HEAD

Comparison
==========

+-----------------------------+---------+-------------------+
|            Name             | NsPerOp | AllocedBytesPerOp |
+-----------------------------+---------+-------------------+
| BenchmarkAppend_Allocate-16 | 52.34%  |      347.83%      |
+-----------------------------+---------+-------------------+

2020/01/12 17:48:39 This commit makes benchmarks worse

Specify a threshold

The following option means the program fails if a benchmark score gets worse than 50%.

$ cob -threshold 0.5 ./...

Specify a base commit compared with HEAD

By default, cob uses HEAD~1. If you compare benchmarks with different commit, you can use --base option.

$ cob --base origin/master ./...

Compare only memory allocation

You can use -compare option.

$ cob -compare B/op
Result
2020/01/15 14:46:31 Run Benchmark: 4363944cbed3da7a8245cbcdc8d8240b8976eb24 HEAD~1
2020/01/15 14:46:33 Run Benchmark: 599a5523729d4d99a331b9d3f71dde9e1e6daef0 HEAD

Result
======

+-----------------------------+----------+---------------+-------------------+
|            Name             |  Commit  |    NsPerOp    | AllocedBytesPerOp |
+-----------------------------+----------+---------------+-------------------+
| BenchmarkAppend_Allocate-16 |   HEAD   |  179.00 ns/op |      121 B/op     |
+                             +----------+---------------+-------------------+
|                             | [email protected]{1} |  104.00 ns/op |      23 B/op      |
+-----------------------------+----------+---------------+-------------------+
|      BenchmarkCall-16       |   HEAD   |   0.50 ns/op  |       0 B/op      |
+                             +----------+---------------+                   +
|                             | [email protected]{1} |   0.49 ns/op  |                   |
+-----------------------------+----------+---------------+-------------------+

Comparison
==========

+-----------------------------+---------+-------------------+
|            Name             | NsPerOp | AllocedBytesPerOp |
+-----------------------------+---------+-------------------+
| BenchmarkAppend_Allocate-16 |    -    |      426.09%      |
+-----------------------------+---------+-------------------+
|      BenchmarkCall-16       |    -    |       0.00%       |
+-----------------------------+---------+-------------------+

2020/01/15 14:46:35 This commit makes benchmarks worse

Skip running cob

If your commit message contains [skip cob], cob is skipped.

$ git add README.md
$ git commit -m "[skip cob] update README.md"
$ cob
2020/04/19 12:46:57 [skip cob] is detected, so the benchmark is skipped

Usage

NAME:
   cob - Continuous Benchmark for Go project

USAGE:
   cob [global options] command [command options] [arguments...]

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --only-degression   Show only benchmarks with worse score (default: false)
   --threshold value   The program fails if the benchmark gets worse than the threshold (default: 0.2)
   --base value        Specify a base commit compared with HEAD (default: "HEAD~1")
   --compare value     Which score to compare (default: "ns/op,B/op")
   --bench-cmd value   Specify a command to measure benchmarks (default: "go")
   --bench-args value  Specify arguments passed to -cmd (default: "test -run '^$' -bench . -benchmem ./...")
   --help, -h          show help (default: false)

Q&A

Benchmarks with the same name

Specify a package name.

$ cob -bench-args "test -bench . -benchmem ./foo" 
$ cob -bench-args "test -bench . -benchmem ./bar" 

A result of benchmarks is unstable

You can specify -benchtime.

$ cob -bench-args "test -bench . -benchmem -benchtime 10s ./..." 

License

This repository is available under the MIT

Author

Teppei Fukuda (knqyf263)

Issues
  • Allow to pass the base commit

    Allow to pass the base commit

    By default, cob uses @{1} as base commit but in my pipeline I'd like to compare against origin/master, one reason is that right now you can make a change that's worse and then make a commit without fixing it and it'll use the previous commit (the bad one) to compare against the new commit (that's equally bad) ... skipping the check

    opened by dcu 2
  • Fails when the package is not present in the base branch

    Fails when the package is not present in the base branch

    I passed --base=origin/master and now I'm getting:

    2020/01/15 15:44:23 failed to run a benchmark: failed to run 'go test -run '^$' -bench . -benchmem ./mypkg' command: exit status 1
    

    Because there are no benchmarks yet in the base. I believe this is a regression since I don't remember seeing this error before.

    I fixed it by adding a git grep command in the pipeline script but it'd be great if this tool did that

    opened by dcu 1
  • cob: Add -tags support for ad-hoc build tags.

    cob: Add -tags support for ad-hoc build tags.

    Description of changes: This PR adds support for passing in ad-hoc build time tags with -tags flag.

    By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

    Signed-off-by: Simarpreet Singh [email protected]

    opened by simar7 1
  • Feature request: exit if worktree is not clean

    Feature request: exit if worktree is not clean

    In order to not destroy any pending changes accidentally, cob should exit with status 1 if there are changes found in the git tree.

    I think you could use WorkTree.Status().IsClean() for this.

    opened by hazcod 1
  • Fix/project

    Fix/project

    Issue #, if available:

    Description of changes:

    By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

    opened by szuecs 0
  • feat: ignore previous benchmark if exit in error (#16)

    feat: ignore previous benchmark if exit in error (#16)

    Issue #, if available:

    Description of changes:

    By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

    opened by mauriciocm9 0
  • Mac Silicon support and binary

    Mac Silicon support and binary

    Hello, thanks for your work in this tool. I wanted to check whether you would be looking forward to adding support for Mac silicon and also exporting binaries for it.

    opened by pedro-stanaka 0
  • Inconsistent behavior of isClean check

    Inconsistent behavior of isClean check

    For some reason !isClean() introduced in the #3 doesn't always work as expected.

    Here is the output from git status and cob:

    $ git status
    On branch master
    Your branch is up to date with 'origin/master'.
    
    nothing to commit, working tree clean
    
    $ cob
    2020/07/02 15:30:59 the repository is dirty: commit all changes before running 'cob'
    

    This fails both locally and in the CI for some repos, but doesn't fail for others.

    Could this check be optional, so it could be force disabled in the CI where one doesn't care about loosing state.

    opened by lwolf 1
  • feat: ignore previous benchmark if exit in error (#16)

    feat: ignore previous benchmark if exit in error (#16)

    *Issue #16

    Description of changes:

    If benchmark in base commit fails, cob exit with error. This PR ignore errors in previous benchmark instead of exit with error.

    What did change?: Create a method to run previous benchmark, if benchmark fail, check for exit status 1 message.

    For benchmark that use a logger, parse.ParseSet could fail due to malformed benchmark line that's why a new function for the parse is created that behave similar to the one in the library.

    By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

    opened by mauriciocm9 0
  • Feature request: store benchmark result & compare with file

    Feature request: store benchmark result & compare with file

    Hi, love the project! Is it an idea to keep the previous bench result in a file, e.g. .github/cob.yaml and compare that? This way you don't always need to re-run the benchmark twice, and it allows to compare against other branches than master.

    opened by hazcod 2
Releases(v0.0.6)
Owner
Teppei Fukuda
Teppei Fukuda
A bytecode-based virtual machine to implement scripting/filtering support in your golang project.

eval-filter Implementation Scripting Facilities Types Built-In Functions Conditionals Loops Functions Case/Switch Use Cases Security Denial of service

Steve Kemp 76 Apr 21, 2022
Project test case recruitment kanggo

Kanggo Project test case recruitment kanggo Installation Need golang to run go run main.go .env file will be served Admin Account email:[email protected]

Tri Agung Prasetio 0 Nov 20, 2021
An open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developersAn open-source, distributed, cloud-native CD (Continuous Delivery) product designed for developers

Developer-oriented Continuous Delivery Product ⁣ English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use? How to

null 0 Oct 19, 2021
Continuous Benchmark for cache libraries written in golang.

Simple performance comparison of cache libraries written in golang. Reports Continuous Bencmark Result (click here) Default parameters 256 shards * 32

bench+ 8 Mar 21, 2022
Redis-benchmark - Simple get, mget and pipelined get benchmark.

redis-benchmark Simple get, mget and pipelined get benchmark. Usage git clone https://github.com/Ali-A-A/redis-benchmark.git cd ./redis-benchmark dock

Ali Abbasi Alaei 0 Dec 31, 2021
Bxd redis benchmark - Redis benchmark tool for golang

使用 redis benchmark 工具, 测试 10 20 50 100 200 1k 5k 字节 value 大小,redis get set 性能。 r

bingxindan 0 Jan 22, 2022
Go-driver-benchmark - Driver benchmark with golang

We use ClickkHouse for time-series databases, and the driver's performance is ve

null 3 Jan 27, 2022
Benchmark - Benchmark of logr implementations

Benchmark of logr implementations Implementations a function (can bridge to non-

null 2 Mar 23, 2022
Enterprise-Grade Continuous Delivery & DevOps Automation Open Source Platform

CDS: Continuous Delivery Service CDS is an Enterprise-Grade Continuous Delivery & DevOps Automation Platform written in Go(lang). This project is unde

OVHcloud 3.8k May 9, 2022
Drone is a Container-Native, Continuous Delivery Platform

Drone is a Continuous Delivery system built on container technology. Drone uses a simple YAML configuration file, a superset of docker-compose, to def

drone.io 25k May 14, 2022
goveralls - Go integration for Coveralls.io continuous code coverage tracking system.

goveralls Go integration for Coveralls.io continuous code coverage tracking system. Installation goveralls requires a working Go installation (Go-1.2

mattn 722 May 16, 2022
Probabilistic data structures for processing continuous, unbounded streams.

Boom Filters Boom Filters are probabilistic data structures for processing continuous, unbounded streams. This includes Stable Bloom Filters, Scalable

Tyler Treat 1.4k May 19, 2022
🔥 Continuous profiling platform — debug performance issues in your code!

Pyroscope is an open source continuous profiling platform.

Pyroscope 5.8k May 13, 2022
Concourse is a container-based continuous thing-doer written in Go and Elm.

Concourse: the continuous thing-doer. Concourse is an automation system written in Go. It is most commonly used for CI/CD, and is built to scale to an

Concourse 6.3k May 17, 2022
Open Source Continuous File Synchronization

Goals Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers. We strive to fulfill the goals belo

The Syncthing Project 44.6k May 10, 2022
Continuous performance analysis reports for software projects 🤖

Performabot Welcome to Performabot! This little helper can be used to provide Continuous Performance Reports within your GitHub project. But please be

Sascha Grunert 40 Feb 24, 2021
Continuous Delivery for Declarative Kubernetes, Serverless and Infrastructure Applications

Continuous Delivery for Declarative Kubernetes, Serverless and Infrastructure Applications Explore PipeCD docs » Overview PipeCD provides a unified co

PipeCD 582 May 12, 2022
A Distributed Continuous Integration System from MongoDB

Evergreen Evergreen is a distributed continuous integration system built by MongoDB. It dynamically allocates hosts to run tasks in parallel across ma

Evergreen 352 Apr 29, 2022
cTRL is a server for remote execution of pending tasks and commands in real time, supporting a queue with continuous thread limiting and throttling.

Документация на русском: https://github.com/eltaline/ctrl/blob/master/README-RUS.md cTRL is a server written in Go language that uses a modified versi

Eltaline 24 Mar 3, 2022
Open Source Continuous File Synchronization

Goals Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers. We strive to fulfill the goals belo

The Syncthing Project 44.7k May 16, 2022
Syncthing is a continuous file synchronization program.

Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers. We strive to fulfill the goals below. The goals are listed in order of importance, the most important one being the first.

The Syncthing Project 44.7k May 18, 2022
Bubbly is an open-source platform that gives you confidence in your continuous release process.

Bubbly Bubbly - Release Readiness in a Bubble Bubbly emerged from a need that many lean software teams practicing Continuous Integration and Delivery

Valocode 32 Apr 22, 2022
perfessor - Continuous Profiling Sidecar

perfessor - Continuous Profiling Sidecar About Perfessor is a continuous profiling agent that can profile running programs using perf It then converts

null 52 Feb 19, 2022
Continuous profiling for long-term postmortem analysis

profefe, a continuous profiling system, collects profiling data from a fleet of running applications and provides API for querying profiling samples for postmortem performance analysis.

profefe - continuous profiling 526 Apr 21, 2022
Zadig is a cloud native, distributed, developer-oriented continuous delivery product.

Zadig Developer-oriented Continuous Delivery Product English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use? Ho

KodeRover 1.3k May 18, 2022
Zadig is a cloud native, distributed, developer-oriented continuous delivery product.

Zadig Developer-oriented Continuous Delivery Product ⁣ English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use?

KodeRover 30 May 12, 2021
conprof - Continuous Profiling

conprof - Continuous Profiling Conprof is a continuous profiling project. Continuous profiling is the act of taking profiles of programs in a systemat

Parca 4 Feb 10, 2022
Continuous profiling for analysis of CPU, memory usage over time, and down to the line number. Saving infrastructure cost, improving performance, and increasing reliability.

Continuous profiling for analysis of CPU, memory usage over time, and down to the line number. Saving infrastructure cost, improving performance, and increasing reliability.

Parca 2.1k May 11, 2022
Continuous Delivery with Spinnaker and Kubernetes

Continuous Delivery with Spinnaker and Kubernetes Test Result Tutorial Build App This tutorial takes you through the process of creating a reliable an

Laith 1 Dec 8, 2021