A tool to aggregate and mine data from JSON reports of Go tests.

Related tags

JSON teststat
Overview

teststat

Build Status Coverage Status GoDevDoc Time Tracker Code lines Comments

A tool to aggregate and mine data from JSON reports of Go tests.

Why?

Mature Go projects often have a lot of tests, and not all of them are implemented in the best way. Some tests exhibit slightly different behavior and fail randomly.

Such unstable tests are annoying and kill productivity, identifying and improving them can have a huge impact on the quality of the project and developer experience.

In general, tests can flake in these ways.

Dependency on the environment

Tests would pass on one platform and fail on another.

Dependency on the initial state

Tests would pass on first invocation and fail on consecutive invocations.

Such tests can be identified by running suite multiple times.

go test -count 5 .

In some cases, this failure behavior is expected and desirable, though it reduces the usefulness of go test tool by noising -count mode. You would need to run the test suite multiple times with -count 1 and compare results to identify cases that flake on first invocation.

Dependency on undetermined or random factors

Tests would pass or fail randomly.

Such behavior almost always indicates a bug either in test or the code, and it needs a fix. Instability can be caused by data races which lead to subtle bugs and data corruption.

Running tests with race detector may help to expose some issues.

go test -race -count 5 ./...

Same as for data races, there is no guaranteed way to expose flaky tests that depend on random things. Running tests multiple times increases chances to hit an abnormal condition, but one can never be sure all issues have been found.

One data race can affect many test cases and "spam" the test output, this can be solved by grouping data races by tails of their stack traces, teststat accepts -race-depth int flag to perform such grouping. The bigger race depth value, more race groups are reported.

Go test tool offers machine-readable output with -json flag, teststat tool can read such output and determine racy, flaky or slow tests.

go test -race -json -count 5 ./... | teststat -race-depth 4 -

Another way of using it can be by running test suite multiple times and analyze reports together. This can help to expose tests that flake on first invocation.

go test -json ./... > test1.jsonl
go test -json ./... > test2.jsonl
go test -json ./... > test3.jsonl
teststat test1.jsonl test2.jsonl test3.jsonl

Resulting report can be formatted with -markdown to make it more readable as issue/pr comment.

Installation

go install github.com/vearutop/[email protected]

Or download the binary from releases.

Usage

Usage: teststat [options] report.jsonl ...
        Use `-` as file name to read from STDIN.
  -buckets int
        number of buckets for histogram (default 10)
  -markdown
        render output as markdown
  -race-depth int
        stacktrace depth to group similar data races (default 5)
  -slow duration
        minimal duration of slow test (default 1s)
  -slowest int
        limit number of slowest tests to list (default 30)
  -version
        show version and exit

Examples

Read from multiple files

Once you've collected JSONL test report, you can analyze it with this tool.

teststat -race-depth 4 -buckets 15 -slowest 7 ./flaky.jsonl ./test.jsonl 
Sample report.
Flaky tests:
github.com/acme/foo/core/affiliate/networks.TestBarSuite/TestOisGetReinvented: 2 passed, 8 failed
github.com/acme/foo/core/affiliate/networks.TestBarSuite/TestOisGetReinstallCallbacks: 2 passed, 8 failed
github.com/acme/foo/core/affiliate/networks.TestBarSuite: 2 passed, 8 failed
github.com/acme/foo/core/kafka.TestClose_Graceful_Pooled: 15 passed, 1 failed
github.com/acme/foo/core/kafka.TestClose_ClosePause: 14 passed, 2 failed

Slowest tests:
pass github.com/acme/foo/manipulation_services/api_server TestCreateLeafTracer_Ok 1m26.4s
pass github.com/acme/foo/manipulation_services/api_server TestCreateTracer_Ok 1m16.55s
pass github.com/acme/foo/manipulation_services/api_server TestCreateTracer_Ok/D4 1m16.45s
pass github.com/acme/foo/manipulation_services/api_server TestCreateLeafTracer_Ok 1m3.28s
pass github.com/acme/foo/manipulation_services/refresh_worker TestConsumeImpression_Success 52.85s
pass github.com/acme/foo/manipulation_services/api_server TestCreateLeafTracer_Ok 31.58s
pass github.com/acme/foo/manipulation_services/refresh_worker TestSubscriptionConsumer_DifferentEventSubtypes 30.39s

Events: map[cont:2368 fail:196 flaky:32 output:1805716 pass:660182 pause:2336 run:780596 skip:120154 slow:863]
Elapsed: 1h36m1.129999952s 
Slow: 40m34.649999952s

Elapsed distribution (seconds):
[  min   max]   cnt total% (37862 events)
[ 0.01  0.10] 32284 85.27% .....................................................................................
[ 0.11  0.24]  3383  8.94% ........
[ 0.25  0.52]   814  2.15% ..
[ 0.53  1.05]   574  1.52% .
[ 1.06  2.03]   552  1.46% .
[ 2.04  3.21]   122  0.32%
[ 3.30  4.90]    37  0.10%
[ 4.99  6.22]    36  0.10%
[ 6.40  8.68]    27  0.07%
[ 8.69 11.41]    22  0.06%
[12.48 14.30]     3  0.01%
[17.92 17.92]     1  0.00%
[30.39 31.58]     2  0.01%
[52.85 63.28]     2  0.01%
[76.45 86.40]     3  0.01%

Read from STDIN

go test -count 5 -json -race ./... | teststat -
Owner
Viacheslav Poturaev
Born in the snows of Siberia, wandering around the world, seeking the gold. 🦥 by day, 9½x developer by night. Building on an oobleck ground.
Viacheslav Poturaev
COBS implementation in Go (Decoder) and C (Encoder & Decoder) with tests.

COBS Table of Contents About The project COBS Specification Getting Started 3.1. Prerequisites 3.2. Installation 3.3. Roadmap Contributing License Con

Thomas Höhenleitner 2 May 22, 2022
Package json implements encoding and decoding of JSON as defined in RFC 7159

Package json implements encoding and decoding of JSON as defined in RFC 7159. The mapping between JSON and Go values is described in the documentation for the Marshal and Unmarshal functions

High Performance, Kubernetes Native Object Storage 2 Jun 26, 2022
Json-go - CLI to convert JSON to go and vice versa

Json To Go Struct CLI Install Go version 1.17 go install github.com/samit22/js

Samit Ghimire 5 Mar 3, 2022
JSON Spanner - A Go package that provides a fast and simple way to filter or transform a json document

JSON SPANNER JSON Spanner is a Go package that provides a fast and simple way to

null 3 Jun 30, 2022
Get JSON values quickly - JSON parser for Go

get json values quickly GJSON is a Go package that provides a fast and simple way to get values from a json document. It has features such as one line

Josh Baker 10.5k Jun 30, 2022
JSON diff library for Go based on RFC6902 (JSON Patch)

jsondiff jsondiff is a Go package for computing the diff between two JSON documents as a series of RFC6902 (JSON Patch) operations, which is particula

William Poussier 173 Jul 3, 2022
Fast JSON encoder/decoder compatible with encoding/json for Go

Fast JSON encoder/decoder compatible with encoding/json for Go

Masaaki Goshima 1.6k Jun 26, 2022
Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang

kazaam Description Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang. This functionality provides

Qntfy 205 Sep 17, 2021
A simple Go package to Query over JSON/YAML/XML/CSV Data

A simple Go package to Query over JSON Data. It provides simple, elegant and fast ODM like API to access, query JSON document Installation Install the

Saddam H 1.9k Jun 29, 2022
library for working amorphous data (as when you decode json into an interface{})

Introduction Decoding json into an interface{} produces an hierarchical arrangement of four data types: float64, string are 'primative types' and form

Chuck Luciano 9 Jul 5, 2021
Jsonmask use for mask sensitive data from json format

Jsonmask use for mask sensitive data from json format Installation go get -u github.com/rkritchat/jsonmask Code example package main import ( "fmt"

rkritchat 2 Mar 26, 2022
Senml-go - a Golang module for the JSON-based SenML sensor data format

ThingWave SenML module for Golang This is a Golang module for the JSON-based Sen

ThingWave 0 Jan 2, 2022
/ˈdʏf/ - diff tool for YAML files, and sometimes JSON

dyff is inspired by the way the old BOSH v1 deployment output reported changes from one version to another by only showing the parts of a YAML file that change.

null 638 Jun 23, 2022
CLI tool to handle JSON logs

logit is a CLI tool that consumes logs in JSON and feeds them into a logs storage like file, Amazon CloudWatch, or Slack.

Life4 22 Nov 9, 2021
Online tool that convert JSON to Go.

json-to-go 将json生成go的数据结构。Online tool that convert JSON to Go. Online Link | 地址 Link1 | 国内 Link2 | 国外 Local Develope | 本地开发 suggest use(强烈建议使用) pnpm !

misaka full stack 15 Dec 18, 2021
Fast JSON parser and validator for Go. No custom structs, no code generation, no reflection

fastjson - fast JSON parser and validator for Go Features Fast. As usual, up to 15x faster than the standard encoding/json. See benchmarks. Parses arb

Aliaksandr Valialkin 1.6k Jul 3, 2022
Fast and flexible JSON encoder for Go

Jettison Jettison is a fast and flexible JSON encoder for the Go programming language, inspired by bet365/jingo, with a richer features set, aiming at

William Poussier 128 Jun 1, 2022
HuJSON: JSON for Humans (comments and trailing commas)

HuJSON - Human JSON The HuJSON decoder is a JSON decoder that also allows comments, both /* ... */ and // to end of line trailing commas on arrays and

Tailscale 394 Jun 28, 2022
tson is JSON viewer and editor written in Go

tson tson is JSON viewer and editor written in Go. This tool displays JSON as a tree and you can search and edit key or values. Support OS Mac Linux I

skanehira 127 Mar 17, 2022