BDD Testing Framework for Go

The first commit to Ginkgo was on August 19th, 2013. Much fruitful community feedback - and code cruft - have accumulated over the years. Both can be addressed with a new major version of Ginkgo: Ginkgo 2.0.

The backlog capturing the work is here: https://www.pivotaltracker.com/n/projects/2493102

The original proposal for Ginkgo 2.0 has been pulled together as a google doc here. Note that the google doc is now stale - the Tracker backlog reflects the latest work most accurately.

V2 is in active development on the ver2 branch and an uptodate changelog and migration doc is maintained on the branch here

If you use Ginkgo and are interested in what's in 2.0 please comment on this issue!

In Kubernetes, we would like to have:

• an overall timeout for the entire suite (-ginkgo.timeout, with a very high value because the suite is large)
• a per spec timeout (???, smaller, to avoid blocking for the entire -ginkgo.timeout duration on a single spec)
• timeouts for AfterEach specs (we try to clean up, but cleaning up itself may get stuck)

So far I have only found -ginkgo.timeout. One downside of it is that it also aborts cleanup operations as soon as those block, which is not useful for Kubernetes because the cleanup operation may involved communication with the apiserver to remove objects.

I tried with this:

var _ = ginkgo.Describe("test", func() {
	ginkgo.BeforeEach(func() {
		fmt.Fprint(ginkgo.GinkgoWriter, "hello\n")
	})

	ginkgo.AfterEach(func() {
		defer fmt.Fprint(ginkgo.GinkgoWriter, "done\n")
		fmt.Fprint(ginkgo.GinkgoWriter, "world\n")
		time.Sleep(time.Hour)
	})

	ginkgo.It("times out", func() {
		time.Sleep(time.Hour)
	})
})

When I run with -ginkgo.timeout=10s -ginkgo.v -ginkgo.progress, I get:

test
  times out
  /nvme/gopath/src/github.com/intel/pmem-csi/test/e2e/e2e.go:177
[BeforeEach] test
  /nvme/gopath/src/github.com/intel/pmem-csi/test/e2e/e2e.go:167
hello
[It] times out
  /nvme/gopath/src/github.com/intel/pmem-csi/test/e2e/e2e.go:177
[AfterEach] test
  /nvme/gopath/src/github.com/intel/pmem-csi/test/e2e/e2e.go:171
world
------------------------------
•! [INTERRUPTED] [10.933 seconds]
test
/nvme/gopath/src/github.com/intel/pmem-csi/test/e2e/e2e.go:166
  [It] times out
  /nvme/gopath/src/github.com/intel/pmem-csi/test/e2e/e2e.go:177

  Begin Captured GinkgoWriter Output >>
    [BeforeEach] test
      /nvme/gopath/src/github.com/intel/pmem-csi/test/e2e/e2e.go:167
    hello
    [It] times out
      /nvme/gopath/src/github.com/intel/pmem-csi/test/e2e/e2e.go:177
    [AfterEach] test
      /nvme/gopath/src/github.com/intel/pmem-csi/test/e2e/e2e.go:171
    world
  << End Captured GinkgoWriter Output

  Interrupted by Timeout

  Here's a stack trace of all running goroutines:
  ...

Note that the cleanup spec didn't run it's defer.

We could provide per-spec timeouts via ctx := context.Timeout(context.Background(), 10*time.Minute). We then need to be careful that the cleanup spec doesn't use the same context because it wouldn't get any work done after a timeout. The downside of this is that we would have to touch a lot of code in Kubernetes, which is always a daunting prospect.

IMHO it would be simpler to have a default -ginkgo.it-timeout (for It specs), -ginkgo.after-timeout (for AfterEach) and perhaps a Timeout(time.Duration) decorator to override those defaults.

@onsi Ok here we go !

Can you help me write the unit test in a follow on patch? I hacked around a little with it, but it seemed a little tricky how the actual tests were working for spec.go, and ive got a few other priorities on hand.

This will be really useful in upstream kubernetes and openshift for us.

First of all, thanks so much for this library. It really makes testing in Go a joy.

RSpec includes before(:all) and after(:all) nodes. These nodes are executed once at the beginning or end of an example group, respectively:

require 'rspec'

RSpec.describe 'before and after all nodes' do
  before(:all) { puts "\nbefore(:all)" }
  after(:all) { puts "\nafter(:all)" }
  it 'is an example' { }
  it 'is another example' { }
end

When run, rspec outputs:

before(:all)
..
after(:all)

I was surprised that Ginkgo does not include BeforeAll or AfterAll functions. Is this a conscious omission? Would you consider a pull request which adds these functions?

My goal here is to allow some grace around flaky tests. Unfortunately, they are a fact of life for the Kubernetes project. Sometimes that's due to programmer error and sometimes due to downstream dependencies, etc--I'm not arguing that I want flaky tests, just that I need to my CI system to understand the difference between "fails occasionally" and "totally broken". So I want to implement a mode that does something like this:

You'd call something like ginkgo ... --test-tries=2 --test-required-passes=1 .... This tells ginkgo that each test is allowed two attempts and must pass at least once. Then ginkgo, after it runs a spec that fails, would run it again. If it passes the second time, then the entire suite is considered to have passed, but the individual failure is still reported in the JUnit output.

Does this seem like a patch that would be accepted?

see https://sunrise.ci.cf-app.com/pipelines/pivnet-resource/jobs/test/builds/135

[2] /tmp/ginkgo371214560/validator.test flag redefined: ginkgo.seed
[2] panic: /tmp/ginkgo371214560/validator.test flag redefined: ginkgo.seed

I see the same thing when locally when not running in parallel.

This is running golang 1.6 and I believe the behavior is not observed on golang 1.5 as I have used the Table extension in other projects on golang 1.5 without issue.

When debugging a test failure in Kubernetes, the steps to investigate are usually:

• look at failure message
• look at stderr for the test (= output captured via GinkgoWriter)
• look at the full job output (last resort, because it can be large)

For the second step, we log test failures as they occur in the GinkgoWriter. This works for gomega because it calls our custom fail handler, which then calls ginkgo.Fail. It does not work for timeouts or other failures detected by ginkgo itself, like an error that is returned to DeferCleanup.

I found that I cannot parse GinkgoT() as an argument to my test helper code that use both testing and benchmark code because it doesn't have Helper() method.

For example:

package gink

import (
	"testing"

	"github.com/onsi/ginkgo"
	"github.com/onsi/gomega"
)

func Test(t *testing.T) {
	gomega.RegisterFailHandler(ginkgo.Fail)
	ginkgo.RunSpecs(t, "suite")
}

var _ = ginkgo.Describe("my suite", func() {
	ginkgo.It("under test", func() {
		myTestHelperCode(ginkgo.GinkgoT())
	})
})

// I use this function across my test code and benchmark.
func myTestHelperCode(tb testing.TB) {
	tb.Helper()
}

The result from go test:

$ go test .
# github.com/wingyplus/gink [github.com/wingyplus/gink.test]
./gink_test.go:17:34: cannot use ginkgo.GinkgoT() (type ginkgo.GinkgoTInterface) as type testing.TB in argument to myTestHelperCode:
	ginkgo.GinkgoTInterface does not implement testing.TB (missing Helper method)
FAIL	github.com/wingyplus/gink [build failed]
FAIL

We have a test suite running ginkgo tests in parallel on Jenkins. I'm currently trying to track down a hanging test in https://github.com/kubernetes/kubernetes/issues/13485, but it's been very hard to figure out what's wrong, since the log output doesn't seem to be including the hanging test.

The log for one of our failing runs shows nothing for over 30m:

01:30:04 STEP: Destroying namespace "e2e-tests-pod-disks-z910m" for this suite.
01:30:04 
01:30:04 
01:30:04 • [SLOW TEST:376.248 seconds]
01:30:04 Pod Disks
01:30:04 /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:267
01:30:04   should schedule a pod w/two RW PDs both mounted to one container, write to PD, verify contents, delete pod, recreate pod, verify contents, and repeat in rapid succession
01:30:04   /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/pd.go:266
01:30:04 ------------------------------
02:03:34 Build timed out (after 60 minutes). Marking the build as aborted.
02:03:34 Build timed out (after 60 minutes). Marking the build as failed.
02:03:34 Build was aborted
02:03:34 Recording test results
02:03:34 
02:03:34 Ran 119 of 212 Specs in 2486.623 seconds
02:03:34 FAIL! -- 116 Passed | 0 Failed | 2 Pending | 91 Skipped 
02:03:34 
02:03:34 Ginkgo ran 1 suite in 41m27.528523653s
02:03:34 Test Suite Failed

I'm guessing this is intentional, since Ginkgo only prints out the log when a test completes. However, it doesn't seem to be handling the interrupt here properly - I'd expect it to dump any in-progress tests so that you could see what is stuck.

(I know about the Ginkgo parallel streaming mode, and I've been trying to use it, but this particular test failure seems to be very difficult to reproduce on demand.)

Hi guys, Im currently trying to upgrade to v2 on our cloudfoundry app-autoscaler project and have hit a significant blocker.

The issue will probably in more complex projects with loads of dependencies that use ginkgo from multiple versions i.e. a transient dependency (because go does not have a test dependency tree)

I believe there are at least 4 dependencies transitily including the v1 ginkgo

This means in our project we have 1.16.5 and 2.0.0 included.

       github.com/onsi/ginkgo v1.16.5 // indirect
       github.com/onsi/ginkgo/v2 v2.0.0

This does not work because of the init(){} functions in both included dependencies are run and they use the flag.CommandLine at init time and modify it creating duplicate flags that causes panics with

/app-autoscaler-release/src/autoscaler/api/brokerserver/brokerserver.test flag redefined: ginkgo.seed
panic: /Users/kevincross/SAPDevelop/cf/app-autoscaler-release/src/autoscaler/api/brokerserver/brokerserver.test flag redefined: ginkgo.seed

goroutine 1 [running]:
flag.(*FlagSet).Var(0xc000218120, 0x1df5fc8, 0x2351f80, 0xc000357640, 0xb, 0x1cd87a3, 0x2a)
       /golang/1.16.8/go/src/flag/flag.go:871 +0x637
flag.(*FlagSet).Int64Var(...)
       /golang/1.16.8/go/src/flag/flag.go:682
github.com/onsi/ginkgo/v2/types.bindFlagSet(0xc0003cc000, 0x20, 0x21, 0x1bbd540, 0xc0003b86f0, 0x232f420, 0xd, 0xd, 0x0, 0x0, ...)
        /go/pkg/mod/github.com/onsi/ginkgo/[email protected]/types/flags.go:161 +0x15e5
github.com/onsi/ginkgo/v2/types.NewAttachedGinkgoFlagSet(...)
        /go/pkg/mod/github.com/onsi/ginkgo/[email protected]/types/flags.go:113
github.com/onsi/ginkgo/v2/types.BuildTestSuiteFlagSet(0x2351f80, 0x23519a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/pkg/mod/github.com/onsi/ginkgo/[email protected]/types/config.go:346 +0x6e8
github.com/onsi/ginkgo/v2.init.0()
        /github.com/onsi/ginkgo/[email protected]/core_dsl.go:47 +0x8f
ginkgo run failed

After finding this I updated the v2 module to make a copy of the command line changing v2 to using a copy... this stops the panic /v2/ginkgo/types/config.go:346

return NewAttachedGinkgoFlagSet(flag.NewFlagSet(os.Args[0], flag.ExitOnError), flags, bindings, FlagSections, extraGoFlagsSection)

but the v1 reports: flag provided but not defined: -ginkgo.randomize-all somehow 😕 I tried looking a bit further but it all gets a bit wierd with the 2nd layer of compliation 🤷

Ive also tried using the

exclude (
	github.com/onsi/ginkgo v1.16.5
)

in the go mod file but too no avail.

to replicate this error try checkout the autoscaler project https://github.com/cloudfoundry/app-autoscaler-release checkout the branch ginkgo-v2 and run make test this should replicate this issue easily (after a bunch of docker ect.)

When trying to run the apt-buildpack integration tests on a computer running Go 1.13, we were unable to and got the following error:

	 -------------------------------------------------------------------
	|                                                                   |
	|  Ginkgo timed out waiting for all parallel nodes to report back!  |
	|                                                                   |
	 -------------------------------------------------------------------

 integration timed out. path: .
[1] flag provided but not defined: -test.timeout
[1] Usage of /tmp/ginkgo538951234/integration.test:
...

Our integration test script:

#!/usr/bin/env bash
set -euo pipefail

cd "$( dirname "${BASH_SOURCE[0]}" )/.."
source .envrc
./scripts/install_tools.sh

GINKGO_NODES=${GINKGO_NODES:-3}
GINKGO_ATTEMPTS=${GINKGO_ATTEMPTS:-2}
export CF_STACK=${CF_STACK:-cflinuxfs3}

UNCACHED_BUILDPACK_FILE=${UNCACHED_BUILDPACK_FILE:-""}

cd src/*/integration

echo "Run Uncached Buildpack"
BUILDPACK_FILE="$UNCACHED_BUILDPACK_FILE" \
  ginkgo -r -mod vendor --flakeAttempts=$GINKGO_ATTEMPTS -nodes $GINKGO_NODES --slowSpecThreshold=60 -- --cached=false

Env:

[email protected]:/app# go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/app/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build280745408=/tmp/go-build -gno-record-gcc-switches"

I use the --junit-report flag to autogenerate Junit reports, and to make the reports a little more user-friendly I'd like to customise the output using the JunitReportConfig struct (source).

If I understand the docs correctly, there's no way to customise the reporter on the command line. Instead I'd need to call GenerateJunitReportWithConfig(...) in the ReportAfterSuite() of each of my test suites. But this has some side-effects:

1. I have a lot of suites so there would be a fair amount of duplication in setting up the reporting
2. When my team mates add new suites they'll need to remember to add the reporting logic or their tests won't report anything
3. I'd have to implement my own XML mangling outside Ginkgo to merge all the reports into a single file

Is there a way to run a reporter for all suites, in the same way that the CLI wires up the Junit reporter when the --junit-report flag is used?

Bumps golang.org/x/sys from 0.3.0 to 0.4.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps golang.org/x/tools from 0.4.0 to 0.5.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

On #130 @nabbas-ca wrote:

I want to write k8s controller/operator tests within 2 suites, one with envtest controlller manager initialized and listening, and one without. So I can have finegrained negative path testing. For example, I want to to be able to setup a mock k8s api with bad CRs(before the webhook is alive or registered), and see how my controller reacts to correct it , as unit tests.

This way, I can setup a set of suites, within the same package like this:

1- clean controller, no CRs preloaded 2- preloaded good CRs, setup controller after the CRs are created 3- preloaded bad CRs, setup controller after the CRs are created .....

Is that possible? remember that envTest is expensive to start. making a BeforeEach at top level of tests that includes manager start, could be possiblle, but it means starting and stopping manager for every test.

Hello, I know there are already a couple of issues and some discussions related to that topic however I wanted to ask this question. Since v2 Ginkgo produces its own json format as an output of the test results.

We are using Sonarqube to report back test coverage and test reports and unfortunately, at least as far as I know, Sonar expects the test reports in the test2json format produced by go test:

Path to test execution reports generated by Go with '-json' key, available since go1.10 (e.g.: go test -json > test-report.out).

One the one hand we could only use go test to get the desired outputs for test reports & coverage however on the other hand the output Ginkgo produces is just much more detailed that we also would like to see (in our CI pipeline). Therefore, our mitigation currently is to execute both steps (go test & ginkgo) to have the best of both but as said, should only be a mitigation.

So, my questions would be:

1. Someone else have already faced & solved this issue? Could a ReportAfterSuite convert the report back into the test2json format?
2. Would it be a feasible that Ginkgo can bring back the support for a test2json output format?

Using v2.1.4.

We like to use BeforeAll/AfterAll feature added recently. When using BeforeAll/AfterAll, it's enforcing Ordered container. We cannot use Ordered container, because using DescribeTable with multiple entries, if one entry fails, ginkgo will not run the following entries.

There's actually two options here: Either have DescribeTable continue to run the next entry if one fails: that will be a backward compatibility breaking change. Or, have BeforeAll/AfterAll not enforcing Ordered container, which I assume will allow DescribeTable to not skip the post-failure entries.