Mutation testing for Go source code

Overview

go-mutesting GoDoc Build Status Coverage Status

go-mutesting is a framework for performing mutation testing on Go source code. Its main purpose is to find source code, which is not covered by any tests.

Quick example

The following command mutates the go-mutesting project with all available mutators.

go-mutesting github.com/zimmski/go-mutesting/...

The execution of this command prints for every mutation if it was successfully tested or not. If not, the source code patch is printed out, so the mutation can be investigated. The following shows an example for a patch of a mutation.

for _, d := range opts.Mutator.DisableMutators {
	pattern := strings.HasSuffix(d, "*")

-	if (pattern && strings.HasPrefix(name, d[:len(d)-2])) || (!pattern && name == d) {
+	if (pattern && strings.HasPrefix(name, d[:len(d)-2])) || false {
		continue MUTATOR
	}
}

The example shows that the right term (!pattern && name == d) of the || operator is made irrelevant by substituting it with false. Since this change of the source code is not detected by the test suite, meaning the test suite did not fail, we can mark it as untested code.

The next mutation shows code from the removeNode method of a linked list implementation.

	}

	l.first = nil
-	l.last = nil
+
	l.len = 0
}

We know that the code originates from a remove method which means that the mutation introduces a leak by ignoring the removal of a reference. This can be tested with go-leaks.

Table of content

What is mutation testing?

The definition of mutation testing is best quoted from Wikipedia:

Mutation testing (or Mutation analysis or Program mutation) is used to design new software tests and evaluate the quality of existing software tests. Mutation testing involves modifying a program in small ways. Each mutated version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. This is called killing the mutant. Test suites are measured by the percentage of mutants that they kill. New tests can be designed to kill additional mutants.
-- https://en.wikipedia.org/wiki/Mutation_testing

Tests can be created to verify the correctness of the implementation of a given software system, but the creation of tests still poses the question whether the tests are correct and sufficiently cover the requirements that have originated the implementation.
-- https://en.wikipedia.org/wiki/Mutation_testing

Although the definition states that the main purpose of mutation testing is finding implementation cases which are not covered by tests, other implementation flaws can be found too. Mutation testing can for example uncover dead and unneeded code.

Mutation testing is also especially interesting for comparing automatically generated test suites with manually written test suites. This was the original intention of go-mutesting which is used to evaluate the generic fuzzing and delta-debugging framework Tavor.

How do I use go-mutesting?

go-mutesting includes a binary which is go-getable.

go get -t -v github.com/zimmski/go-mutesting/...

The binary's help can be invoked by executing the binary without arguments or with the --help argument.

go-mutesting --help

Note: This README describes only a few of the available arguments. It is therefore advisable to examine the output of the --help argument.

The targets of the mutation testing can be defined as arguments to the binary. Every target can be either a Go source file, a directory or a package. Directories and packages can also include the ... wildcard pattern which will search recursively for Go source files. Test source files with the suffix _test are excluded, since this would interfere with the testing process most of the time.

The following example gathers all Go files which are defined by the targets and generate mutations with all available mutators of the binary.

go-mutesting parse.go example/ github.com/zimmski/go-mutesting/mutator/...

Every mutation has to be tested using an exec command. By default the built-in exec command is used, which tests a mutation using the following steps:

  • Replace the original file with the mutation.
  • Execute all tests of the package of the mutated file.
  • Report if the mutation was killed.

Alternatively the --exec argument can be used to invoke an external exec command. The /scripts/exec directory holds basic exec commands for Go projects. The test-mutated-package.sh script implements all steps and almost all features of the built-in exec command. It can be for example used to test the github.com/zimmski/go-mutesting/example package.

go-mutesting --exec "$GOPATH/src/github.com/zimmski/go-mutesting/scripts/exec/test-mutated-package.sh" github.com/zimmski/go-mutesting/example

The execution will print the following output.

Note: This output is from an older version of go-mutesting. Up to date versions of go-mutesting will have different mutations.

PASS "/tmp/go-mutesting-422402775//home/zimmski/go/src/github.com/zimmski/go-mutesting/example/example.go.0" with checksum b705f4c99e6d572de509609eb0a625be
PASS "/tmp/go-mutesting-422402775//home/zimmski/go/src/github.com/zimmski/go-mutesting/example/example.go.1" with checksum eb54efffc5edfc7eba2b276371b29836
PASS "/tmp/go-mutesting-422402775//home/zimmski/go/src/github.com/zimmski/go-mutesting/example/example.go.2" with checksum 011df9567e5fee9bf75cbe5d5dc1c81f
--- /home/zimmski/go/src/github.com/zimmski/go-mutesting/example/example.go
+++ /tmp/go-mutesting-422402775//home/zimmski/go/src/github.com/zimmski/go-mutesting/example/example.go.3
@@ -16,7 +16,7 @@
        }

        if n < 0 {
-               n = 0
+
        }

        n++
FAIL "/tmp/go-mutesting-422402775//home/zimmski/go/src/github.com/zimmski/go-mutesting/example/example.go.3" with checksum 82fc14acf7b561598bfce25bf3a162a2
PASS "/tmp/go-mutesting-422402775//home/zimmski/go/src/github.com/zimmski/go-mutesting/example/example.go.4" with checksum 5720f1bf404abea121feb5a50caf672c
PASS "/tmp/go-mutesting-422402775//home/zimmski/go/src/github.com/zimmski/go-mutesting/example/example.go.5" with checksum d6c1b5e25241453128f9f3bf1b9e7741
--- /home/zimmski/go/src/github.com/zimmski/go-mutesting/example/example.go
+++ /tmp/go-mutesting-422402775//home/zimmski/go/src/github.com/zimmski/go-mutesting/example/example.go.6
@@ -24,7 +24,6 @@
        n += bar()

        bar()
-       bar()

        return n
 }
FAIL "/tmp/go-mutesting-422402775//home/zimmski/go/src/github.com/zimmski/go-mutesting/example/example.go.6" with checksum 5b1ca0cfedd786d9df136a0e042df23a
PASS "/tmp/go-mutesting-422402775//home/zimmski/go/src/github.com/zimmski/go-mutesting/example/example.go.8" with checksum 6928f4458787c7042c8b4505888300a6
The mutation score is 0.750000 (6 passed, 2 failed, 0 skipped, total is 8)

The output shows that eight mutations have been found and tested. Six of them passed which means that the test suite failed for these mutations and the mutations were therefore killed. However, two mutations did not fail the test suite. Their source code patches are shown in the output which can be used to investigate these mutations.

The summary also shows the mutation score which is a metric on how many mutations are killed by the test suite and therefore states the quality of the test suite. The mutation score is calculated by dividing the number of passed mutations by the number of total mutations, for the example above this would be 6/8=0.75. A score of 1.0 means that all mutations have been killed.

Blacklist false positives

Mutation testing can generate many false positives since mutation algorithms do not fully understand the given source code. early exits are one common example. They can be implemented as optimizations and will almost always trigger a false-positive since the unoptimized code path will be used which will lead to the same result. go-mutesting is meant to be used as an addition to automatic test suites. It is therefore necessary to mark such mutations as false-positives. This is done with the --blacklist argument. The argument defines a file which contains in every line a MD5 checksum of a mutation. These checksums can then be used to ignore mutations.

Note: The blacklist feature is currently badly implemented as a change in the original source code will change all checksums.

The example output of the How do I use go-mutesting? section describes a mutation example.go.6 which has the checksum 5b1ca0cfedd786d9df136a0e042df23a. If we want to mark this mutation as a false-positive, we simple create a file with the following content.

5b1ca0cfedd786d9df136a0e042df23a

The blacklist file, which is named example.blacklist in this example, can then be used to invoke go-mutesting.

go-mutesting --blacklist example.blacklist github.com/zimmski/go-mutesting/example

The execution will print the following output.

Note: This output is from an older version of go-mutesting. Up to date versions of go-mutesting will have different mutations.

PASS "/tmp/go-mutesting-208240643/example.go.0" with checksum b705f4c99e6d572de509609eb0a625be
PASS "/tmp/go-mutesting-208240643/example.go.1" with checksum eb54efffc5edfc7eba2b276371b29836
PASS "/tmp/go-mutesting-208240643/example.go.2" with checksum 011df9567e5fee9bf75cbe5d5dc1c81f
--- example.go  2014-12-29 23:37:42.813320040 +0100
+++ /tmp/go-mutesting-208240643/example.go.3    2014-12-30 00:49:33.573285038 +0100
@@ -16,7 +16,7 @@
        }

        if n < 0 {
-               n = 0
+
        }

        n++
FAIL "/tmp/go-mutesting-208240643/example.go.3" with checksum 82fc14acf7b561598bfce25bf3a162a2
PASS "/tmp/go-mutesting-208240643/example.go.4" with checksum 5720f1bf404abea121feb5a50caf672c
PASS "/tmp/go-mutesting-208240643/example.go.5" with checksum d6c1b5e25241453128f9f3bf1b9e7741
PASS "/tmp/go-mutesting-208240643/example.go.8" with checksum 6928f4458787c7042c8b4505888300a6
The mutation score is 0.857143 (6 passed, 1 failed, 0 skipped, total is 7)

By comparing this output to the original output we can state that we now have 7 mutations instead of 8.

How do I write my own mutation exec commands?

A mutation exec command is invoked for every mutation which is necessary to test a mutation. Commands should handle at least the following phases.

  1. Setup the source to include the mutation.
  2. Test the source by invoking the test suite and possible other test functionality.
  3. Cleanup all changes and remove all temporary assets.
  4. Report if the mutation was killed.

It is important to note that each invocation should be isolated and therefore stateless. This means that an invocation must not interfere with other invocations.

A set of environment variables, which define exactly one mutation, is passed on to the command.

Name Description
MUTATE_CHANGED Defines the filename to the mutation of the original file.
MUTATE_DEBUG Defines if debugging output should be printed.
MUTATE_ORIGINAL Defines the filename to the original file which was mutated.
MUTATE_PACKAGE Defines the import path of the origianl file.
MUTATE_TIMEOUT Defines a timeout which should be taken into account by the exec command.
MUTATE_VERBOSE Defines if verbose output should be printed.
TEST_RECURSIVE Defines if tests should be run recursively.

A command must exit with an appropriate exit code.

Exit code Description
0 The mutation was killed. Which means that the test led to a failed test after the mutation was applied.
1 The mutation is alive. Which means that this could be a flaw in the test suite or even in the implementation.
2 The mutation was skipped, since there are other problems e.g. compilation errors.
>2 The mutation produced an unknown exit code which might be a flaw in the exec command.

Examples for exec commands can be found in the scripts directory.

Which mutators are implemented?

Branch mutators

Name Description
branch/case Empties case bodies.
branch/if Empties branches of if and else if statements.
branch/else Empties branches of else statements.

Expression mutators

Name Description
expression/comparison Searches for comparison operators, such as > and <=, and replaces them with similar operators to catch off-by-one errors, e.g. > is replaced by >=.
expression/remove Searches for && and || operators and makes each term of the operator irrelevant by using true or false as replacements.

Statement mutators

Name Description
statement/remove Removes assignment, increment, decrement and expression statements.

How do I write my own mutators?

Each mutator must implement the Mutator interface of the github.com/zimmski/go-mutesting/mutator package. The methods of the interface are described in detail in the source code documentation.

Additionally each mutator has to be registered with the Register function of the github.com/zimmski/go-mutesting/mutator package to make it usable by the binary.

Examples for mutators can be found in the github.com/zimmski/go-mutesting/mutator package and its sub-packages.

Other mutation testing projects and their flaws

go-mutesting is not the first project to implement mutation testing for Go source code. A quick search uncovers the following projects.

All of them have significant flaws in comparison to go-mutesting:

  • Only one type (or even one case) of mutation is implemented.
  • Can only be used for one mutator at a time (manbearpig, Golang-Mutation-testing).
  • Mutation is done by content which can lead to lots of invalid mutations (Golang-Mutation-testing).
  • New mutators are not easily implemented and integrated.
  • Can only be used for one package or file at a time.
  • Other scenarios as go test cannot be applied.
  • Do not properly clean up or handle fatal failures.
  • No automatic tests to ensure that the algorithms are working at all.
  • Uses another language (Golang-Mutation-testing).

Can I make feature requests and report bugs and problems?

Sure, just submit an issue via the project tracker and I will see what I can do. Please note that I do not guarantee to implement anything soon and bugs and problems are more important to me than new features. If you need something implemented or fixed right away you can contact me via mail [email protected] to do contract work for you.

Issues
  • IMPORTANT - Go-mutesting is deleting everything

    IMPORTANT - Go-mutesting is deleting everything

    Hi there,

    Thank you for your awesome project. I ran go-mutesting on my project and it deleted everything on the directory including the git metadata files (.git directory). Fortunately, I didn't have too much code not pushed to the repo but it can be very dangerous.

    How to reproduce:

    1. Install go-mutesting via go install
    2. Clone the project bench.
    3. Run go-mutesting . inside of the project root directory. Note: I'm able to reproduce in Go 1.17.8 and Go 1.18.1
    opened by thiagonache 6
  • Type checking failure

    Type checking failure

    Since the switch to go/types, go-mutesting throws an error on files using functions defined in other files (I've also had errors with imports).

    I've only just discovered the project, so I may have missed something, but here's what I get:

    With the last version: image As you can see the conf.Check throws an error (https://github.com/zimmski/go-mutesting/blob/master/cmd/go-mutesting/main.go#L274).

    Without the last merge: image

    bug 
    opened by loamhoof 6
  • [ReservedKeywords] More exhaustive list of blacklisted identifiers

    [ReservedKeywords] More exhaustive list of blacklisted identifiers

    This branch blacklists the full list of reserved keywords and builtin functions (Go 1.7). This reduces the risk of failed compilation after a mutation.

    opened by seborama 5
  • Panic while testing github.com/mstone/focus/ot

    Panic while testing github.com/mstone/focus/ot

    I tried go-mutesting on an operational transformation algorithm I'm implementing (package github.com/mstone/focus/ot) and got this panic while running:

    go-mutesting --exec "$GOPATH/src/github.com/zimmski/go-mutesting/scripts/simple.sh" github.com/mstone/focus/ot
    
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal 0xb code=0x1 addr=0x0 pc=0x448bd5]
    
    goroutine 16 [running]:
    runtime.panic(0x5dd0c0, 0x71caf3)
        /usr/lib/go/src/pkg/runtime/panic.c:279 +0xf5
    go/ast.(*Ident).Pos(0x0, 0x57f2c0)
        /usr/lib/go/src/pkg/go/ast/ast.go:423 +0x5
    go/ast.(*BinaryExpr).Pos(0xc2080682d0, 0xc2080a4798)
        /usr/lib/go/src/pkg/go/ast/ast.go:441 +0x3f
    go/printer.(*printer).binaryExpr(0xc20804e000, 0xc208068300, 0x0, 0x6, 0x1)
        /usr/lib/go/src/pkg/go/printer/nodes.go:647 +0x399
    go/printer.(*printer).expr1(0xc20804e000, 0x7f46a38d5ae0, 0xc208068300, 0x0, 0x1)
        /usr/lib/go/src/pkg/go/printer/nodes.go:686 +0xf81
    go/printer.(*printer).expr(0xc20804e000, 0x7f46a38d5ae0, 0xc208068300)
        /usr/lib/go/src/pkg/go/printer/nodes.go:893 +0x4d
    go/printer.(*printer).controlClause(0xc20804e000, 0xc2080a4d00, 0x0, 0x0, 0x7f46a38d5ae0, 0xc208068300, 0x0, 0x0)
        /usr/lib/go/src/pkg/go/printer/nodes.go:996 +0x12c
    go/printer.(*printer).stmt(0xc20804e000, 0x7f46a38d5b18, 0xc20806e3c0, 0x100)
        /usr/lib/go/src/pkg/go/printer/nodes.go:1151 +0x29f2
        /usr/lib/go/src/pkg/go/printer/nodes.go:942 +0xfe
    go/printer.(*printer).adjBlock(0xc20804e000, 0x33, 0xb, 0xc208068510)
        /usr/lib/go/src/pkg/go/printer/nodes.go:1514 +0x683
    go/printer.(*printer).funcDecl(0xc20804e000, 0xc208068540)
        /usr/lib/go/src/pkg/go/printer/nodes.go:1538 +0x2b5
    go/printer.(*printer).decl(0xc20804e000, 0x7f46a38d59c8, 0xc208068540)
        /usr/lib/go/src/pkg/go/printer/nodes.go:1548 +0xa4
    go/printer.(*printer).declList(0xc20804e000, 0xc20804a400, 0x13, 0x20)
        /usr/lib/go/src/pkg/go/printer/nodes.go:1589 +0x1b0
    go/printer.(*printer).file(0xc20804e000, 0xc20804c500)
        /usr/lib/go/src/pkg/go/printer/nodes.go:1597 +0x18b
    go/printer.(*printer).printNode(0xc20804e000, 0x5ce020, 0xc20804c500, 0x0, 0x0)
        /usr/lib/go/src/pkg/go/printer/printer.go:1087 +0x8b2
    go/printer.(*Config).fprint(0xc2080a57d8, 0x7f46a38d6170, 0xc208088060, 0xc20800e040, 0x5ce020, 0xc20804c500, 0xc208068150, 0x0, 0x0)
        /usr/lib/go/src/pkg/go/printer/printer.go:1226 +0xad
    go/printer.(*Config).Fprint(0xc2080a57d8, 0x7f46a38d6170, 0xc208088060, 0xc20800e040, 0x5ce020, 0xc20804c500, 0x0, 0x0)
        /usr/lib/go/src/pkg/go/printer/printer.go:1284 +0x96
    go/printer.Fprint(0x7f46a38d6170, 0xc208088060, 0xc20800e040, 0x5ce020, 0xc20804c500, 0x0, 0x0)
        /usr/lib/go/src/pkg/go/printer/printer.go:1291 +0xb0
    main.saveAST(0xc208024720, 0xc2080b00c0, 0x57, 0xc20800e040, 0x7f46a38d5ba0, 0xc20804c500, 0x0, 0x0, 0xc208024500, 0x0, ...)
        /home/mistone/go/src/github.com/zimmski/go-mutesting/cmd/go-mutesting/main.go:396 +0x202
    main.mainCmd(0xc20800e010, 0x3, 0x3, 0xa)
        /home/mistone/go/src/github.com/zimmski/go-mutesting/cmd/go-mutesting/main.go:257 +0x1dfc
    main.main()
        /home/mistone/go/src/github.com/zimmski/go-mutesting/cmd/go-mutesting/main.go:352 +0x69
    
    goroutine 19 [finalizer wait]:
    runtime.park(0x417dc0, 0x721070, 0x71fb89)
        /usr/lib/go/src/pkg/runtime/proc.c:1369 +0x89
    runtime.parkunlock(0x721070, 0x71fb89)
        /usr/lib/go/src/pkg/runtime/proc.c:1385 +0x3b
    runfinq()
        /usr/lib/go/src/pkg/runtime/mgc0.c:2644 +0xcf
    runtime.goexit()
        /usr/lib/go/src/pkg/runtime/proc.c:1445
    
    goroutine 22 [chan receive]:
    github.com/zimmski/go-mutesting/mutator/expression.(*MutatorRemoveTerm).Mutate(0x721570, 0x7f46a38d60a8, 0xc2080682d0, 0xc2080b0240)
        /home/mistone/go/src/github.com/zimmski/go-mutesting/mutator/expression/remove.go:76 +0x1b6
    github.com/zimmski/go-mutesting.(*mutateWalk).Visit(0xc2080b6020, 0x7f46a38d60a8, 0xc2080682d0, 0x0, 0x0)
        /home/mistone/go/src/github.com/zimmski/go-mutesting/walk.go:71 +0x115
    go/ast.Walk(0x7f46a38d5bd0, 0xc2080b6020, 0x7f46a38d60a8, 0xc2080682d0)
        /usr/lib/go/src/pkg/go/ast/walk.go:52 +0x58
    go/ast.Walk(0x7f46a38d5bd0, 0xc2080b6020, 0x7f46a38d60a8, 0xc208068300)
        /usr/lib/go/src/pkg/go/ast/walk.go:147 +0x1ead
    go/ast.Walk(0x7f46a38d5bd0, 0xc2080b6020, 0x7f46a38d6078, 0xc20806e3c0)
        /usr/lib/go/src/pkg/go/ast/walk.go:230 +0x43bc
    go/ast.walkStmtList(0x7f46a38d5bd0, 0xc2080b6020, 0xc20806e580, 0x3, 0x4)
        /usr/lib/go/src/pkg/go/ast/walk.go:32 +0xd1
    go/ast.Walk(0x7f46a38d5bd0, 0xc2080b6020, 0x7f46a38d5ec8, 0xc208068510)
        /usr/lib/go/src/pkg/go/ast/walk.go:224 +0x41e7
    go/ast.Walk(0x7f46a38d5bd0, 0xc2080b6020, 0x7f46a38d5e68, 0xc208068540)
        /usr/lib/go/src/pkg/go/ast/walk.go:342 +0xccb
    go/ast.walkDeclList(0x7f46a38d5bd0, 0xc2080b6020, 0xc20804a400, 0x13, 0x20)
        /usr/lib/go/src/pkg/go/ast/walk.go:38 +0xd1
    go/ast.Walk(0x7f46a38d5bd0, 0xc2080b6020, 0x7f46a38d5ba0, 0xc20804c500)
        /usr/lib/go/src/pkg/go/ast/walk.go:351 +0x2cb6
    github.com/zimmski/go-mutesting.func·001()
        /home/mistone/go/src/github.com/zimmski/go-mutesting/walk.go:51 +0x71
    created by github.com/zimmski/go-mutesting.MutateWalk
        /home/mistone/go/src/github.com/zimmski/go-mutesting/walk.go:54 +0xdb
    
    opened by mstone 5
  • Replace stmt deletion with noop assignment. Fixes #17

    Replace stmt deletion with noop assignment. Fixes #17

    This patch changes the statement mutator to replace statements with no-op assignments instead of just deleting them. For example:

    for i, x := range a {
        doSomething(i, x)
    }
    

    Used to be mutated into this:

    for i, x := range a {
    }
    

    Which does not compile. Now it is mutated into this:

    for i, x := range a {
        _, _ = i, x
    }
    

    This way it will compile but will have the same effect as deleting the line.

    opened by Logiraptor 4
  • example from README doesn't work, skips everthing

    example from README doesn't work, skips everthing

    Actual:

    $ go-mutesting --exec "$GOPATH/src/github.com/zimmski/go-mutesting/scripts/simple.sh" --exec-timeout 10 github.com/zimmski/go-mutesting/example 
    SKIP...
    < SKIP 11 more >
    The mutation score is NaN (0 passed, 0 failed, 12 skipped, total is 12)
    

    Expected:

    $ go-mutesting --exec "$GOPATH/src/github.com/zimmski/go-mutesting/scripts/simple.sh" --exec-timeout 10 github.com/zimmski/go-mutesting/example 
    PASS...
    < 11 more things >
    The mutation score is 0.750000 (6 passed, 2 failed, 0 skipped, total is 8)
    

    Looks like there's four more mutators in master now than there were when the README was written, but that still doesn't explain everything being skipped.

    opened by azylman 4
  • Add comparison expression mutator

    Add comparison expression mutator

    This PR adds a comparison expression mutator which generates mutations with every other type of comparison.

    ~~I noticed something odd with this - it seems to have caused the number of failures to drop for some tests? I don't quite understand this, as the tests would still surely fail (i.e. the mutants would escape) for the previous mutations, but some new tests are being generated too. I'd expect that the number of failures would stay the same or go up, as if a test passes when removing an expression but fails when changing the comparison then a mutant has still escaped (and the failed count should increase).~~

    ~~You can see the changes I had to make in a7d34c1.~~

    Turns out I had run go-mutesting on go-mutesting itself and CTRL-C'd half way through, which left me with some mutated source files! I still had to update the tests but the numbers look a lot better now. See cc978965e63c9869d453b89b030a7be37151c14a for the updated commit.

    This PR fixes #23.

    opened by jackwilsdon 3
  • Cannot go get on macOs

    Cannot go get on macOs

    Trying to run go get -t -v github.com/zimmski/go-mutesting/...

    The following happens:

    github.com/zimmski/osutil
    # github.com/zimmski/osutil
    ../../../../github.com/zimmski/osutil/capture.go:79: cannot assign to _Cmacro_stderr()
    ../../../../github.com/zimmski/osutil/capture.go:79: cannot assign to _Cmacro_stdout()
    ../../../../github.com/zimmski/osutil/capture.go:103: cannot assign to _Cmacro_stderr()
    ../../../../github.com/zimmski/osutil/capture.go:103: cannot assign to _Cmacro_stdout()
    

    What can I do to solve it?

    bug 
    opened by ladydascalie 2
  • Improper handling of structs instantiation

    Improper handling of structs instantiation

    Hi,

    Empty structs instantiations do not mutate correctly and produce code that does not compile.

    Example 1:

    return http.Header{}, err
    

    will be mutated to

    _, _ = http.Header, err
    

    The {} are missing which prevents the mutated code from compiling.

    Example 2:

    hdr["Hash"] = []string{hash}`
    

    will be mutated to

    _, _, _ = hdr, string, hash
    

    The mutated code is broken.

    Regards.

    opened by seborama 2
  • Reports PASS for an uncaught mutation

    Reports PASS for an uncaught mutation

    Here is a small Go package: https://github.com/vfaronov/debounce

    It has some tests:

    $ go test
    PASS
    ok  	vasiliy.faronov.name/exercises/debounce	16.845s
    

    I run go-mutesting on it:

    $ go-mutesting --do-not-remove-tmp-folder
    [...]
    PASS "/tmp/go-mutesting-2228185653/debounce.go.5" with checksum 385181e8103f64a5202da88bfad5a208
    [...]
    The mutation score is 0.947368 (18 passed, 0 failed, 4 duplicated, 1 skipped, total is 19)
    

    debounce.go.5 is the following mutation:

    $ diff debounce.go /tmp/go-mutesting-2228185653/debounce.go.5
    51c51
    < 						<-timer.C // Drain it.
    ---
    > 						_ = timer.C // Drain it.
    

    But the test suite does not actually detect it:

    $ cp /tmp/go-mutesting-2228185653/debounce.go.5 debounce.go
    $ go test
    PASS
    ok  	vasiliy.faronov.name/exercises/debounce	19.751s
    

    Why does go-mutesting report it as PASS?

    opened by vfaronov 1
  • Go-mutesting doesn't build on Go 1.10

    Go-mutesting doesn't build on Go 1.10

    When trying to build, I get this error message:

    $ go build cmd/go-mutesting/main.go
    # github.com/zimmski/osutil
    ../osutil/capture.go:79: cannot assign to _Cmacro_stderr()
    ../osutil/capture.go:79: cannot assign to _Cmacro_stdout()
    ../osutil/capture.go:103: cannot assign to _Cmacro_stderr()
    ../osutil/capture.go:103: cannot assign to _Cmacro_stdout()
    

    After some digging I believe the problem is this: https://github.com/golang/go/issues/25221

    Here's a workaround for building it for MacOS (darwin):

    docker run -it --name go-mutesting golang:1.9 /bin/bash -c "go get -t -v github.com/zimmski/go-mutesting/...; cd /go/src/github.com/zimmski/go-mutesting; GOOS=darwin go build cmd/go-mutesting/main.go"
    docker cp go-mutesting:/go/src/github.com/zimmski/go-mutesting/main go-mutesting
    docker rm go-mutesting
    

    And for Linux:

    docker run -it --name go-mutesting golang:1.9 /bin/bash -c "go get -t -v github.com/zimmski/go-mutesting/...; cd /go/src/github.com/zimmski/go-mutesting; go build cmd/go-mutesting/main.go"
    docker cp go-mutesting:/go/src/github.com/zimmski/go-mutesting/main go-mutesting
    docker rm go-mutesting
    
    opened by riccardofreixo 1
  • Bump gopkg.in/yaml.v2 from 2.2.2 to 2.2.8

    Bump gopkg.in/yaml.v2 from 2.2.2 to 2.2.8

    Bumps gopkg.in/yaml.v2 from 2.2.2 to 2.2.8.

    Commits
    • 53403b5 Optimize cases with long potential simple_keys (#555)
    • 1f64d61 Fix issue in simple_keys improvements (#548)
    • a95acef Update travis config to use latest go versions (#540)
    • 36babc3 Port stale simple_keys fix to v2 (#543)
    • 770b8da Fix Decorder doc typo (#494)
    • 1ed5951 Add Go 1.10-13 to travis setup.
    • f90ceb4 Fix check for non-map alias merging in v2 (#529)
    • 970885f Trivial style tuning on last change.
    • f221b84 Improve heuristics preventing CPU/memory abuse (#515)
    • bb4e33b Add logic to catch cases of alias abuse.
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Remove osutil dependency

    Remove osutil dependency

    The osutil package doesn't appear to be Windows compatible. The only method being used was CopyUtil. I copied the CopyUtil function into the main.go file and cleaned up the go.mod file.

    opened by bhasden 1
  • How to avoid mutation did not compile when the variable is a struct type?

    How to avoid mutation did not compile when the variable is a struct type?

    Mutation did not compile @@ -1122,7 +1122,7 @@ -- enqueue(&e.heapLoc) ++ _, _ = enqueue, e.heapLoc{} Exited with 2

    Mutation did not compile @@ -620,8 +620,7 @@ -- #e.stmts(n.Ninit) ++ _, _ = e.stmts, n.Ninit{} Exited with 2

    In the preceding example, heapLoc and Ninit are variables of the struct type. If the variables are used as structs, the compilation fails after mutation. The type of this variable should be correctly identified so that it can be compiled and executed. In the above mutated code, {} is unnecessary.

    opened by xjtuhorse 0
  • Remove skipped tests from total score

    Remove skipped tests from total score

    In order to maintain consistency in output results, skipped tests are removed from final total score. Skipped tests are still visible on result details, but final score will be calculated based only in passed and failed tests.

    Fixes #89

    opened by fbustamant3 0
Releases(v1.2)
  • v1.2(Jun 10, 2021)

  • v1.1(Nov 1, 2018)

    Changes from 1.0 to 1.1

    • Mutator for the comparison operators “<”, “<=”, “>” and “>=” to catch off-by-one problems
    • Complete blacklist of reserved and built-in keywords that should not be used as identifiers (reduces invalid mutations)
    • Metric for how many duplicated mutations were generated
    • Switched to go/loader for type checking entire packages (reduces invalid mutations)
    • Ignore mutations that cannot be compiled (reduces invalid mutations)
    • Skipped mutations now count towards the total mutation counter and not the duplicated counter
    • Support Go 1.10 and 1.11 and drop support for Go < 1.10
    • Support MacOSX
    Source code(tar.gz)
    Source code(zip)
Owner
Markus Zimmermann
Markus Zimmermann
A yaml data-driven testing format together with golang testing library

Specimen Yaml-based data-driven testing Specimen is a yaml data format for data-driven testing. This enforces separation between feature being tested

Design it, Run it 0 Jan 31, 2022
Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻

Gnomock – tests without mocks ??️ Spin up entire dependency stack ?? Setup initial dependency state – easily! ?? Test against actual, close to product

Yury Fedorov 845 Jun 26, 2022
Demo repository for Infrastructure as Code testing tools and frameworks.

Testing Infrastructure as Code Demo repository for Infrastructure as Code testing tools and frameworks. Maintainer M.-Leander Reimer (@lreimer), mario

M.-Leander Reimer 4 Jan 23, 2022
A Master list of Go Programming Tutorials, their write-ups, their source code and their current build status!

TutorialEdge TutorialEdge.net Go Tutorials ??‍?? ??‍?? Welcome to the TutorialEdge Go Repository! The goal of this repo is to be able to keep track of

TutorialEdge 274 Jun 15, 2022
Automatically generate Go test boilerplate from your source code.

gotests gotests makes writing Go tests easy. It's a Golang commandline tool that generates table driven tests based on its target source files' functi

Charles Weill 4k Jun 23, 2022
Fortio load testing library, command line tool, advanced echo server and web UI in go (golang). Allows to specify a set query-per-second load and record latency histograms and other useful stats.

Fortio Fortio (Φορτίο) started as, and is, Istio's load testing tool and now graduated to be its own project. Fortio is also used by, among others, Me

Fortio (Φορτίο) 2.6k Jun 24, 2022
:exclamation:Basic Assertion Library used along side native go testing, with building blocks for custom assertions

Package assert Package assert is a Basic Assertion library used along side native go testing Installation Use go get. go get github.com/go-playground/

Go Playgound 39 Apr 17, 2022
Expressive end-to-end HTTP API testing made easy in Go

baloo Expressive and versatile end-to-end HTTP API testing made easy in Go (golang), built on top of gentleman HTTP client toolkit. Take a look to the

Tom 725 May 12, 2022
Simple Go snapshot testing

Incredibly simple Go snapshot testing: cupaloy takes a snapshot of your test output and compares it to a snapshot committed alongside your tests. If t

Bradley Kemp 217 Jun 16, 2022
Clean database for testing, inspired by database_cleaner for Ruby

DbCleaner Clean database for testing, inspired by database_cleaner for Ruby. It uses flock syscall under the hood to make sure the test can runs in pa

Scott Le 140 Jun 28, 2022
Golang HTTP client testing framework

flute Golang HTTP client testing framework Presentation https://speakerdeck.com/szksh/flute-golang-http-client-testing-framework Overview flute is the

Shunsuke Suzuki 17 Apr 19, 2022
API testing framework inspired by frisby-js

frisby REST API testing framework inspired by frisby-js, written in Go Proposals I'm starting to work on frisby again with the following ideas: Read s

_Hofstadter 272 Jun 13, 2022
Extremely flexible golang deep comparison, extends the go testing package and tests HTTP APIs

go-testdeep Extremely flexible golang deep comparison, extends the go testing package. Latest news Synopsis Description Installation Functions Availab

Maxime Soulé 303 Jun 19, 2022
Minimal and Beautiful Go testing framework

Goblin A Mocha like BDD testing framework written in Go that requires no additional dependencies. Requires no extensive documentation nor complicated

null 852 Jun 26, 2022
A Comprehensive Coverage Testing System for The Go Programming Language

goc 中文页 | goc is a comprehensive coverage testing system for The Go Programming Language, especially for some complex scenarios, like system testing c

Qiniu Cloud 535 Jul 1, 2022
Go testing in the browser. Integrates with `go test`. Write behavioral tests in Go.

GoConvey is awesome Go testing Welcome to GoConvey, a yummy Go testing tool for gophers. Works with go test. Use it in the terminal or browser accordi

SmartyStreets 7.3k Jun 28, 2022
Testing API Handler written in Golang.

Gofight API Handler Testing for Golang Web framework. Support Framework Http Handler Golang package http provides HTTP client and server implementatio

Bo-Yi Wu 402 Jun 19, 2022
Library created for testing JSON against patterns.

Gomatch Library created for testing JSON against patterns. The goal was to be able to validate JSON focusing only on parts essential in given test cas

null 41 Dec 1, 2021
Testing framework for Go. Allows writing self-documenting tests/specifications, and executes them concurrently and safely isolated. [UNMAINTAINED]

GoSpec GoSpec is a BDD-style testing framework for the Go programming language. It allows writing self-documenting tests/specs, and executes them in p

Esko Luontola 112 Apr 5, 2022