Auto-gen fuzzing wrappers from normal code. Automatically find buggy call sequences, including data races & deadlocks. Supports rich signature types.

Overview

fzgen

fzgen auto-generates fuzzing wrappers for Go 1.18, optionally finds problematic API call sequences, can automatically wire outputs to inputs across API calls, and supports rich types such as structs, maps, slices, named types, and common interfaces.

Why?

Fewer bugs, happy Gophers.

Modern fuzzing has had a large amount of success and can be almost eerily smart, but has been most heavily used in the realm of security, often with a focus on parsing untrusted inputs.

Security is critical, but:

  1. Eventually, the bigger success for fuzzing might be finding correctness & stability problems in a broader set of code bases, and beyond today's more common security focus.
  2. There is also a large opportunity to make fuzzing easier to pick up by a broader community.

This project aims to pitch in on those two fronts.

If enough people work to make the fuzzing ecosystem accessible enough, "coffee break fuzzing" might eventually become as common as unit tests. And of course, increased adoption of fuzzing helps security as well. 😊

Quick Start: Install & Automatically Create Fuzz Targets

Starting from an empty directory, create a module and install the dev version of Go 1.18 via gotip:

$ go mod init example
$ go install golang.org/dl/[email protected]
$ gotip download

Download and install the fzgen binary from source, as well as add its fuzzer to our go.mod:

$ go install github.com/thepudds/fzgen/cmd/[email protected]
$ go get github.com/thepudds/fzgen/fuzzer

Use fzgen to automatically create a set of fuzz targets -- in this case for the encoding/ascii85 package from the Go standard library:

$ fzgen encoding/ascii85
fzgen: created autofuzz_test.go

That's it -- now we can start fuzzing!

$ gotip test -fuzz=Fuzz_Encode

Within a few seconds, you should get a crash:

[...]
fuzz: minimizing 56-byte failing input file
fuzz: elapsed: 0s, minimizing
--- FAIL: Fuzz_Encode (0.06s)

Without any manual work, you just found a bug in the standard library. (It's a very minor bug though -- probably at the level of "perhaps the doc could be more explicit about an expected panic").

That's enough for you to get started on your own, but let's also briefly look at a more interesting example.

Example: Easily Finding a Data Race

Again starting from an empty directory, we'll set up a module, and add the fzgen fuzzer to go.mod:

$ go mod init temp
$ go get github.com/thepudds/fzgen/fuzzer

Next, we automatically create a new fuzz target. This time:

  • We ask fzgen to "chain" a set of methods together in a calling sequence controlled by fzgen.Fuzzer (via the -chain argument).
  • We also tell fzgen that it should in theory be safe to do parallel execution of those methods across multiple goroutines (via the -parallel argument).
$ fzgen -chain -parallel github.com/thepudds/fzgen/examples/inputs/race
fzgen: created autofuzzchain_test.go

That's it! Let's get fuzzing.

This time, we also enable the race detector as we fuzz:

$ gotip test -fuzz=. -race

This is a harder challenge than our first example, but within several minutes or so, you should get a data race detected:

--- FAIL: Fuzz_NewMySafeMap (4.26s)
    --- FAIL: Fuzz_NewMySafeMap (0.01s)
        testing.go:1282: race detected during execution of test

If we want to see what exact calls triggered this, along with their input arguments, we can set a fzgen debug flag asking it to show us a reproducer, and then ask 'go test' to re-run the failing input that was just found. (Your failing example will almost certainly have a different filename and show a different pattern of calls).

$ export FZDEBUG=repro=1                   # On Windows:  set FZDEBUG=repro=1
$ gotip test -run=./9800b52 -race

This will output a snippet of valid Go code that represents the reproducer:

        // PLANNED STEPS (loop count: 1, spin: true)

        Fuzz_MySafeMap_Store(
                [16]uint8{152,152,152,152,152,152,152,152,152,152,152,152,152,152,152,152},
                &raceexample.Request{Answer:42,},
        )

        var wg sync.WaitGroup
        wg.Add(2)

        // Execute next steps in parallel.
        go func() {
                defer wg.Done()
                Fuzz_MySafeMap_Load(
                        [16]uint8{152,152,152,152,152,152,152,152,152,152,152,152,152,152,152,152},
                )
        }()
        go func() {
                defer wg.Done()
                Fuzz_MySafeMap_Load(
                        [16]uint8{152,152,152,152,152,152,152,152,152,152,152,152,152,152,152,152},
                )
        }()
        wg.Wait()

        // Resume sequential execution.
        Fuzz_MySafeMap_Load(
                [16]uint8{0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0},
        )

[...]
    --- FAIL: Fuzz_NewMySafeMap (0.01s)
        testing.go:1282: race detected during execution of test

Note that just running a regular test under the race detector might not catch this bug, including because the race detector only finds data races that happen at runtime, which means a diversity of code paths and input data is imporant for the race detector to do its job. fz.Chain helps supply those code paths and data -- in this case, usually hundreds of thousands of coverage-guided variations before hitting the data race.

For this particular bug to be observable by the race detector, it requires:

  1. A Store must complete, then be followed by two Loads, and all three must use the same key.
  2. The Store must have certain payload data (Answer: 42).
  3. The two Loads must happen concurrently.
  4. Prior to the two Loads, no other Store can update the key to have a non-matching payload.

Here, the 42 seen in the reproducer must be 42. On the other hand, the exact value of 152,152,152,... in the key doesn't matter, but what does matter is that the same key value must be used across this sequence of three calls to trigger the bug.

fz.Chain has logic to sometimes re-use input arguments of the same type across different calls (e.g., to make it easier to have a meaningful Get(key) following a Put(key)), as well logic to feed outputs of one step as the input to a subsequent step, which is helpful in other cases.

Rewinding slightly, if you look at the code you just automatically generated in autofuzzchain_test.go, you can see there are a set of possible "steps" listed that each invoke the target, but the most important line there is:

    // Execute a specific chain of steps, with the count, sequence and arguments controlled by fz.Chain
    fz.Chain(steps, fuzzer.ChainParallel)

In other words, at execution time, fz.Chain takes over and guides the underlying fuzzing engine towards interesting calling patterns & arguments, while simultaneously exploring what runs in parallel vs. sequentially, the timing of parallel calls, and so on.

Example: Finding a Real Concurrency Bug in Real Code

The prior example was a small but challenging example that takes a few minutes of fuzzing to find, but you can see an example of a deadlock found in real code here from the xsync.Map from github.com/puzpuzpuz/xsync.

In this case, it usually takes fz.Chain several million coverage-guided variations over a few hours of executing the generated autogenchain_test.go before it finds the deadlock.

Interestingly, the deadlock then typically only reproduces about 1 out of 1,000 attempts for a particular discovered problematic calling pattern and arguments.

Fortunately, once a problem is reported, we can paste the output of FZDEBUG=repro=1 into a standalone_repro_test.go file and use the handy -count argument in a normal go test -count=10000 invocation, and now we can reproduce the deadlock cleanly on demand. At that point, the reproducer is completely standalone and does not rely on fzgen any longer.

fzgen status

  • fzgen is still a work in progress, but hopefully will soon be approaching beta quality.
  • Emitting reproducers for a chain is currently best effort, but the intent is to improve to creating a complete standalone reproducer.
  • Corpus encoding in particular will change in the near term.
  • Roughly by the time of Go 1.18 graduates from Beta, the current intent is that fzgen will reach a 1.0 status.
    • By 1.0, fzgen will have a stable corpus encoding, or an equivalent (such as perhaps the ability to programmatically set an encoding version number to keep using a corpus that is an older fzgen encoding).

What next?

Any and all feedback is welcome! 😀

Please feel free to open a new issue here, or to contact the author on Twitter (@thepudds1).

The roadmap issue (TODO) in particular is a reasonable starting place.

If you made it this far -- thanks!

Issues
  • failures running on github.com/tailscale/tailscale

    failures running on github.com/tailscale/tailscale

    github.com/tailscale/tailscale at 5a9914a92fa72d61510b5f2aacd6665e3f8fa659

    Using Go 1.18:

    $ fzgen ./...
    2021/12/17 10:34:42 internal error: package "errors" without types was imported from "tailscale.com/util/pidowner"
    

    Using the Tailscale Go fork (based on 1.17):

    $ fzgen ./...
    panic: runtime error: index out of range [0] with length 0
    
    goroutine 1 [running]:
    github.com/thepudds/fzgen/gen.avoidCollision(0x14003ef2730, 0x0, 0x14003cf4cd0, {0x14001cc5800, 0x2, 0x2})
    	/Users/josh/pkg/mod/github.com/thepudds/[email protected]/gen/genfuncs.go:634 +0x348
    github.com/thepudds/fzgen/gen.emitIndependentWrapper(0x140070a5d28, {{0x140001e6e88, 0x11}, {0x14000a680bc, 0x3}, {0x14000a33e18, 0x15}, {0x140062b7500, 0x19}, 0x14003ed6410}, ...)
    	/Users/josh/pkg/mod/github.com/thepudds/[email protected]/gen/genfuncs.go:207 +0x690
    github.com/thepudds/fzgen/gen.emitIndependentWrappers({0x16d6837f6, 0x5}, {0x14006310000, 0x449, 0x471}, {0x1, 0x1, {0x1029a9e23, 0x4}, 0x0})
    	/Users/josh/pkg/mod/github.com/thepudds/[email protected]/gen/genfuncs.go:86 +0x4c0
    github.com/thepudds/fzgen/gen.FzgenMain()
    	/Users/josh/pkg/mod/github.com/thepudds/[email protected]/gen/fzgen.go:130 +0x56c
    main.main()
    	/Users/josh/pkg/mod/github.com/thepudds/[email protected]/cmd/fzgen/main.go:10 +0x20
    

    I'm not fussed about this, just reporting it in case it is useful.

    opened by josharian 9
  • remove emitted comment about goimports

    remove emitted comment about goimports

    It's easy enough to run goimports programatically (it's exposed as a package, not just an executable), so fzgen should just do it, rather than:

    // if needed, fill in imports or run 'goimports'
    
    opened by josharian 2
  • gen: make

    gen: make "." the default constructor regex pattern and improve emitted comments

    • A wider set of candidate constructors are examined by default.
      • Previously, the regexp '^New' was the default, but now '.' is the default -ctor regexp.
      • Hopefully people will only rarely use the -ctor flag now.
    • We no longer emit a comment suggest running goimports given we have been doing the equivalent via golang.org/x/tools/imports API. (Fixes #4).
    • Other smaller improvements to emitted comments or skip messages.
    opened by thepudds 0
  • gen: continue after non-fatal errors for multi-package targets without chains

    gen: continue after non-fatal errors for multi-package targets without chains

    • Continue and skip creating a useless file after non-fatal errors for multi-package targets when -chain is not specified.
    • Handle unnamed receivers.
    • Upgrade to master for golang.org/x/tools.

    Fixes #5

    opened by thepudds 0
  • running the tailscale test script fails

    running the tailscale test script fails

            fzgen: created wgengine/filter/autofuzz_test.go
            fzgen: created wgengine/magicsock/autofuzz_test.go
            fzgen: created wgengine/monitor/autofuzz_test.go
            fzgen: created wgengine/netstack/autofuzz_test.go
            fzgen: created wgengine/router/autofuzz_test.go
            fzgen: created wgengine/wgcfg/autofuzz_test.go
            fzgen: skipping tailscale.com/wgengine/wgcfg/nmcfg: unsupported parameters: WGCfg takes tailscale.com/types/logger.Logf
            fzgen: created wgengine/wglog/autofuzz_test.go
            fzgen: skipping tailscale.com/words: no fuzzable functions found: Scales has 0 input params
            fzgen: created 91 files
            > stdout 'fzgen: created 91 files'
            > exec gotip test -exec=true ./...
            [stderr]
            package tailscale.com/cmd/tailscaled
                    imports tailscale.com/ipn/ipnserver
                    imports tailscale.com/ipn/ipnlocal
                    imports tailscale.com/wgengine
                    imports tailscale.com/net/tstun
                    imports inet.af/netstack/tcpip
                    imports inet.af/netstack/atomicbitops
                    imports inet.af/netstack/state
                    imports inet.af/netstack/state/wire
                    imports inet.af/netstack/gohacks: build constraints exclude all Go files in $WORK/gopath/pkg/mod/inet.af/[email protected]/gohacks
            [exit status 1]
            FAIL: testscripts/external_tailscale.txt:27: unexpected command failure
            
    --- FAIL: TestScripts (0.05s)
        --- FAIL: TestScripts/external_tailscale (88.29s)
    FAIL
    

    presumably this is just a fail b/c tailscale has changed in the meantime perhaps?

    opened by jasikpark 0
  • What would I need to work on to add support for logrus.FieldLogger as a function argument?

    What would I need to work on to add support for logrus.FieldLogger as a function argument?

    // skipping Fuzz_Bits_Check because parameters include func, chan, or unsupported interface: github.com/sirupsen/logrus.FieldLogger

    ^^ the project I'm testing out fzgen w/ to setup fuzzers in uses l logrus.FieldLogger as the first argument to 90% of it's functions - I suppose maybe I should see how they test things...

    opened by jasikpark 0
  • Is `export FZDEBUG=repro=1` only useful for chained fuzzers?

    Is `export FZDEBUG=repro=1` only useful for chained fuzzers?

    When running the reproduction with a regular generated fuzzer, I only get a backtrace, rather than having the full code used like shown in the README / that I was able to replicate in the parallel+chain tutorial in the readme

    func Fuzz_TimerWheel_NewTimerWheel(f *testing.F) {
    	f.Fuzz(func(t *testing.T, data []byte) {
    		var min time.Duration
    		var max time.Duration
    		fz := fuzzer.NewFuzzer(data)
    		fz.Fill(&min, &max)
    
    		NewTimerWheel(min, max)
    	})
    }
    

    doesn't produce a length test description.. I suppose maybe it wouldn't, since presumably it's just a single function call?

    opened by jasikpark 0
  • autofuzzed errors for

    autofuzzed errors for "Less(i,j int) bool" tend to be spurious

    Not sure if this counts as a bug (I am torturing fzgen in ways you might not have imagined -- I automated it against the benchmarks in the "bent" collection), but if a type has a Less(i,j int) bool method (for sorting, e.g.) it's pretty easy for random values of i and j to trigger a range panic. For example: https://github.com/Masterminds/semver/blob/master/collection.go#L16.

    The bent-fuzz experiment is also a work-in-process; in particular, it lacks documentation.

    opened by dr2chase 0
Releases(v0.4.2)
  • v0.4.2(Jan 3, 2022)

    What's Changed

    • 'fzgen ./...' now works, placing output files in directory of target packages https://github.com/thepudds/fzgen/pull/11
    • A wider set of candidate constructors are examined by default https://github.com/thepudds/fzgen/pull/16
      • Previously, the regexp '^New' was the default, but now '.' is the default -ctor regexp
      • Hopefully the -ctor flag is now less commonly used
    • Multiple chains for a single package are now created https://github.com/thepudds/fzgen/pull/12
    • Bug fixes that enable more targets to work by default https://github.com/thepudds/fzgen/pull/9
    • Performance improvements
    • Other smaller improvements, such as better messages explaining why a target package was skipped, and better emitted comments.

    Thanks especially to @josharian (Tailscale) for the helpful bug reports!

    Full v0.4.2 Changelog: https://github.com/thepudds/fzgen/compare/v0.4.1...v0.4.2

    Please see the v0.4.0 release notes for larger recent updates.

    Source code(tar.gz)
    Source code(zip)
  • v0.4.1(Dec 17, 2021)

    This release includes:

    • Replace emitted nil checks for user arguments with fuzzer.Chain checks at execution time.

    Please see the v0.4.0 release for larger recent updates (https://github.com/thepudds/fzgen/releases/tag/v0.4.0).

    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Dec 16, 2021)

    This release includes:

    • Rename to fzgen (from older https://github.com/thepudds/fzgo), with new repo.
    • Support cmd/go 1.18 fuzzing.
    • Better runtime chaining of function calls under test. (See README).
    • Option to emit valid Go code for standalone crash reproducers (via FZDEBUG=repro=1).
    • Automatic roundtrip tests for targets that implement encoding.BinaryMarshaler/TextMarshaler interfaces.
    • Better corpus encoding.
    • Updated README.
    • An intention to bring to 1.0.0 quality.
    Source code(tar.gz)
    Source code(zip)
Owner
thepudds
thepudds
Package strit introduces a new type of string iterator, along with a number of iterator constructors, wrappers and combinators.

strit Package strit (STRing ITerator) assists in development of string processing pipelines by providing a simple iteration model that allows for easy

Maxim 84 Jun 21, 2022
:triangular_ruler:gofmtmd formats go source code block in Markdown. detects fenced code & formats code using gofmt.

gofmtmd gofmtmd formats go source code block in Markdown. detects fenced code & formats code using gofmt. Installation $ go get github.com/po3rin/gofm

po3rin 90 Apr 3, 2022
:book: A Golang library for text processing, including tokenization, part-of-speech tagging, and named-entity extraction.

prose prose is a natural language processing library (English only, at the moment) in pure Go. It supports tokenization, segmentation, part-of-speech

Joseph Kato 2.9k Aug 1, 2022
Automation Tool to auto generate markdown notes from online classes/talks/presentations.

autonotes Automation tool to autocapture screenshots and join them with a supplied .srt or .txt file and output a notes file in markdown. Problem? Wat

Marcos Soares 3 Aug 29, 2021
This is a simple text completion/editing/auto-correction tool I created in Go.

Auto Text Completion Tool This is a simple text completion/editing/auto-correction tool I created in Go. The tool receives the following arguments: Th

null 1 Oct 28, 2021
This package provides Go (golang) types and helper functions to do some basic but useful things with mxGraph diagrams in XML, which is most famously used by app.diagrams.net, the new name of draw.io.

Go Draw - Golang MX This package provides types and helper functions to do some basic but useful things with mxGraph diagrams in XML, which is most fa

null 1 Nov 30, 2021
gomtch - find text even if it doesn't want to be found

gomtch - find text even if it doesn't want to be found Do your users have clever ways to hide some terms from you? Sometimes it is hard to find forbid

Nicolas Augusto Sassi 27 Apr 22, 2022
Produces a set of tags from given source. Source can be either an HTML page, Markdown document or a plain text. Supports English, Russian, Chinese, Hindi, Spanish, Arabic, Japanese, German, Hebrew, French and Korean languages.

Tagify Gets STDIN, file or HTTP address as an input and returns a list of most popular words ordered by popularity as an output. More info about what

ZoomIO 23 Jul 17, 2022
Geziyor, a fast web crawling & scraping framework for Go. Supports JS rendering.

Geziyor Geziyor is a blazing fast web crawling and web scraping framework. It can be used to crawl websites and extract structured data from them. Gez

null 1.7k Aug 1, 2022
:evergreen_tree: Parses indented code and returns a tree structure.

codetree Parses indented code (Python, Pug, Stylus, Pixy, codetree, etc.) and returns a tree structure. Installation go get github.com/aerogo/codetree

Aero 20 Mar 6, 2022
:zap: Transfer files over wifi from your computer to your mobile device by scanning a QR code without leaving the terminal.

$ qrcp Transfer files over Wi-Fi from your computer to a mobile device by scanning a QR code without leaving the terminal. You can support development

Claudio d'Angelis 8.7k Aug 7, 2022
Go package for syntax highlighting of code

syntaxhighlight Package syntaxhighlight provides syntax highlighting for code. It currently uses a language-independent lexer and performs decently on

Sourcegraph 253 Jul 27, 2022
Search for Go code using syntax trees

gogrep GO111MODULE=on go get mvdan.cc/gogrep Search for Go code using syntax trees. Work in progress. gogrep -x 'if $x != nil { return $x, $*_ }' In

Daniel Martí 471 Jul 27, 2022
Frecuency of ASCII characters in Typescript and Javascript code

Tool to traverse Javascript and Typescript codebases counting the number of occurrences of each ASCII character. Usefull for optimizing tokenizers / lexers

Elian Cordoba 0 Jan 31, 2022
Pryrite, interactively execute shell code blocks in a markdown file

Pryrite Pryrite is a command line tool that interactively runs executable blocks in a markdown file. One can think of pryrite as a console REPL/debugg

Rama Shenai 165 Jul 14, 2022
Extract structured data from web sites. Web sites scraping.

Dataflow kit Dataflow kit ("DFK") is a Web Scraping framework for Gophers. It extracts data from web pages, following the specified CSS Selectors. You

Dmitry Narizhnykh 538 Aug 3, 2022
Encoding and decoding for fixed-width formatted data

fixedwidth Package fixedwidth provides encoding and decoding for fixed-width formatted Data. go get github.com/ianlopshire/go-fixedwidth Usage Struct

Ian Lopshire 64 Jul 22, 2022
Gotabulate - Easily pretty-print your tabular data with Go

Gotabulate - Easily pretty-print tabular data Summary Go-Tabulate - Generic Go Library for easy pretty-printing of tabular data. Installation go get g

Vadim Kravcenko 286 Jul 27, 2022
Faker is a Go library that generates fake data for you.

Faker is a Go library that generates fake data for you. Whether you need to bootstrap your database, create good-looking XML documents, fill-in your p

Jonathan Schweder 295 Aug 1, 2022