faster JSON serialization for Go

Related tags

JSON ffjson

ffjson: faster JSON for Go

Build Status Fuzzit Status

ffjson generates static MarshalJSON and UnmarshalJSON functions for structures in Go. The generated functions reduce the reliance upon runtime reflection to do serialization and are generally 2 to 3 times faster. In cases where ffjson doesn't understand a Type involved, it falls back to encoding/json, meaning it is a safe drop in replacement. By using ffjson your JSON serialization just gets faster with no additional code changes.

When you change your struct, you will need to run ffjson again (or make it part of your build tools).

Blog Posts

Getting Started

If myfile.go contains the struct types you would like to be faster, and assuming GOPATH is set to a reasonable value for an existing project (meaning that in this particular example if myfile.go is in the myproject directory, the project should be under $GOPATH/src/myproject), you can just run:

go get -u
ffjson myfile.go
git add myfile_ffjson.go

Performance Status:

  • MarshalJSON is 2x to 3x faster than encoding/json.
  • UnmarshalJSON is 2x to 3x faster than encoding/json.


  • Unmarshal Support: Since v0.9, ffjson supports Unmarshaling of structures.
  • Drop in Replacement: Because ffjson implements the interfaces already defined by encoding/json the performance enhancements are transparent to users of your structures.
  • Supports all types: ffjson has native support for most of Go's types -- for any type it doesn't support with fast paths, it falls back to using encoding/json. This means all structures should work out of the box. If they don't, open a issue!
  • ffjson: skip: If you have a structure you want ffjson to ignore, add ffjson: skip to the doc string for this structure.
  • Extensive Tests: ffjson contains an extensive test suite including fuzz'ing against the JSON parser.

Using ffjson

ffjson generates code based upon existing struct types. For example, ffjson foo.go will by default create a new file foo_ffjson.go that contains serialization functions for all structs found in foo.go.

Usage of ffjson:

        ffjson [options] [input_file]

ffjson generates Go code for optimized JSON serialization.

  -go-cmd="": Path to go command; Useful for `goapp` support.
  -import-name="": Override import name in case it cannot be detected.
  -nodecoder: Do not generate decoder functions
  -noencoder: Do not generate encoder functions
  -w="": Write generate code to this path instead of ${input}_ffjson.go.

Your code must be in a compilable state for ffjson to work. If you code doesn't compile ffjson will most likely exit with an error.

Disabling code generation for structs

You might not want all your structs to have JSON code generated. To completely disable generation for a struct, add ffjson: skip to the struct comment. For example:

// ffjson: skip
type Foo struct {
   Bar string

You can also choose not to have either the decoder or encoder generated by including ffjson: nodecoder or ffjson: noencoder in your comment. For instance, this will only generate the encoder (marshal) part for this struct:

// ffjson: nodecoder
type Foo struct {
   Bar string

You can also disable encoders/decoders entirely for a file by using the -noencoder/-nodecoder commandline flags.

Using ffjson with go generate

ffjson is a great fit with go generate. It allows you to specify the ffjson command inside your individual go files and run them all at once. This way you don't have to maintain a separate build file with the files you need to generate.

Add this comment anywhere inside your go files:

//go:generate ffjson $GOFILE

To re-generate ffjson for all files with the tag in a folder, simply execute:

go generate

To generate for the current package and all sub-packages, use:

go generate ./...

This is most of what you need to know about go generate, but you can sese more about go generate on the golang blog.

Should I include ffjson files in VCS?

That question is really up to you. If you don't, you will have a more complex build process. If you do, you have to keep the generated files updated if you change the content of your structs.

That said, ffjson operates deterministically, so it will generate the same code every time it run, so unless your code changes, the generated content should not change. Note however that this is only true if you are using the same ffjson version, so if you have several people working on a project, you might need to synchronize your ffjson version.

Performance pitfalls

ffjson has a few cases where it will fall back to using the runtime encoder/decoder. Notable cases are:

  • Interface struct members. Since it isn't possible to know the type of these types before runtime, ffjson has to use the reflect based coder.
  • Structs with custom marshal/unmarshal.
  • Map with a complex value. Simple types like map[string]int is fine though.
  • Inline struct definitions type A struct{B struct{ X int} } are handled by the encoder, but currently has fallback in the decoder.
  • Slices of slices / slices of maps are currently falling back when generating the decoder.

Reducing Garbage Collection

ffjson already does a lot to help garbage generation. However whenever you go through the json.Marshal you get a new byte slice back. On very high throughput servers this can lead to increased GC pressure.

Tip 1: Use ffjson.Marshal() / ffjson.Unmarshal()

This is probably the easiest optimization for you. Instead of going through encoding/json, you can call ffjson. This will disable the checks that encoding/json does to the json when it receives it from struct functions.

	import ""

	buf, err := json.Marshal(&item)

	// AFTER:
	buf, err := ffjson.Marshal(&item)

This simple change is likely to double the speed of your encoding/decoding.

[![GoDoc][1]][2] [1]: [2]:

Tip 2: Pooling the buffer

On servers where you have a lot of concurrent encoding going on, you can hand back the byte buffer you get from json.Marshal once you are done using it. An example could look like this:

import ""

func Encode(item interface{}, out io.Writer) {
	// Encode
	buf, err := ffjson.Marshal(&item)
	// Write the buffer
	_,_ = out.Write(buf)
	// We are now no longer need the buffer so we pool it. 

Note that the buffers you put back in the pool can still be reclaimed by the garbage collector, so you wont risk your program building up a big memory use by pooling the buffers.

[![GoDoc][1]][2] [1]: [2]:

Tip 3: Creating an Encoder

There might be cases where you need to encode many objects at once. This could be a server backing up, writing a lot of entries to files, etc.

To do this, there is an interface similar to encoding/json, that allow you to create a re-usable encoder. Here is an example where we want to encode an array of the Item type, with a comma between entries:

import ""

func EncodeItems(items []Item, out io.Writer) {
        // We create an encoder.
	enc := ffjson.NewEncoder(out)
	for i, item := range items {
		// Encode into the buffer
		err := enc.Encode(&item)
		// If err is nil, the content is written to out, so we can write to it as well.
		if i != len(items) -1 {
			_,_ = out.Write([]byte{','})

Documentation: [![GoDoc][1]][2] [1]: [2]:

Tip 4: Avoid interfaces

We don't want to dictate how you structure your data, but having interfaces in your code will make ffjson use the golang encoder for these. When ffjson has to do this, it may even become slower than using json.Marshal directly.

To see where that happens, search the generated _ffjson.go file for the text Falling back, which will indicate where ffjson is unable to generate code for your data structure.

Tip 5: ffjson all the things!

You should not only create ffjson code for your main struct, but also any structs that is included/used in your json code.

So if your struct looks like this:

type Foo struct {
  V Bar

You should also make sure that code is generated for Bar if it is placed in another file. Also note that currently it requires you to do this in order, since generating code for Foo will check if code for Bar exists. This is only an issue if Foo and Bar are placed in different files. We are currently working on allowing simultaneous generation of an entire package.

Improvements, bugs, adding features, and taking ffjson new directions!

Please open issues in Github for ideas, bugs, and general thoughts. Pull requests are of course preferred :)

Similar projects

  • go-codec. Very good project, that also allows streaming en/decoding, but requires you to call the library to use.
  • megajson. This has limited support, and development seems to have almost stopped at the time of writing.


ffjson has recieved significant contributions from:


ffjson is licensed under the Apache License, Version 2.0

  • ffjson 2x slower for unmarshal than encoding/json

    ffjson 2x slower for unmarshal than encoding/json

    I'm experimenting with ffjson for a struct we use very heavily in our advertising platform. I was surprised to get a 2x slowdown in unmarshal performance when compared to encoding/json:

    BenchmarkUnmarshalRequestStdlib    20000         68320 ns/op        5953 B/op        142 allocs/op
    BenchmarkUnmarshalRequestFfjson    10000        119168 ns/op       17420 B/op        376 allocs/op

    I think I am using ffjson correctly, but perhaps I've made a mistake

    However, it may be that struct I'm unmarshalling is not a good fit for ffjson and so it might be of interest to the authors. The struct is quite complex with several sub structs, but it's main feature is that nearly every field is optional and is expressed as a pointer.

    Benchmark code is here (ffjson branch of Struct being unmarshalled is at

    question performance decoder 
    opened by iand 12
  • Continuous fuzzing

    Continuous fuzzing

    Proposing to integrate with Fuzzit for automated bug discovery.

    This fuzzes code generation and unmarshaling of some generated code. It's complementary to nice existing fuzzing of marshaling. For unmarshaling I generated from the nice test struct in the existing fuzzing.

    If you wanted to get marshaling running on the platform too I think it's possible to link up by fuzzing the entropy used to populate objects. If you're interested I can look into it.

    There's a successful run under my Travis account. The PR build will fail due to missing an API key. If you're interested in trying it setup is like this:

    • In Fuzzit create targets ffjson-generate ffjson-unmarshal.
    • In Fuzzit settings grab an API key. In repo settings in Travis paste it to envvar FUZZIT_API_KEY.
    opened by bookmoons 11
  • json.RawMessage doesn't work

    json.RawMessage doesn't work

    This library has some issues with json.RawMessage

    I think even outside of my example, I have seen some issues with *json.RawMessage as well.

    package main
    import (
    type Blah struct {
      x json.RawMessage `json:"value"`
    func main() {
       data := []byte(`{"value":null}`)
       var v Blah
       err := json.Unmarshal(data, &v)

    Would this work with ffjson. It is not working for me, at least in my payload. This is just an example. Can you please verify ?

    json.RawMessage is something that ffjson library should just leave alone and fall back on regular stdlib json.

    opened by nikunjy 11
  • Allow buffer reuse for less GC stress

    Allow buffer reuse for less GC stress

    With this addition there is now two ways to reuse the buffer.

    1. Reuse the fflib.Buffer and use the custom encoding. Good for file encoding, etc.
    2. Handing back the byte array after you have finished using it by useing fflib.Pool(bytes). good for webservers where everything is in separate goroutines.

    Note that fflib will now require Go 1.3+

    Benchmarks on a complex data structure:

    enconding/json:             37964 ns/op   34.90 MB/s        6088 B/op         27 allocs/op
    ffjson/bare                 17307 ns/op   76.56 MB/s        3035 B/op         24 allocs/op
    ffjson/pool                 13231 ns/op  100.14 MB/s         664 B/op         22 allocs/op
    ffjson/reuse-buffer         12716 ns/op  104.19 MB/s         584 B/op         20 allocs/op
    opened by klauspost 8
  • inception: Reset undefined fields when reusing objects.

    inception: Reset undefined fields when reusing objects.

    When reusing objects with sync.Pool, for example, and unmarshaling objects, this resets the old fields, which are not defined in the current JSON to their Zero values.

    For example:

    type M struct {
         Y []int
    type A struct {
         I int
     T *M
    type B struct {
         X int
         Z *A    `json:"Z,omitempty"`
         Y int   `json:"Y,omitempty"`
         K []int `json:"K,omitempty"`
    buf0 = []byte(`{"X":1,"Z":{"I":3, "T":[3,4]},"Y":4,"K":[1,2,3]}`)
    buf1 = []byte(`{"X":1,"K":[1,2,3]}`)
    x0 := B{}
    x1 := x0
    • This ensures that x0 != x1 by setting the fields in x1.Z = nil and x1.Y = 0 after marshalling all JSON objects.
    opened by cmeon 6
  • Error: Could not find source directory

    Error: Could not find source directory

    I get the following error while trying to run ffjson for a file

    Error: error=Could not find source directory: GOPATH=["/home/sztanpet/go/gocode"] REL="/home/sztanpet/go/destinyggchat" path="":

    must REL be a relative path or something?

    opened by sztanpet 6
  • package/type not correctly noted

    package/type not correctly noted

    In playing with ffjson with at 6fe04d580dfb71c9e34cbce2f4df9eefd1e1241e I get:

    ./response_ffjson.go:811: undefined: Time

    when I try to build something that uses it. It's a quick fix to import "time" and change:

    if uj.Expiration == nil {
                uj.Expiration = new(Time)


    if uj.Expiration == nil {
                uj.Expiration = new(time.Time)

    Point me in the right direction and I may can dig around and figure out what's happening.

    opened by bakins 6
  • Support unexported types

    Support unexported types

    I have a package with several types that must be unexported, but are marshalled/unmarshalled to/from json. Running ffjson returns several errors:

    > ffjson server.gen.go
    Error: Go Run Failed for: /tmp/ffjson-inception430580992.go
    # command-line-arguments
    /tmp/ffjson-inception430580992.go:19: cannot refer to unexported name chatservice.angoInMsg
    /tmp/ffjson-inception430580992.go:21: cannot refer to unexported name chatservice.angoClientArgsDataDisplayNotification
    /tmp/ffjson-inception430580992.go:23: cannot refer to unexported name chatservice.angoServerArgsDataAdd8
    /tmp/ffjson-inception430580992.go:25: cannot refer to unexported name chatservice.angoServerRetsDataDostuff
    /tmp/ffjson-inception430580992.go:27: cannot refer to unexported name chatservice.angoOutMsg
    /tmp/ffjson-inception430580992.go:29: cannot refer to unexported name chatservice.angoClientRetsDataAskQuestion
    /tmp/ffjson-inception430580992.go:31: cannot refer to unexported name chatservice.angoServerArgsDataDostuff
    /tmp/ffjson-inception430580992.go:33: cannot refer to unexported name chatservice.angoOutError
    /tmp/ffjson-inception430580992.go:37: cannot refer to unexported name chatservice.angoServerArgsDataAdd
    /tmp/ffjson-inception430580992.go:39: cannot refer to unexported name chatservice.angoServerRetsDataAdd8
    /tmp/ffjson-inception430580992.go:39: too many errors

    I believe that creating ffjson with unexported types should be possible.

    opened by GeertJohan 6
  • go tool: no such tool

    go tool: no such tool "6a" for appengine project.

    goapp does not have 6a. Can I use go tool 6a with go-cmd=goapp?

    $ go version
    go version go1.4.2 linux/amd64
    $ go tool
    $ goapp version
    go version go1.4.2 (appengine-1.9.21) linux/amd64
    $ goapp tool
    opened by ghost 5
  • Go 1.2 backport

    Go 1.2 backport

    I wanted to use ffjson on Go 1.2, which is the version shipped by Ubuntu 14.10. This set of patches fixes the build on that Go version (and corrects a small mistake in the README). It mostly shuffles some code around, but I also had to comment out a call to the assembler routine scanStringSSE, which the build system wasn't picking up.

    opened by larsmans 5
  • Travis-CI build failing

    Travis-CI build failing

    After committing 5c410f733010199d66597f0c8ff110a62846cf50 travis-ci has been failing the build.

    For example:

    I'm not able to reproduce the test failures locally. (OSX and Linux)

    opened by pquerna 5
  • add BytesCopy and StringCopy

    add BytesCopy and StringCopy

    Bytes() and String() cannot be used when using fflib.Buffer in sync.Pool , like:

    var bufPool = sync.Pool{
    	New: func() interface{} {
    		return fflib.NewBuffer([]byte{})
    func GetBuf() *fflib.Buffer {
    	return bufPool.Get().(*fflib.Buffer)
    func PutBuf(buf *fflib.Buffer) {

    So I add copy version of Bytes() and String()

    opened by l-i-h-m 0
  • Issue with boolean custom types in generated ffjson file

    Issue with boolean custom types in generated ffjson file

    As per my API definition, I have defined some custom types which actually are booleans: e.g.

    type EmbedParam bool

    in the generated code, I get

    	var tval bool
    	if bytes.Compare([]byte{'t', 'r', 'u', 'e'}, tmpb) == 0 {
    		tval = true
    	} else if bytes.Compare([]byte{'f', 'a', 'l', 's', 'e'}, tmpb) == 0 {
    		tval = false
    	} else {
    		err = errors.New("unexpected bytes for true/false value")
    		return fs.WrapErr(err)
    	j.Embed = &tval

    tval is defined as a bool instead of EmbedParam like Embed, meaning I get the following compilation error:

    .\****_ffjson.go:***:12: cannot use &tval (type *bool) as type *EmbedParam in assignment

    Could the local variable be defined using the custom type i.e. var tval EmbedParam?

    opened by acabarbaye 0
  • golang 1.16: fs.FileMode undefined (type *v1.FFLexer has no field or method FileMode)

    golang 1.16: fs.FileMode undefined (type *v1.FFLexer has no field or method FileMode)

    On a Fedora 34 VM with golang 1.16, the ffjson generated code (archive_ffjson.go) fails to compile:

    GO111MODULE=on go build -mod=vendor -compiler gc -tags "  "  ./cmd/containers-storage
    pkg/archive/archive_ffjson.go:13:2: imported and not used: "io/fs"
    pkg/archive/archive_ffjson.go:1652:17: fs.FileMode undefined (type *v1.FFLexer has no field or method FileMode)
    make: *** [Makefile:54: containers-storage] Error 2

    I do not have this problem on my Ubuntu VM using golang 1.15 on the same code.

    I tried with the "latest" ffjson commit aa0246cd15f7

    opened by cevich 0
  • add fully-worked tiny example, mention it works in tinygo add fully-worked tiny example, mention it works in tinygo

    I found it difficult to get started from the README, so I created a fully-worked example showing off the bare minimum steps from start to finish to use ffjson.

    Happily, the result appears to work in tinygo, which seems worth mentioning.

    opened by dkegel-fastly 0
  • "make test" fails

    With go 1.16.3, doing make test fails with

    go install
    go install: version is required when current directory is not in a module
    	Try 'go install[email protected]' to install the latest version

    Following the suggestion does not work; it seems to produce the same error.

    Working around this by creating a trivial go.mod, then doing go mod tidy, fails with cannot find module providing package module git ls-remote -q origin in /Users/dkegel/go/pkg/mod/cache/vcs/d5727a80ed1da50c9aefa6067dac442a4ec8c5b1a6b78d2164bed13a43544b3d: exit status 128:

    which is rather like a modern version of

    opened by dkegel-fastly 0
  • Wrong google/uuid serialization

    Wrong google/uuid serialization

    ffjson generates wrong marshaler for github/google/uuid#UUID type

    It implements fmt.Stringer, encoding.BinaryMarshaler and encoding.TextMarshaler.

    Insead of expected


    I get



    package ffjson_uuid
    import ""
    //go:generate ffjson main.go
    type Foo struct {
    	Bar uuid.UUID


    // MarshalJSONBuf marshal buff to json - template
    func (j *Foo) MarshalJSONBuf(buf fflib.EncodingBuffer) error {
    	if j == nil {
    		return nil
    	var err error
    	var obj []byte
    	_ = obj
    	_ = err
    	for i, v := range j.Bar {
    		if i != 0 {
    		fflib.FormatBits2(buf, uint64(v), 10, false)
    	return nil
    opened by prochac 0
Paul Querna
See also
Paul Querna
JSON diff library for Go based on RFC6902 (JSON Patch)

jsondiff jsondiff is a Go package for computing the diff between two JSON documents as a series of RFC6902 (JSON Patch) operations, which is particula

William Poussier 215 Dec 4, 2022
Fast JSON encoder/decoder compatible with encoding/json for Go

Fast JSON encoder/decoder compatible with encoding/json for Go

Masaaki Goshima 2k Jan 6, 2023
Package json implements encoding and decoding of JSON as defined in RFC 7159

Package json implements encoding and decoding of JSON as defined in RFC 7159. The mapping between JSON and Go values is described in the documentation for the Marshal and Unmarshal functions

High Performance, Kubernetes Native Object Storage 2 Jun 26, 2022
Json-go - CLI to convert JSON to go and vice versa

Json To Go Struct CLI Install Go version 1.17 go install

Samit Ghimire 7 Jul 29, 2022
JSON Spanner - A Go package that provides a fast and simple way to filter or transform a json document

JSON SPANNER JSON Spanner is a Go package that provides a fast and simple way to

null 6 Sep 14, 2022
Abstract JSON for golang with JSONPath support

Abstract JSON Abstract JSON is a small golang package provides a parser for JSON with support of JSONPath, in case when you are not sure in its struct

Stepan Pyzhov 145 Jan 5, 2023
Fast JSON parser and validator for Go. No custom structs, no code generation, no reflection

fastjson - fast JSON parser and validator for Go Features Fast. As usual, up to 15x faster than the standard encoding/json. See benchmarks. Parses arb

Aliaksandr Valialkin 1.7k Jan 5, 2023
Small utility to create JSON objects

gjo Small utility to create JSON objects. This was inspired by jpmens/jo. Support OS Mac Linux Windows Requirements Go 1.1.14~ Git Installtion Build $

skanehira 116 Dec 8, 2022
A Go package for handling common HTTP JSON responses.

go-respond A Go package for handling common HTTP JSON responses. Installation go get Usage The goal of go-respond is to

Nick Law 50 Sep 26, 2022
JSON query in Golang

gojq JSON query in Golang. Install go get -u This library serves three purposes: makes parsing JSON configuration file much easie

Qian Chen 183 Dec 28, 2022
Automatically generate Go (golang) struct definitions from example JSON

gojson gojson generates go struct definitions from json or yaml documents. Example $ curl -s | gojson

Aditya Mukerjee 2.6k Jan 1, 2023
A JSON diff utility

JayDiff A JSON diff utility. Install Downloading the compiled binary Download the latest version of the binary: releases extract the archive and place

Guillaume de Sagazan 95 Dec 11, 2022
Fast and flexible JSON encoder for Go

Jettison Jettison is a fast and flexible JSON encoder for the Go programming language, inspired by bet365/jingo, with a richer features set, aiming at

William Poussier 141 Dec 21, 2022
Create go type representation from json

json2go Package json2go provides utilities for creating go type representation from json inputs. Json2go can be used in various ways: CLI tool Web pag

Maciek ZajÄ…c 112 Dec 26, 2022
Console JSON formatter with query feature

Console JSON formatter with query feature. Install: $ go get Usage: Usage of jsonf: -c=true: colorize output -d=false: de

Artem Andreenko 63 Dec 4, 2022
Fluent API to make it easier to create Json objects.

Jsongo Fluent API to make it easier to create Json objects. Install go get Usage To create this: { "name":"Ricar

Ricardo Longa 107 Nov 7, 2022
Arbitrary transformations of JSON in Golang

kazaam Description Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang. This functionality provides

Qntfy 245 Dec 18, 2022
Parsing JSON is a hassle in golang

GoJSON Parsing JSON is a hassle in golang. This package will allow you to parse and search elements in a json without structs. Install gojson go get g

swaraj18 25 Nov 12, 2021
A JSON stream parser for Go

pjson A JSON stream parser for Go Example The example below prints all string values from a JSON document. package

Josh Baker 63 Oct 3, 2022