Fast JSON parser and validator for Go. No custom structs, no code generation, no reflection

Overview

Build Status GoDoc Go Report codecov

fastjson - fast JSON parser and validator for Go

Features

  • Fast. As usual, up to 15x faster than the standard encoding/json. See benchmarks.
  • Parses arbitrary JSON without schema, reflection, struct magic and code generation contrary to easyjson.
  • Provides simple API.
  • Outperforms jsonparser and gjson when accessing multiple unrelated fields, since fastjson parses the input JSON only once.
  • Validates the parsed JSON unlike jsonparser and gjson.
  • May quickly extract a part of the original JSON with Value.Get(...).MarshalTo and modify it with Del and Set functions.
  • May parse array containing values with distinct types (aka non-homogenous types). For instance, fastjson easily parses the following JSON array [123, "foo", [456], {"k": "v"}, null].
  • fastjson preserves the original order of object items when calling Object.Visit.

Known limitations

  • Requies extra care to work with - references to certain objects recursively returned by Parser must be released before the next call to Parse. Otherwise the program may work improperly. The same applies to objects returned by Arena. Adhere recommendations from docs.
  • Cannot parse JSON from io.Reader. There is Scanner for parsing stream of JSON values from a string.

Usage

One-liner accessing a single field:

	s := []byte(`{"foo": [123, "bar"]}`)
	fmt.Printf("foo.0=%d\n", fastjson.GetInt(s, "foo", "0"))

	// Output:
	// foo.0=123

Accessing multiple fields with error handling:

        var p fastjson.Parser
        v, err := p.Parse(`{
                "str": "bar",
                "int": 123,
                "float": 1.23,
                "bool": true,
                "arr": [1, "foo", {}]
        }`)
        if err != nil {
                log.Fatal(err)
        }
        fmt.Printf("foo=%s\n", v.GetStringBytes("str"))
        fmt.Printf("int=%d\n", v.GetInt("int"))
        fmt.Printf("float=%f\n", v.GetFloat64("float"))
        fmt.Printf("bool=%v\n", v.GetBool("bool"))
        fmt.Printf("arr.1=%s\n", v.GetStringBytes("arr", "1"))

        // Output:
        // foo=bar
        // int=123
        // float=1.230000
        // bool=true
        // arr.1=foo

See also examples.

Security

  • fastjson shouldn't crash or panic when parsing input strings specially crafted by an attacker. It must return error on invalid input JSON.
  • fastjson requires up to sizeof(Value) * len(inputJSON) bytes of memory for parsing inputJSON string. Limit the maximum size of the inputJSON before parsing it in order to limit the maximum memory usage.

Performance optimization tips

  • Re-use Parser and Scanner for parsing many JSONs. This reduces memory allocations overhead. ParserPool may be useful in this case.
  • Prefer calling Value.Get* on the value returned from Parser instead of calling Get* one-liners when multiple fields must be obtained from JSON, since each Get* one-liner re-parses the input JSON again.
  • Prefer calling once Value.Get for common prefix paths and then calling Value.Get* on the returned value for distinct suffix paths.
  • Prefer iterating over array returned from Value.GetArray with a range loop instead of calling Value.Get* for each array item.

Fuzzing

Install go-fuzz & optionally the go-fuzz-corpus.

go get -u github.com/dvyukov/go-fuzz/go-fuzz github.com/dvyukov/go-fuzz/go-fuzz-build

Build using go-fuzz-build and run go-fuzz with an optional corpus.

mkdir -p workdir/corpus
cp $GOPATH/src/github.com/dvyukov/go-fuzz-corpus/json/corpus/* workdir/corpus
go-fuzz-build github.com/valyala/fastjson
go-fuzz -bin=fastjson-fuzz.zip -workdir=workdir

Benchmarks

Go 1.12 has been used for benchmarking.

Legend:

  • small - parse small.json (190 bytes).

  • medium - parse medium.json (2.3KB).

  • large - parse large.json (28KB).

  • canada - parse canada.json (2.2MB).

  • citm - parse citm_catalog.json (1.7MB).

  • twitter - parse twitter.json (617KB).

  • stdjson-map - parse into a map[string]interface{} using encoding/json.

  • stdjson-struct - parse into a struct containing a subset of fields of the parsed JSON, using encoding/json.

  • stdjson-empty-struct - parse into an empty struct using encoding/json. This is the fastest possible solution for encoding/json, may be used for json validation. See also benchmark results for json validation.

  • fastjson - parse using fastjson without fields access.

  • fastjson-get - parse using fastjson with fields access similar to stdjson-struct.

$ GOMAXPROCS=1 go test github.com/valyala/fastjson -bench='Parse$'
goos: linux
goarch: amd64
pkg: github.com/valyala/fastjson
BenchmarkParse/small/stdjson-map         	  200000	      7305 ns/op	  26.01 MB/s	     960 B/op	      51 allocs/op
BenchmarkParse/small/stdjson-struct      	  500000	      3431 ns/op	  55.37 MB/s	     224 B/op	       4 allocs/op
BenchmarkParse/small/stdjson-empty-struct         	  500000	      2273 ns/op	  83.58 MB/s	     168 B/op	       2 allocs/op
BenchmarkParse/small/fastjson                     	 5000000	       347 ns/op	 547.53 MB/s	       0 B/op	       0 allocs/op
BenchmarkParse/small/fastjson-get                 	 2000000	       620 ns/op	 306.39 MB/s	       0 B/op	       0 allocs/op
BenchmarkParse/medium/stdjson-map                 	   30000	     40672 ns/op	  57.26 MB/s	   10196 B/op	     208 allocs/op
BenchmarkParse/medium/stdjson-struct              	   30000	     47792 ns/op	  48.73 MB/s	    9174 B/op	     258 allocs/op
BenchmarkParse/medium/stdjson-empty-struct        	  100000	     22096 ns/op	 105.40 MB/s	     280 B/op	       5 allocs/op
BenchmarkParse/medium/fastjson                    	  500000	      3025 ns/op	 769.90 MB/s	       0 B/op	       0 allocs/op
BenchmarkParse/medium/fastjson-get                	  500000	      3211 ns/op	 725.20 MB/s	       0 B/op	       0 allocs/op
BenchmarkParse/large/stdjson-map                  	    2000	    614079 ns/op	  45.79 MB/s	  210734 B/op	    2785 allocs/op
BenchmarkParse/large/stdjson-struct               	    5000	    298554 ns/op	  94.18 MB/s	   15616 B/op	     353 allocs/op
BenchmarkParse/large/stdjson-empty-struct         	    5000	    268577 ns/op	 104.69 MB/s	     280 B/op	       5 allocs/op
BenchmarkParse/large/fastjson                     	   50000	     35210 ns/op	 798.56 MB/s	       5 B/op	       0 allocs/op
BenchmarkParse/large/fastjson-get                 	   50000	     35171 ns/op	 799.46 MB/s	       5 B/op	       0 allocs/op
BenchmarkParse/canada/stdjson-map                 	      20	  68147307 ns/op	  33.03 MB/s	12260502 B/op	  392539 allocs/op
BenchmarkParse/canada/stdjson-struct              	      20	  68044518 ns/op	  33.08 MB/s	12260123 B/op	  392534 allocs/op
BenchmarkParse/canada/stdjson-empty-struct        	     100	  17709250 ns/op	 127.11 MB/s	     280 B/op	       5 allocs/op
BenchmarkParse/canada/fastjson                    	     300	   4182404 ns/op	 538.22 MB/s	  254902 B/op	     381 allocs/op
BenchmarkParse/canada/fastjson-get                	     300	   4274744 ns/op	 526.60 MB/s	  254902 B/op	     381 allocs/op
BenchmarkParse/citm/stdjson-map                   	      50	  27772612 ns/op	  62.19 MB/s	 5214163 B/op	   95402 allocs/op
BenchmarkParse/citm/stdjson-struct                	     100	  14936191 ns/op	 115.64 MB/s	    1989 B/op	      75 allocs/op
BenchmarkParse/citm/stdjson-empty-struct          	     100	  14946034 ns/op	 115.56 MB/s	     280 B/op	       5 allocs/op
BenchmarkParse/citm/fastjson                      	    1000	   1879714 ns/op	 918.87 MB/s	   17628 B/op	      30 allocs/op
BenchmarkParse/citm/fastjson-get                  	    1000	   1881598 ns/op	 917.94 MB/s	   17628 B/op	      30 allocs/op
BenchmarkParse/twitter/stdjson-map                	     100	  11289146 ns/op	  55.94 MB/s	 2187878 B/op	   31266 allocs/op
BenchmarkParse/twitter/stdjson-struct             	     300	   5779442 ns/op	 109.27 MB/s	     408 B/op	       6 allocs/op
BenchmarkParse/twitter/stdjson-empty-struct       	     300	   5738504 ns/op	 110.05 MB/s	     408 B/op	       6 allocs/op
BenchmarkParse/twitter/fastjson                   	    2000	    774042 ns/op	 815.86 MB/s	    2541 B/op	       2 allocs/op
BenchmarkParse/twitter/fastjson-get               	    2000	    777833 ns/op	 811.89 MB/s	    2541 B/op	       2 allocs/op

Benchmark results for json validation:

$ GOMAXPROCS=1 go test github.com/valyala/fastjson -bench='Validate$'
goos: linux
goarch: amd64
pkg: github.com/valyala/fastjson
BenchmarkValidate/small/stdjson 	 2000000	       955 ns/op	 198.83 MB/s	      72 B/op	       2 allocs/op
BenchmarkValidate/small/fastjson         	 5000000	       384 ns/op	 493.60 MB/s	       0 B/op	       0 allocs/op
BenchmarkValidate/medium/stdjson         	  200000	     10799 ns/op	 215.66 MB/s	     184 B/op	       5 allocs/op
BenchmarkValidate/medium/fastjson        	  300000	      3809 ns/op	 611.30 MB/s	       0 B/op	       0 allocs/op
BenchmarkValidate/large/stdjson          	   10000	    133064 ns/op	 211.31 MB/s	     184 B/op	       5 allocs/op
BenchmarkValidate/large/fastjson         	   30000	     45268 ns/op	 621.14 MB/s	       0 B/op	       0 allocs/op
BenchmarkValidate/canada/stdjson         	     200	   8470904 ns/op	 265.74 MB/s	     184 B/op	       5 allocs/op
BenchmarkValidate/canada/fastjson        	     500	   2973377 ns/op	 757.07 MB/s	       0 B/op	       0 allocs/op
BenchmarkValidate/citm/stdjson           	     200	   7273172 ns/op	 237.48 MB/s	     184 B/op	       5 allocs/op
BenchmarkValidate/citm/fastjson          	    1000	   1684430 ns/op	1025.39 MB/s	       0 B/op	       0 allocs/op
BenchmarkValidate/twitter/stdjson        	     500	   2849439 ns/op	 221.63 MB/s	     312 B/op	       6 allocs/op
BenchmarkValidate/twitter/fastjson       	    2000	   1036796 ns/op	 609.10 MB/s	       0 B/op	       0 allocs/op

FAQ

  • Q: There are a ton of other high-perf packages for JSON parsing in Go. Why creating yet another package? A: Because other packages require either rigid JSON schema via struct magic and code generation or perform poorly when multiple unrelated fields must be obtained from the parsed JSON. Additionally, fastjson provides nicer API.

  • Q: What is the main purpose for fastjson? A: High-perf JSON parsing for RTB and other JSON-RPC services.

  • Q: Why fastjson doesn't provide fast marshaling (serialization)? A: Actually it provides some sort of marshaling - see Value.MarshalTo. But I'd recommend using quicktemplate for high-performance JSON marshaling :)

  • Q: fastjson crashes my program! A: There is high probability of improper use.

    • Make sure you don't hold references to objects recursively returned by Parser / Scanner beyond the next Parser.Parse / Scanner.Next call if such restriction is mentioned in docs.
    • Make sure you don't access fastjson objects from concurrently running goroutines if such restriction is mentioned in docs.
    • Build and run your program with -race flag. Make sure the race detector detects zero races.
    • If your program continue crashing after fixing issues mentioned above, file a bug.
Issues
  • Add some handy/fallible methods to the Object public API

    Add some handy/fallible methods to the Object public API

    • TryVisit: fallible version of Visit
    • TryGet: fallible version of Get
    • Take and TryTake: handy methods to simultaneously Get/TryGet and Del a Value from an object
    opened by midnightexigent 0
  • How to construct nested JSON using for loops and arenapools?

    How to construct nested JSON using for loops and arenapools?

    Hi, I am currently having to construct a JSON file containing three things. Instances, versions and function names.

    Each instance has a version, each version has multiple function names.

    Therefore, each 'version' is an array. Each 'instance' has multiple versions.

    In pseudocode, I would append each 'function name' to an array (which will be 'version'), and then I would set a key within 'instance' to the 'version', which holds this array of 'function name'.

    The problem is that I can't for the life of me figure out how to use the ArenaPool approach to make this safe. It would appear I am always ending up with the 'version' array containing the entries of the last for loop that has been iterated through.

    See the following code:

    
    	instances := []Stat{}
    	versions := []Stat{}
    	functions := []Stat{}
    	var ap fastjson.ArenaPool
    	are := ap.Get()
    	verJSON := are.NewObject()
    	instJSON := are.NewObject()
    
    	err := sts.db.Select(&instances, "select distinct on (instance) instance from stats")
    	if err != nil {
    		[handle error]
    		return nil, err
    	}
    
    
    	for _, inst := range instances {
    		err := sts.db.Select(&versions, "select distinct on (version) version from stats WHERE instance = $1", inst.Instance)
    		if err != nil {
    			[handle error]
    			return nil, err
    		}
    		for _, ver := range versions {
    			err := sts.db.Select(&functions, "select distinct on (func_name) func_name from stats where instance = $1 and version = $2", inst.Instance, ver.Version)
    			if err != nil {
    				[handle error]
    				return nil, err
    			}
    			newAr := ap.Get()
    			defer ap.Put(newAr)
    			arr := newAr.NewArray()
    			for i, fun := range functions {
    				arr.SetArrayItem(i, newAr.NewString(fun.FuncName))
    			}
    			verJSON.Set(ver.Version, arr)
    			functions = nil
    			instJSON.Set(inst.Instance, verJSON)
    		}
    		versions = nil
    	}
    
    	fmt.Println("This is instJSON: ", instJSON)
    

    'instJSON' always ends up containing the 'verJSON' from the second to last and last loop of 'range instances'.

    opened by clarencefoy 0
  • Add ValidateParse, ValidateParser and ValidateScanner

    Add ValidateParse, ValidateParser and ValidateScanner

    Presently, users that want to validate their json before parsing need to run Validate and Parse separately.
    This is inefficient because each json string must be scanned twice.

    This PR introduces a ValidateParser which validates as it parses.

    It is slightly less efficient than the normal parser, but much more efficient than running Validate and Parse separately.

    BenchmarkParse/small/fastjson-12                        10067671      111.0 ns/op  1712.14 MB/s    0 B/op  0 allocs/op
    BenchmarkParse/small/fastjson-validate-parser-12        10503465      109.8 ns/op  1729.86 MB/s    0 B/op  0 allocs/op
    BenchmarkParse/small/fastjson-validate-and-parse-12      6553486      176.5 ns/op  1076.23 MB/s    0 B/op  0 allocs/op
    
    BenchmarkParse/medium/fastjson-12                        1797662      639.2 ns/op  3643.68 MB/s    0 B/op  0 allocs/op
    BenchmarkParse/medium/fastjson-validate-parser-12        1635453      795.7 ns/op  2927.14 MB/s    0 B/op  0 allocs/op
    BenchmarkParse/medium/fastjson-validate-and-parse-12      945916     1400 ns/op    1663.97 MB/s    0 B/op  0 allocs/op
    
    BenchmarkParse/large/fastjson-12                          121093     9001 ns/op    3123.79 MB/s    0 B/op  0 allocs/op
    BenchmarkParse/large/fastjson-validate-parser-12          116598    10946 ns/op    2568.69 MB/s    0 B/op  0 allocs/op
    BenchmarkParse/large/fastjson-validate-and-parse-12        57669    20630 ns/op    1362.95 MB/s    0 B/op  0 allocs/op
    
    BenchmarkParse/canada/fastjson-12                           1072  1205458 ns/op    1867.39 MB/s    1 B/op  0 allocs/op
    BenchmarkParse/canada/fastjson-validate-parser-12           1022  1151216 ns/op    1955.38 MB/s    1 B/op  0 allocs/op
    BenchmarkParse/canada/fastjson-validate-and-parse-12         678  1781026 ns/op    1263.91 MB/s    2 B/op  0 allocs/op
    
    BenchmarkParse/citm/fastjson-12                             2584   476059 ns/op    3628.13 MB/s    0 B/op  0 allocs/op
    BenchmarkParse/citm/fastjson-validate-parser-12             2528   481104 ns/op    3590.09 MB/s    0 B/op  0 allocs/op
    BenchmarkParse/citm/fastjson-validate-and-parse-12          1443   844153 ns/op    2046.08 MB/s    1 B/op  0 allocs/op
    
    BenchmarkParse/twitter/fastjson-12                          6711   164653 ns/op    3835.42 MB/s  756 B/op  0 allocs/op
    BenchmarkParse/twitter/fastjson-validate-parser-12          5918   192253 ns/op    3284.80 MB/s    0 B/op  0 allocs/op
    BenchmarkParse/twitter/fastjson-validate-and-parse-12       3758   317174 ns/op    1991.06 MB/s    0 B/op  0 allocs/op
    

    Adds 7 new symbols to the API:

    type ValidateScanner
    type ValidateParser 
    type ValidateParserPool
    func ValidateParse(s string) (*Value, error)
    func ValidateParseBytes(b []byte) (*Value, error)
    func MustValidateParse(s string) *Value
    func MustValidateParseBytes(b []byte) *Value
    

    See #58

    opened by kevburnsjr 1
  • compatibility with strconv.ParseInt

    compatibility with strconv.ParseInt

    The strconv.ParseInt in Golang std lib

    • has syntax check and return an error if s is empty or contains invalid digits.
    • has range check and return the the closest number among the range if overflow.
    // The errors that ParseInt returns have concrete type *NumError
    // and include err.Num = s. If s is empty or contains invalid
    // digits, err.Err = ErrSyntax and the returned value is 0;
    // if the value corresponding to s cannot be represented by a
    // signed integer of the given size, err.Err = ErrRange and the
    // returned value is the maximum magnitude integer of the
    // appropriate bitSize and sign.
    func ParseInt(s string, base int, bitSize int) (i int64, err error) {
    

    I need these but fastjson.ParseInt64BestEffort behaves differently.

    opened by yuzhichang 0
Owner
Aliaksandr Valialkin
Working on @VictoriaMetrics
Aliaksandr Valialkin
Get JSON values quickly - JSON parser for Go

get json values quickly GJSON is a Go package that provides a fast and simple way to get values from a json document. It has features such as one line

Josh Baker 10.3k May 19, 2022
A fast json parser for go

rjson rjson is a json parser that relies on Ragel-generated state machines for most parsing. rjson's api is minimal and focussed on efficient parsing.

WillAbides 49 Feb 23, 2022
JSON Spanner - A Go package that provides a fast and simple way to filter or transform a json document

JSON SPANNER JSON Spanner is a Go package that provides a fast and simple way to

null 2 May 20, 2022
Fast JSON encoder/decoder compatible with encoding/json for Go

Fast JSON encoder/decoder compatible with encoding/json for Go

Masaaki Goshima 1.5k May 14, 2022
Slow and unreliable JSON parser generator (in progress)

VivaceJSON Fast and reliable JSON parser generator Todo List parse fields parse types generate struct generate (keypath+key) to struct Value Mapping F

null 7 Mar 3, 2022
A JSON stream parser for Go

pjson A JSON stream parser for Go Example The example below prints all string values from a JSON document. package

Josh Baker 59 Dec 2, 2021
One of the fastest alternative JSON parser for Go that does not require schema

Alternative JSON parser for Go (10x times faster standard library) It does not require you to know the structure of the payload (eg. create structs),

Leonid Bugaev 4.5k May 19, 2022
JSON:API compatible query string parser

QParser The package helps to parse part of the URL path and its parameters string to a handy structure. The structure format is compatible with the JS

Velmie 1 Dec 21, 2021
A JSON parser/generator for Go

FASTJSON fastjson是java版本的fastjson库的api做一个翻译,方便习惯java的人操作json数据 主要适用场景:层级和字段都不能确定的json 这个库并不实现高性能json解析,依赖标准库json 这个库并没有100%实现java版的api 安装 go get -u gi

wuyunhua 2 Dec 29, 2021
Package json implements encoding and decoding of JSON as defined in RFC 7159

Package json implements encoding and decoding of JSON as defined in RFC 7159. The mapping between JSON and Go values is described in the documentation for the Marshal and Unmarshal functions

High Performance, Kubernetes Native Object Storage 3 May 10, 2022
Json-go - CLI to convert JSON to go and vice versa

Json To Go Struct CLI Install Go version 1.17 go install github.com/samit22/js

Samit Ghimire 5 Mar 3, 2022
Fast and flexible JSON encoder for Go

Jettison Jettison is a fast and flexible JSON encoder for the Go programming language, inspired by bet365/jingo, with a richer features set, aiming at

William Poussier 126 May 13, 2022
Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang

kazaam Description Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang. This functionality provides

Qntfy 205 Sep 17, 2021
JSON diff library for Go based on RFC6902 (JSON Patch)

jsondiff jsondiff is a Go package for computing the diff between two JSON documents as a series of RFC6902 (JSON Patch) operations, which is particula

William Poussier 166 May 7, 2022
Fast JSON serializer for golang.

easyjson Package easyjson provides a fast and easy way to marshal/unmarshal Go structs to/from JSON without the use of reflection. In performance test

Free and open source software developed at Mail.Ru 3.7k May 22, 2022
Fast Color JSON Marshaller + Pretty Printer for Golang

ColorJSON: The Fast Color JSON Marshaller for Go What is this? This package is based heavily on hokaccha/go-prettyjson but has some noticible differen

Tyler Brock 107 May 17, 2022
A blazingly fast JSON serializing & deserializing library

Sonic A blazingly fast JSON serializing & deserializing library, accelerated by JIT(just-in-time compiling) and SIMD(single-instruction-multi-data). B

Bytedance Inc. 2.8k May 12, 2022
Fast json for go. Lightweight fork of jsoniter.

jx Fast json for go. Lightweight fork of jsoniter. Features Reduced scope (no reflection or encoding/json adapter) Fuzzing, improved test coverage Dra

ogen 53 May 8, 2022
Simple Email Parser

mp - mail parser mp is a simple cli email parser. It currently takes stdin and outputs JSON. Example: cat fixtures/test.eml | mp { "Text": "Hello w

Mark 45 Apr 8, 2022