A blazingly fast JSON serializing & deserializing library

Related tags

JSON sonic
Overview

Sonic

A blazingly fast JSON serializing & deserializing library, accelerated by JIT(just-in-time compiling) and SIMD(single-instruction-multi-data).

Benchmarks

For all sizes of json and all scenes of usage, Sonic performs almost best.

  • Small (400B, 11 keys, 3 levels) small benchmarks
  • Medium (110KB, 300+ keys, 3 levels, with many quoted-json values) medium benchmarks
  • Large (550KB, 10000+ key, 6 levels) large benchmarks

For a 13KB TwitterJson(cpu i9-9880H, goarch amd64), Sonic is 1.5x faster than json-iterator in decoding, 2.5x faster in encoding.

BenchmarkDecoder_Generic_Sonic-16                          10000             54309 ns/op         240.01 MB/s       46149 B/op        303 allocs/op
BenchmarkDecoder_Generic_StdLib-16                         10000            135268 ns/op          96.36 MB/s       50899 B/op        772 allocs/op
BenchmarkDecoder_Generic_JsonIter-16                       10000             96701 ns/op         134.80 MB/s       55791 B/op       1068 allocs/op
BenchmarkDecoder_Binding_Sonic-16                          10000             29478 ns/op         442.20 MB/s       26062 B/op         34 allocs/op
BenchmarkDecoder_Binding_StdLib-16                         10000            119348 ns/op         109.22 MB/s       10560 B/op        207 allocs/op
BenchmarkDecoder_Binding_JsonIter-16                       10000             37646 ns/op         346.25 MB/s       14673 B/op        385 allocs/op
BenchmarkEncoder_Generic_Sonic-16                          10000             25894 ns/op         503.39 MB/s       19096 B/op         42 allocs/op
BenchmarkEncoder_Generic_JsonIter-16                       10000             50275 ns/op         259.27 MB/s       13432 B/op         77 allocs/op
BenchmarkEncoder_Generic_StdLib-16                         10000            154901 ns/op          84.15 MB/s       48173 B/op        827 allocs/op
BenchmarkEncoder_Binding_Sonic-16                          10000              7373 ns/op        1768.04 MB/s       13861 B/op          4 allocs/op
BenchmarkEncoder_Binding_JsonIter-16                       10000             23223 ns/op         561.31 MB/s        9489 B/op          2 allocs/op
BenchmarkEncoder_Binding_StdLib-16                         10000             19512 ns/op         668.07 MB/s        9477 B/op          1 allocs/op

More detail see ast/search_test.go, decoder/decoder_test.go, encoder/encoder_test.go,

Usage

Marshal/Unmarshal

The behaviors are mostly consistent with encoding/json, except some uncommon escaping and key sorting (see issue4)

import "github.com/bytedance/sonic"

// Marshal
output, err := sonic.Marshal(&data) 
// Unmarshal
err := sonic.Unmarshal(input, &data) 

Get

Search partial json by given pathes, which must be non-negative integer or string or nil

import "github.com/bytedance/sonic"

input := []byte(`{"key1":[{},{"key2":{"key3":[1,2,3]}}]}`)

// no path, returns entire json
root, err := sonic.Get(input)
raw := root.Raw() // == string(input)

// multiple pathes
root, err := sonic.Get(input, "key1", 1, "key2")
sub := root.Get("key3").Index(2).Int64() // == 3

Returned ast.Node supports:

  • secondary search: Get(), Index(), GetByPath()
  • type assignment: Int64(), Float64(), String(), Number(), Bool(), Map(), Array()
  • children traversal: Values(), Properties()
  • supplement: Set(), SetByIndex(), Add(), Cap(), Len()

Use Number/Use Int64

import "github.com/bytedance/sonic/decoder"

input := `1`
var data interface{}

// default float64
dc := decoder.NewDecoder(input) 
dc.Decode(&data) // data == float64(1)
// use json.Number
dc = decoder.NewDecoder(input)
dc.UseNumber()
dc.Decode(&data) // data == json.Number("1")
// use int64
dc = decoder.NewDecoder(input)
dc.UseInt64()
dc.Decode(&data) // data == int64(1)

root, err := sonic.GetFromString(input)
// Get json.Number
jn := root.Number()
jm := root.InterfaceUseNumber().(json.Number) // jn == jm
// Get float64
fn := root.Float64()
fm := root.Interface().(float64) // jn == jm

Tips

Pretouch

Since Sonic uses JIT(just-in-time) compiling for decoder/encoder, huge schema may cause request-timeout. For better stability, we suggest to use Pretouch() for more-than-10000-field schema(struct) before Marshal()/Unmarshal().

import (
    "reflect"
    "github.com/bytedance/sonic"
)

func init() {
    var v HugeStruct
    err := sonic.Pretouch(reflect.TypeOf(v))
}

Pass string or []byte?

For alignment to encoding/json, we provide API to pass []byte as arguement, but the string-to-bytes copy is conducted at the same time considering safety, which may lose performance when origin json is huge. Therefore, you can use UnmarshalString, GetFromString to pass string, as long as your origin data is string or nocopy-cast is safe for your []byte.

Avoid repeating work

Get() overlapping pathes from the same root may cause repeating parsing. Instead of using Get() several times, you can use parser and searcher together like this:

import "github.com/bytedance/sonic"

root, err := sonic.GetByString(_TwitterJson, "statuses", 3, "user")
a = root.GetByPath( "entities","description")
b = root.GetByPath( "entities","url")
c = root.GetByPath( "created_at")

No need to worry about the overlaping or overparsing of a, b and c, because the inner parser of their root is lazy-loaded.

Better performance for generic deserializing

In most cases of fully-load generic json, Unmarshal() performs better than ast.Loads(). But if you only want to search a partial json and convert it into interface{} (or map[string]interface{}, []interface{}), we advise you to combine Get() and Unmarshal():

import "github.com/bytedance/sonic"

node, err := sonic.GetByString(_TwitterJson, "statuses", 3, "user")
var user interface{}
err = sonic.UnmarshalString(node.Raw(), &user)
Issues
  • Lefting unnecessary `json.Unmarshaler`/`json.Marshaler` implementation on main struct may introduce significant performance decline (ex: using easyjson code-gen)

    Lefting unnecessary `json.Unmarshaler`/`json.Marshaler` implementation on main struct may introduce significant performance decline (ex: using easyjson code-gen)

    `goos: darwin
    goarch: amd64
    pkg: gateway/test
    cpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
    BenchmarkTestUnJsoniter-4         100000              2211 ns/op            4672 B/op         10 allocs/op
    BenchmarkTestUnJsoniter-4         100000              2106 ns/op            4672 B/op         10 allocs/op
    BenchmarkTestUnJsoniter-4         100000              1514 ns/op            4672 B/op         10 allocs/op
    BenchmarkTestUnJsoniter-4         100000              1447 ns/op            4672 B/op         10 allocs/op
    BenchmarkTestUnSonicJson-4        100000             31926 ns/op           14922 B/op        238 allocs/op
    BenchmarkTestUnSonicJson-4        100000             30677 ns/op           14907 B/op        238 allocs/op
    BenchmarkTestUnSonicJson-4        100000             30704 ns/op           14922 B/op        238 allocs/op
    BenchmarkTestUnSonicJson-4        100000             31479 ns/op           14905 B/op        238 allocs/op
    BenchmarkTestJsoniter-4           100000             20541 ns/op            7176 B/op         38 allocs/op
    BenchmarkTestJsoniter-4           100000             21243 ns/op            7176 B/op         38 allocs/op
    BenchmarkTestJsoniter-4           100000             20656 ns/op            7176 B/op         38 allocs/op
    BenchmarkTestJsoniter-4           100000             21710 ns/op            7176 B/op         38 allocs/op
    BenchmarkTestSonic-4              100000             26187 ns/op            7314 B/op         41 allocs/op
    BenchmarkTestSonic-4              100000             26092 ns/op            7314 B/op         41 allocs/op
    BenchmarkTestSonic-4              100000             25784 ns/op            7298 B/op         41 allocs/op
    BenchmarkTestSonic-4              100000             26414 ns/op            7377 B/op         41 allocs/op
    
    invalid wrong practice 
    opened by lxj15398019970 17
  • Would you please add goccy/go-json as a benchmark target ?

    Would you please add goccy/go-json as a benchmark target ?

    I develop goccy/go-json. Since this library has same concept of yours about compatibility with encoding/json and high performance, I would be grateful if you could add it to the benchmark target.

    opened by goccy 16
  • 反序列化结构中的字符串是不安全的

    反序列化结构中的字符串是不安全的

    import (
    	"bytes"
    	"encoding/json"
    	"testing"
    
    	"github.com/bytedance/sonic"
    	jsoniter "github.com/json-iterator/go"
    	"github.com/stretchr/testify/require"
    )
    
    func TestSonicUnmarshal(t *testing.T) {
    	type test struct {
    		Str1 string
    		Str2 string
    	}
    	buf := bytes.NewBuffer(make([]byte, 0, 4096))
    	data, _ := sonic.Marshal(test{"1234567", "7654321"})
    	buf.Write(data)
    	var s test
    	sonic.Unmarshal(buf.Bytes(), &s)
    	require.Equal(t, "1234567", s.Str1)
    	require.Equal(t, "7654321", s.Str2)
    	buf.Reset()
    	data, _ = sonic.Marshal(test{"7654321", "1234567"})
    	buf.Write(data)
    	require.Equal(t, "1234567", s.Str1)
    	require.Equal(t, "7654321", s.Str2)
    }
    
    func TestStdJsonUnmarshal(t *testing.T) {
    	type test struct {
    		Str1 string
    		Str2 string
    	}
    	buf := bytes.NewBuffer(make([]byte, 0, 4096))
    	data, _ := json.Marshal(test{"1234567", "7654321"})
    	buf.Write(data)
    	var s test
    	json.Unmarshal(buf.Bytes(), &s)
    	require.Equal(t, "1234567", s.Str1)
    	require.Equal(t, "7654321", s.Str2)
    	buf.Reset()
    	data, _ = json.Marshal(test{"7654321", "1234567"})
    	buf.Write(data)
    	require.Equal(t, "1234567", s.Str1)
    	require.Equal(t, "7654321", s.Str2)
    }
    
    func TestJsoniterUnmarshal(t *testing.T) {
    	type test struct {
    		Str1 string
    		Str2 string
    	}
    	json := jsoniter.ConfigFastest
    	buf := bytes.NewBuffer(make([]byte, 0, 4096))
    	data, _ := json.Marshal(test{"1234567", "7654321"})
    	buf.Write(data)
    	var s test
    	json.Unmarshal(buf.Bytes(), &s)
    	require.Equal(t, "1234567", s.Str1)
    	require.Equal(t, "7654321", s.Str2)
    	buf.Reset()
    	data, _ = json.Marshal(test{"7654321", "1234567"})
    	buf.Write(data)
    	require.Equal(t, "1234567", s.Str1)
    	require.Equal(t, "7654321", s.Str2)
    }
    

    如果把 sonic 换成 json 或者 jsoniter,都能测试通过,而 sonic 则会报错,因为它反序列化结果中的 string 字段会直接引用输入参数中的值,如果在反序列化之后,输入参数被修改,则会导致反序列化结果错乱。

    known-issue 
    opened by andot 10
  • Not 100% compatibility with encoding/json

    Not 100% compatibility with encoding/json

    func TestCompatibility(t *testing.T) {
    	inputs := []interface{}{`"<&>"`, `"\"<&>\""`, "\b", float64(-0), float32(-0), map[string]int{"3": 3, "2": 2, "1": 1}}
    	for _, input := range inputs {
    		t.Run(fmt.Sprintf("case %v", input), func(t *testing.T) {
    			buf1, err1 := json.Marshal(input)
    			buf2, err2 := Marshal(input)
    			require.Nil(t, err1)
    			require.Nil(t, err2)
    			require.Equal(t, string(buf1), string(buf2))
    		})
    	}
    }
    
    --- FAIL: TestCompatibility (0.00s)
        --- FAIL: TestCompatibility/case_"<&>" (0.00s)
            fuzz_test.go:112: 
                    Error Trace:    fuzz_test.go:112
                    Error:          Not equal: 
                                    expected: "\"\\\"\\u003c\\u0026\\u003e\\\"\""
                                    actual  : "\"\\\"<&>\\\"\""
                                
                                    Diff:
                                    --- Expected
                                    +++ Actual
                                    @@ -1 +1 @@
                                    -"\"\u003c\u0026\u003e\""
                                    +"\"<&>\""
                    Test:           TestCompatibility/case_"<&>"
        --- FAIL: TestCompatibility/case_"\"<&>\"" (0.00s)
            fuzz_test.go:112: 
                    Error Trace:    fuzz_test.go:112
                    Error:          Not equal: 
                                    expected: "\"\\\"\\\\\\\"\\u003c\\u0026\\u003e\\\\\\\"\\\"\""
                                    actual  : "\"\\\"\\\\\\\"<&>\\\\\\\"\\\"\""
                                
                                    Diff:
                                    --- Expected
                                    +++ Actual
                                    @@ -1 +1 @@
                                    -"\"\\\"\u003c\u0026\u003e\\\"\""
                                    +"\"\\\"<&>\\\"\""
                    Test:           TestCompatibility/case_"\"<&>\""
        --- FAIL: TestCompatibility/case_\b (0.00s)
            fuzz_test.go:112: 
                    Error Trace:    fuzz_test.go:112
                    Error:          Not equal: 
                                    expected: "\"\\u0008\""
                                    actual  : "\"\\b\""
                                
                                    Diff:
                                    --- Expected
                                    +++ Actual
                                    @@ -1 +1 @@
                                    -"\u0008"
                                    +"\b"
                    Test:           TestCompatibility/case_\b
        --- FAIL: TestCompatibility/case_map[1:1_2:2_3:3] (0.00s)
            fuzz_test.go:112: 
                    Error Trace:    fuzz_test.go:112
                    Error:          Not equal: 
                                    expected: "{\"1\":1,\"2\":2,\"3\":3}"
                                    actual  : "{\"2\":2,\"1\":1,\"3\":3}"
                                
                                    Diff:
                                    --- Expected
                                    +++ Actual
                                    @@ -1 +1 @@
                                    -{"1":1,"2":2,"3":3}
                                    +{"2":2,"1":1,"3":3}
                    Test:           TestCompatibility/case_map[1:1_2:2_3:3]
    FAIL
    exit status 1
    FAIL    github.com/bytedance/sonic      0.144s
    
    question 
    opened by wangkechun 10
  • bug: go test panic in /encoder and /decoder

    bug: go test panic in /encoder and /decoder

    overview

    The unit test panicked in /encoder and /decoder, saying :

    # github.com/goccy/go-json/internal/encoder/vm 
    fatal error: runtime: out of memory
    

    go version

    go version go1.16 linux/amd64

    go env

    GO111MODULE=""
    GOARCH="amd64"
    GOBIN="/home/liwm29/gopath/bin"
    GOCACHE="/home/liwm29/.cache/go-build"
    GOENV="/home/liwm29/.config/go/env"
    GOEXE=""
    GOFLAGS=""
    GOHOSTARCH="amd64"
    GOHOSTOS="linux"
    GOINSECURE=""
    GOMODCACHE="/home/liwm29/gopath/pkg/mod"
    GONOPROXY=""
    GONOSUMDB=""
    GOOS="linux"
    GOPATH="/home/liwm29/gopath"
    GOPRIVATE=""
    GOPROXY="https://goproxy.cn,direct"
    GOROOT="/usr/local/go"
    GOSUMDB="sum.golang.org"
    GOTMPDIR=""
    GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
    GOVCS=""
    GOVERSION="go1.16"
    GCCGO="gccgo"
    AR="ar"
    CC="gcc"
    CXX="g++"
    CGO_ENABLED="1"
    GOMOD="/home/liwm29/sonic_raw/sonic/go.mod"
    CGO_CFLAGS="-g -O2"
    CGO_CPPFLAGS=""
    CGO_CXXFLAGS="-g -O2"
    CGO_FFLAGS="-g -O2"
    CGO_LDFLAGS="-g -O2"
    PKG_CONFIG="pkg-config"
    GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1917513849=/tmp/go-build -gno-record-gcc-switches"
    

    what happened

    [[email protected] sonic]$ go test ./...
    ok      github.com/bytedance/sonic      4.103s
    ok      github.com/bytedance/sonic/ast  0.006s
    # github.com/goccy/go-json/internal/encoder/vm
    fatal error: runtime: out of memory
    
    runtime stack:
    runtime.throw(0xcac67f, 0x16)
            /usr/local/go/src/runtime/panic.go:1117 +0x72
    runtime.sysMap(0xc030000000, 0xc000000, 0x11e7b90)
            /usr/local/go/src/runtime/mem_linux.go:169 +0xc6
    runtime.(*mheap).sysAlloc(0x11c8bc0, 0xa000000, 0x42bcf7, 0x11c8bc8)
            /usr/local/go/src/runtime/malloc.go:729 +0x1e5
    runtime.(*mheap).grow(0x11c8bc0, 0x4ef7, 0x0)
            /usr/local/go/src/runtime/mheap.go:1346 +0x85
    runtime.(*mheap).allocSpan(0x11c8bc0, 0x4ef7, 0x440100, 0x7f1fa3d21688)
            /usr/local/go/src/runtime/mheap.go:1173 +0x609
    runtime.(*mheap).alloc.func1()
            /usr/local/go/src/runtime/mheap.go:910 +0x59
    runtime.systemstack(0x46d334)
            /usr/local/go/src/runtime/asm_amd64.s:379 +0x66
    runtime.mstart()
            /usr/local/go/src/runtime/proc.go:1246
    
    goroutine 1 [running]:
    runtime.systemstack_switch()
            /usr/local/go/src/runtime/asm_amd64.s:339 fp=0xc00286cac0 sp=0xc00286cab8 pc=0x46d460
    runtime.(*mheap).alloc(0x11c8bc0, 0x4ef7, 0xc002860001, 0xc00458c000)
            /usr/local/go/src/runtime/mheap.go:904 +0x85 fp=0xc00286cb10 sp=0xc00286cac0 pc=0x4279a5
    runtime.(*mcache).allocLarge(0x7f1fcac23108, 0x9dee000, 0xc002be0100, 0x7f1fa3d21688)
            /usr/local/go/src/runtime/mcache.go:224 +0x97 fp=0xc00286cb68 sp=0xc00286cb10 pc=0x418017
    runtime.mallocgc(0x9dee000, 0x0, 0x0, 0xc002be2000)
            /usr/local/go/src/runtime/malloc.go:1078 +0x925 fp=0xc00286cbf0 sp=0xc00286cb68 pc=0x40de65
    runtime.growslice(0xc686a0, 0xc025b74000, 0x7e5800, 0x7e5800, 0x7e5801, 0x1122970, 0x1, 0xc0)
            /usr/local/go/src/runtime/slice.go:224 +0x154 fp=0xc00286cc58 sp=0xc00286cbf0 pc=0x44fcd4
    cmd/compile/internal/ssa.(*debugState).appendLiveSlot(...)
            /usr/local/go/src/cmd/compile/internal/ssa/debug.go:290
    cmd/compile/internal/ssa.(*debugState).liveness(0xc001ccd660, 0xc0014b33b0, 0xc0032bb030, 0xc0047a58d8)
            /usr/local/go/src/cmd/compile/internal/ssa/debug.go:512 +0x7e5 fp=0xc00286ce58 sp=0xc00286cc58 pc=0x6705a5
    cmd/compile/internal/ssa.BuildFuncDebug(0xc0000fa000, 0xc0018c51e0, 0xc0041be300, 0xcc81f8, 0x6a9)
            /usr/local/go/src/cmd/compile/internal/ssa/debug.go:436 +0xa65 fp=0xc00286d0d0 sp=0xc00286ce58 pc=0x66f345
    cmd/compile/internal/gc.genssa(0xc0018c51e0, 0xc004c62000)
            /usr/local/go/src/cmd/compile/internal/gc/ssa.go:6527 +0x28ac fp=0xc00286d5b0 sp=0xc00286d0d0 pc=0xb437ec
    cmd/compile/internal/gc.compileSSA(0xc0000e5e40, 0x0)
            /usr/local/go/src/cmd/compile/internal/gc/pgen.go:329 +0x3a5 fp=0xc00286d690 sp=0xc00286d5b0 pc=0xad7685
    cmd/compile/internal/gc.compile(0xc0000e5e40)
            /usr/local/go/src/cmd/compile/internal/gc/pgen.go:277 +0x39e fp=0xc00286d708 sp=0xc00286d690 pc=0xad711e
    cmd/compile/internal/gc.funccompile(0xc0000e5e40)
            /usr/local/go/src/cmd/compile/internal/gc/pgen.go:220 +0xc5 fp=0xc00286d760 sp=0xc00286d708 pc=0xad6c65
    cmd/compile/internal/gc.Main(0xcc7d20)
            /usr/local/go/src/cmd/compile/internal/gc/main.go:762 +0x3525 fp=0xc00286df10 sp=0xc00286d760 pc=0xaabbe5
    main.main()
            /usr/local/go/src/cmd/compile/main.go:52 +0xb1 fp=0xc00286df88 sp=0xc00286df10 pc=0xbf90d1
    runtime.main()
            /usr/local/go/src/runtime/proc.go:225 +0x256 fp=0xc00286dfe0 sp=0xc00286df88 pc=0x439d76
    runtime.goexit()
            /usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00286dfe8 sp=0xc00286dfe0 pc=0x46f161
    FAIL    github.com/bytedance/sonic/decoder [build failed]
    FAIL    github.com/bytedance/sonic/encoder [build failed]
    ok      github.com/bytedance/sonic/internal/caching     0.002s
    ?       github.com/bytedance/sonic/internal/cpu [no test files]
    ok      github.com/bytedance/sonic/internal/jit 0.004s
    ok      github.com/bytedance/sonic/internal/loader      0.003s
    ?       github.com/bytedance/sonic/internal/native      [no test files]
    ok      github.com/bytedance/sonic/internal/native/avx  0.006s
    ok      github.com/bytedance/sonic/internal/native/avx2 0.006s
    ?       github.com/bytedance/sonic/internal/native/types        [no test files]
    ok      github.com/bytedance/sonic/internal/resolver    0.002s
    ?       github.com/bytedance/sonic/internal/rt  [no test files]
    ?       github.com/bytedance/sonic/unquote      [no test files]
    FAIL
    

    other

    i have changed nothing, except cloning and cd /sonic and run go test ./...

    opened by wymli 7
  • sonic.Unmarshal decode json string failed when field's type is

    sonic.Unmarshal decode json string failed when field's type is "stringint"

    test code below:

    type P struct {
    	Cip        string      `json:"cip"`
    	Oip        string      `json:"oip"`
    	Oid        string      `json:"oid"`
    	Aid        json.Number `json:"aid"`
    	Pcid       string      `json:"pcid"`
            Num        int `json:"num"`
    }
    
    func main() {
    	var pstruct P
    	jsonStr := `{"num":123, "cip":"127.0.0.1","oip":"10.206.85.204",oid":"52852","aid":"6079759","pcid":"75","cid":"76"}`
    	//json.Unmarshal([]byte(jsonStr), &pstruct)
    	sonic.Unmarshal([]byte(jsonStr), &pstruct)
    	fmt.Println(pstruct.Cip) // output: "127.0.0.1"
          fmt.Println(pstruct.Num) // output:123
          fmt.Println(pstruct.Aid) // output: ""
          fmt.Println(pstruct.Oid) // output: ""
    }
    

    if this is normal, how make this print correctly? PS: sonic v1.1.1

    i found out the problem rule: when the type of struct defined is not match with the json, the field and the fields behind it all empty. eg :

    type Test struct {
         Num1 string `json:"num1"`
         Num2 string `json:"num2"`
         Num3 string `json:"num3"`
         Num4 string `json:"num4"`
         Num5 string `json:"num5"`
     }
    testStr := `{"num1":"123","num2":"this is string2", "num3":456, "num4":789, "num5":"this is string"}`
    var t Test
    sonic.Unmarshal([]byte(testStr), &t)
    fmt.Printf("%#v", t)
    //output:  {Num1:"123", Num2:"this is string2", Num3:"", Num4:"", Num5:""}
    
    opened by gy0624ww 6
  • ast.Node Get a key that doesn't exist

    ast.Node Get a key that doesn't exist

    http: panic serving 127.0.0.1:44846: runtime error: invalid memory address or nil pointer dereference goroutine 2315 [running]: net/http.(*conn).serve.func1() /usr/local/go/src/net/http/server.go:1802 +0xb9 panic({0xd8a3c0, 0x171dd60}) /usr/local/go/src/runtime/panic.go:1047 +0x266 github.com/bytedance/sonic/ast.(*Node).checkRaw(0xc02713a1b0) /home/zhounan1/gopath/src/github.com/bytedance/sonic/ast/node.go:159 +0x14 github.com/bytedance/sonic/ast.(*Node).String(0x0) /home/zhounan1/gopath/src/github.com/bytedance/sonic/ast/node.go:245 +0x25

    my program just like this:

    list.Get("res").Index(key).Get("item").Get("json_data").Index(cardKey).String()

    When using ast.Node to get a non-existing key, the program will panic. Is there a way to get the null value of the specified type like the get method of simplejson?

    opened by meng-y 5
  • 生产环境报错,程序重启,求救。

    生产环境报错,程序重启,求救。

    生产环境偶尔报这个错误,recover不到这个异常,导致程序重启。

    traceback: unexpected SPWRITE function github.com/bytedance/sonic/internal/native/avx2.__native_entry__
    fatal error: traceback
    
    runtime stack:
    runtime.throw({0x12ef91c, 0x1de4260})
    	/usr/local/go/src/runtime/panic.go:1198 +0x71
    runtime.gentraceback(0x7fc3972db850, 0x1580, 0xc01949fba0, 0xc0eb5aa4e0, 0x0, 0x0, 0x7fffffff, 0x7fc385ff7460, 0x400000002, 0x0)
    	/usr/local/go/src/runtime/traceback.go:238 +0x19c5
    runtime.addOneOpenDeferFrame.func1()
    	/usr/local/go/src/runtime/panic.go:751 +0x6b
    runtime.systemstack()
    	/usr/local/go/src/runtime/asm_amd64.s:383 +0x49
    
    goroutine 6113840 [running]:
    runtime.systemstack_switch()
    	/usr/local/go/src/runtime/asm_amd64.s:350 fp=0xc000367488 sp=0xc000367480 pc=0x4d08a0
    runtime.addOneOpenDeferFrame(0xc8, 0x1fffffff, 0x47012d)
    	/usr/local/go/src/runtime/panic.go:750 +0x77 fp=0xc0003674c8 sp=0xc000367488 pc=0x49e537
    panic({0x11df0a0, 0x1d84b40})
    	/usr/local/go/src/runtime/panic.go:998 +0x114 fp=0xc000367588 sp=0xc0003674c8 pc=0x49f354
    runtime.panicmem(...)
    	/usr/local/go/src/runtime/panic.go:221
    runtime.sigpanic()
    	/usr/local/go/src/runtime/signal_unix.go:735 +0x327 fp=0xc0003675d8 sp=0xc000367588 pc=0x4b5c47
    github.com/bytedance/sonic/internal/native/avx2.__native_entry__()
    	/data/gopath/pkg/mod/github.com/bytedance/[email protected]/internal/native/avx2/native_amd64.s:1618 +0x1565 fp=0xc0003675e0 sp=0xc0003675d8 pc=0x910925
    created by gateway/handler.selectAppItemsBidResponse
    	/home/ubuntu/remotejenkins/workspace/Gears_Docker_tar2/dsp-gateway/src/handler/dispatcher-handler.go:114 +0x2af
    
    opened by lxj15398019970 5
  • style: using gofmt to format your code

    style: using gofmt to format your code

    It is weird to see

    import (
        `io/ioutil`
        `testing`
    
        `github.com/stretchr/testify/require`
    )
    

    where use `` instead of ""

    if we want others to participate in, we should follow some basic go-style code style

    just use gofmt -w . in /sonic to format all *.go files, please

    opened by wymli 5
  • internal\loader\loader.go:28:11: undefined: syscall.MAP_ANON

    internal\loader\loader.go:28:11: undefined: syscall.MAP_ANON

    D:\go\pkg\mod\github.com\bytedance\[email protected]\internal\loader\loader.go:28:11: undefined: syscall.MAP_ANON D:\go\pkg\mod\github.com\bytedance\[email protected]\internal\loader\loader.go:28:31: undefined: syscall.MAP_PRIVATE D:\go\pkg\mod\github.com\bytedance\[email protected]\internal\loader\loader.go:29:11: undefined: syscall.PROT_READ D:\go\pkg\mod\github.com\bytedance\[email protected]\internal\loader\loader.go:29:31: undefined: syscall.PROT_EXEC D:\go\pkg\mod\github.com\bytedance\[email protected]\internal\loader\loader.go:30:11: undefined: syscall.PROT_READ D:\go\pkg\mod\github.com\bytedance\[email protected]\internal\loader\loader.go:30:31: undefined: syscall.PROT_WRITE D:\go\pkg\mod\github.com\bytedance\[email protected]\internal\loader\loader.go:59:19: undefined: syscall.RawSyscall6 D:\go\pkg\mod\github.com\bytedance\[email protected]\internal\loader\loader.go:59:39: undefined: syscall.SYS_MMAP D:\go\pkg\mod\github.com\bytedance\[email protected]\internal\loader\loader.go:67:21: undefined: syscall.RawSyscall D:\go\pkg\mod\github.com\bytedance\[email protected]\internal\loader\loader.go:67:40: undefined: syscall.SYS_MPROTECT D:\go\pkg\mod\github.com\bytedance\[email protected]\internal\loader\loader.go:67:40: too many errors

    documentation 
    opened by overstarry 5
  • sonic.Unmarshal decode json string failed when field's type is

    sonic.Unmarshal decode json string failed when field's type is "stringint"

    i found out the problem rule: when the type of struct defined is not match with the json, the field and the fields behind it all empty.

    type Test struct {
         Num1 string `json:"num1"`
         Num2 string `json:"num2"`
         Num3 string `json:"num3"`
         Num4 string `json:"num4"`
         Num5 string `json:"num5"`
     }
    testStr := `{"num1":"123","num2":"this is string2", "num3":456, "num4":789, "num5":"this is string"}`
    var t Test
    sonic.Unmarshal([]byte(testStr), &t)
    fmt.Printf("%#v", t)
    //output:  {Num1:"123", Num2:"this is string2", Num3:"", Num4:"", Num5:""}
    

    如果有一个字段类型不匹配的话,那么他后面的字段都会变成空值。我觉得这个处理很不友好吧。直接影响了其他字段的值。 而且err可以捕获,但是错误非结构化信息,无法针对化处理,原生json处理并不会影响后续字段解析。

    之前的问题还没讨论结果就关闭了,最好问题明确之后再关闭,谢谢 @chenzhuoyu @AsterDY

    opened by gy0624ww 4
  • test: error behaviors

    test: error behaviors

    It seems there are three major differences with encoding/json:

    • TestErrorUnmarshalInvalidJSON: Std will validate input JSON before assigning values and leave the original object intact. But sonic DOES NOT validate ahead, thus the original object may be modified;
    • TestErrorUnmarshalDismatchedJSON: If any part of the input is a valid JSON value but mismatches the assigned object, std will skip this value and continue decoding. Sonic DOES NOT skip it and will immediately stop decoding;
    • TestErrorMarshalInvalidUTF8: Std will validate UTF-8 format of a string and output \ufffd if it is invalid. Sonic DOES NOT validate UTF-8 format.
    opened by AsterDY 0
  • sonic will clean interface's value pointer when encountering error

    sonic will clean interface's value pointer when encountering error

    At present for generic decoding, sonic will clean the output interface's value pointer when encountering an error. Such as:

    var (
        testErrStr   = `{"a":[1,2,3,4,],"b":"1"}`
        testErrBytes = []byte(testErrStr)
    )
    
    func TestDecodeGenericError(t *testing.T) {
        var v interface{} 
    
        e := json.NewDecoder(bytes.NewBuffer(testErrBytes))
        err := e.Decode(&v)
        assert.NotNil(t,err)
        assert.NotNil(t,v)
        t.Logf("%#v",v)
    
        err = sonic.ConfigStd.Unmarshal(testErrBytes, &v)
        assert.NotNil(t,err)
        assert.NotNil(t,v)
        t.Logf("%#v",v)
    }
    -------------output-------------
    --- FAIL: TestUnmarshal (0.00s)
        /Users/admin/Desktop/kitex/sonic3/issue_test/issuex_test.go:24: 
            	Error Trace:	issuex_test.go:24
            	Error:      	Expected value not to be nil.
            	Test:       	TestUnmarshal
        /Users/admin/Desktop/kitex/sonic3/issue_test/issuex_test.go:25: <nil>
        /Users/admin/Desktop/kitex/sonic3/issue_test/issuex_test.go:30: map[string]interface {}{"a":[]interface {}{1, 2, 3, 4, interface {}(nil)}}
    FAIL
    

    This is not consistent with std lib.

    opened by AsterDY 0
  • [GLCC 赛题] 开发arm64汇编到go plan9汇编转换工具

    [GLCC 赛题] 开发arm64汇编到go plan9汇编转换工具

    🚩GLCC 高校编程夏令营赛题 👏欢迎大家参与~

    GLCC 是由中国计算机学会举办的高校开源编程夏令营活动,活动为项目准备了开源奖学金。请在以下链接中报名参与⬇️ 报名链接: https://www.gitlink.org.cn/glcc/subjects/detail/224 关于issue相关问题、解题思路的答疑将在群内进行

    image

    1、题目描述 sonic是字节自研的一款高性能JSON库,其底层文本处理算子采用C语言开发,并基于go-assembly机制结合汇编转换工具(asm2asm)嵌入到go语言接口中。但是当前的asm2asm工具仅支持x86-64到plan9的转换,限制了sonic在arm架构环境下的使用。随着arm架构芯片的流行,sonic计划支持arm架构,asm2asm工具支持arm64到plan9的转换也在规划之中。

    2、编码任务

    • 实现一个ARMv8-A汇编到go plan9汇编的转换工具
    • 支持命令行交互(linux or macOSX),最好可以直接对原始C文件调用clang(或gcc)生成汇编 (参考https://github.com/bytedance/sonic/blob/main/Makefile)
    • 自动生成对应的go函数接口及其单测(可选)

    3、技能要求和编程语言

    • ARMv8-A 汇编
    • Go plan9 汇编 (https://go.dev/doc/asm)
    • 编译原理,可以借助第三方lib做前端解析处理
    • 实现语言不限

    4、预期完成结果

    • 将纯c实现的文件(sonic/native/xx.c) 转换为go arm64汇编文件(sonic/internal/native/xx_arm64.s),并且在arm64环境下可跑通go代码单测(sonic/internal/native/native_amd64_test.go)

    GitHub 链接:

    • sonic - https://github.com/bytedance/sonic
    • asm2asm - https://github.com/chenzhuoyu/asm2asm
    enhancement 
    opened by cloudwegoIce 1
  • feat (encoder): add encoder option `NoNullSliceOrMap`

    feat (encoder): add encoder option `NoNullSliceOrMap`

    Benchmark

    • before
    goos: darwin
    goarch: amd64
    pkg: github.com/bytedance/sonic/encoder
    cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    BenchmarkOptionSliceOrMapNoNull/true-16                 10000000               246.6 ns/op           286 B/op          5 allocs/op
    BenchmarkOptionSliceOrMapNoNull/false-16                10000000               247.4 ns/op           288 B/op          5 allocs/op
    
    • after
    goos: darwin
    goarch: amd64
    pkg: github.com/bytedance/sonic/encoder
    cpu: Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    BenchmarkOptionSliceOrMapNoNull/true-16                 10000000               247.4 ns/op           282 B/op          5 allocs/op
    BenchmarkOptionSliceOrMapNoNull/false-16                10000000               236.5 ns/op           283 B/op          5 allocs/op
    
    • conclusion no significant effects on performance

    • interesting discovery writing 4 chars (instruction MOVL) is faster than writing 2 chars (instruction MOVW)?

    opened by AsterDY 0
  • decodeTypedPointer panic

    decodeTypedPointer panic

    unexpected fault address 0x0 fatal error: fault [signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x7f07d800953c]

    goroutine 1 [running]: runtime.throw({0x1414f52, 0xcfabd3}) runtime/panic.go:1198 +0x71 fp=0xc0004fd8d8 sp=0xc0004fd8a8 pc=0x829371 runtime.sigpanic() runtime/signal_unix.go:742 +0x2f6 fp=0xc0004fd928 sp=0xc0004fd8d8 pc=0x83fa16 runtime.__json_decoder_7f07d8006000() ?:0 +0x353c fp=0xc0004fda28 sp=0xc0004fd928 pc=0x7f07d800953c github.com/bytedance/sonic/decoder.decodeTypedPointer({0xc007a98000, 0x8810}, 0x8, 0xa000, 0x0, 0x7f07d81e4100, 0x2) github.com/bytedance/[email protected]/decoder/primitives.go:33 +0xb0 fp=0xc0004fdaa8 sp=0xc0004fda28 pc=0xd6e250 github.com/bytedance/sonic/decoder.(*Decoder).Decode(0xc0004fdb68, {0x1279e60, 0xc0008b3580}) github.com/bytedance/[email protected]/decoder/decoder.go:73 +0xec fp=0xc0004fdb48 sp=0xc0004fdaa8 pc=0xd59e0c github.com/bytedance/sonic.UnmarshalString({0xc007a98000, 0x0}, {0x1279e60, 0xc0008b3580}) github.com/bytedance/[email protected]/sonic.go:46 +0x67 fp=0xc0004fdbb8 sp=0xc0004fdb48 pc=0xd815c7

    opened by jank1369 2
Releases(v1.3.1)
  • v1.3.1(Jun 9, 2022)

    Feature

    • [#237] opt: remove bound and loop unrolling in the quote process. The benchmark showed the encoding performance was accelerated by 10%~20%.

    CI

    • [#239] test: add more JSON samples and corresponding benchmarks, aiming at performance comparison with https://github.com/go-json-experiment/json

    Full Changelog: https://github.com/bytedance/sonic/compare/v1.3.0...v1.3.1

    Source code(tar.gz)
    Source code(zip)
  • v1.3.0(May 25, 2022)

    Feature

    • [#233] add configurable sonic.API to be compatible with encoding/json library and ARM arch.
    • [#225] support streaming IO with encoder.StreamEncoder and decoder.StreamDecoder.
    • [#228] support windows OS.

    Bugfix

    • [#230] (encoder) fix the invalid length of output buffer caused by register-scratching bug under encoding error.
    • [#224] (encoder) add defensive string-pointer-checking to avoid crashes on native-C stack.

    Full Changelog: https://github.com/bytedance/sonic/compare/v1.2.0...v1.3.0

    Source code(tar.gz)
    Source code(zip)
  • v1.2.2(May 23, 2022)

    What's Changed

    • fix: some typos by @liuq19 in https://github.com/bytedance/sonic/pull/221
    • fix: add nil pointer check by @AsterDY in https://github.com/bytedance/sonic/pull/224
    • fix (encoder): spill RL (buffer length) register while calling internal encoders in case of it got scratched by @AsterDY in https://github.com/bytedance/sonic/pull/230
    • feat: support for Windows by @ii64 in https://github.com/bytedance/sonic/pull/228

    New Contributors

    • @ii64 made their first contribution in https://github.com/bytedance/sonic/pull/228

    Full Changelog: https://github.com/bytedance/sonic/compare/v1.2.1...v1.2.2

    Source code(tar.gz)
    Source code(zip)
  • v1.2.1(Apr 21, 2022)

    What's Changed

    [#216 ]doc: update readme [#215] fix: base64x' native decoder stack-overflow under AVX2 CPU

    Full Changelog: https://github.com/bytedance/sonic/compare/v1.2.0...v1.2.1

    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Apr 2, 2022)

    Feature

    • [#214] support Go 1.18
    • [#208] add fuzz test in Go 1.18
    • [#197] add API MarshalString()
    • [#192] (decoder) add option CopyString()
    • [#196] allow decoding string value as json.Number in binding scenario

    Bugfix

    • [#194] check nil pointer when encoding a pointer receiver of json.Marshaler
    • [#188] fix nil panic when loading on plugin mode
    • [#209] ignore overflow-check while decoding json.Number and optimize

    Full Changelog: https://github.com/bytedance/sonic/compare/v1.1.1...v1.2.0

    Source code(tar.gz)
    Source code(zip)
  • v1.1.1(Feb 28, 2022)

    Feature

    • [#192] (decoder) add option CopyString()

    Bugfix

    • [#194] check nil pointer for Marshaler by @AsterDY in https://github.com/bytedance/sonic/pull/194

    Full Changelog: https://github.com/bytedance/sonic/compare/v1.1.0...v1.1.1

    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Feb 22, 2022)

    Feature

    [#171] support HTML escaping [#189] support JSON validation and skipping [#168] add optimizing options encoder.CompactMarshaler and encoder.NoQuoteTextMarshaler

    Bugfix

    [#187] prevent premature GC on decoded string values [#181] add GC write barrier for Go1.15 [#179] check number ending when skipping numbers

    Full Changelog: https://github.com/bytedance/sonic/compare/v1.0.0...v1.1.0

    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Dec 31, 2021)

    Feature

    • [#155] ast.Node supports containing interface{}
    • [#164] ast.Node supports sorting pair keys
    • [#152] syntax error always shows related JSON (quoted)

    Bugfix

    • [#148] align 'string' option in struct tag as encoding/json
    • [#155] quote string when call node.MarshalJSON()
    • [#156, #159] relocate stack pointers for GC recognition
    • [#161] prevent out-of-bounds accessing in native.advance_string()
    • [#162] shrink function stacks and achieve stack-split check
    • [#166] keep passed buffer alive when encoding

    Full Changelog: https://github.com/bytedance/sonic/compare/v1.0.0-rc.3...v1.0.0

    Source code(tar.gz)
    Source code(zip)
  • v1.0.0-rc.3(Dec 2, 2021)

    Feature

    • [#116] support Go 1.17
    • [#137] add compiler option WithCompileRecursiveDepth to pretouch recursively for large/deep struct

    Bugfix

    • [#142] correct the field priority as encoding/json in case-insensitive match
    • [#145] use global variables to keep all arguments and locals alive
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0-rc.2(Nov 18, 2021)

    Bugfix

    • [#127] keep internal pointers alive whileMarshal() or Unmarshal()
    • [#131] generate gcWriteBarrier in JIT (only support Go1.16 [#134])
    • [#132] fix stack-overflow error (json.UnsupportedValue) after frequent errors
    • [#129] avoid the pointer of argumentval cleared while debugging
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0-rc.1(Oct 28, 2021)

    Improvement

    • [#124] add node.Load()/LoadAll() to support concurrency

    Bugfix

    • [#122 ] add runtime.KeepAlive() to avoid premature GC during Marshal()/Unmarshal()
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0-rc(Oct 22, 2021)

    Bugfix

    • [ #107 ] ignored the UTF-16 surrogate pair after an invalid unicode
    • [ #108 ] couldn't skip zero-based exponent in skip_number()
    • [ #112 ] returned error when unmarshalling invalid base64-encoded strings
    • [ #113 ] redundantly handled struct-typed fields' "omitempty" tag
    • [ #115 ] encoder.SortMapKeys flag didn't work when marshalling integer-key map
    Source code(tar.gz)
    Source code(zip)
Owner
Bytedance Inc.
Bytedance Inc.
JSON Spanner - A Go package that provides a fast and simple way to filter or transform a json document

JSON SPANNER JSON Spanner is a Go package that provides a fast and simple way to

null 3 Jun 30, 2022
JSON diff library for Go based on RFC6902 (JSON Patch)

jsondiff jsondiff is a Go package for computing the diff between two JSON documents as a series of RFC6902 (JSON Patch) operations, which is particula

William Poussier 172 Jun 23, 2022
Get JSON values quickly - JSON parser for Go

get json values quickly GJSON is a Go package that provides a fast and simple way to get values from a json document. It has features such as one line

Josh Baker 10.5k Jun 30, 2022
Package json implements encoding and decoding of JSON as defined in RFC 7159

Package json implements encoding and decoding of JSON as defined in RFC 7159. The mapping between JSON and Go values is described in the documentation for the Marshal and Unmarshal functions

High Performance, Kubernetes Native Object Storage 3 May 10, 2022
Json-go - CLI to convert JSON to go and vice versa

Json To Go Struct CLI Install Go version 1.17 go install github.com/samit22/js

Samit Ghimire 5 Mar 3, 2022
Fast JSON parser and validator for Go. No custom structs, no code generation, no reflection

fastjson - fast JSON parser and validator for Go Features Fast. As usual, up to 15x faster than the standard encoding/json. See benchmarks. Parses arb

Aliaksandr Valialkin 1.6k Jun 21, 2022
Fast and flexible JSON encoder for Go

Jettison Jettison is a fast and flexible JSON encoder for the Go programming language, inspired by bet365/jingo, with a richer features set, aiming at

William Poussier 128 Jun 1, 2022
Fast JSON serializer for golang.

easyjson Package easyjson provides a fast and easy way to marshal/unmarshal Go structs to/from JSON without the use of reflection. In performance test

Free and open source software developed at Mail.Ru 3.8k Jun 22, 2022
Fast Color JSON Marshaller + Pretty Printer for Golang

ColorJSON: The Fast Color JSON Marshaller for Go What is this? This package is based heavily on hokaccha/go-prettyjson but has some noticible differen

Tyler Brock 110 Jun 16, 2022
A fast json parser for go

rjson rjson is a json parser that relies on Ragel-generated state machines for most parsing. rjson's api is minimal and focussed on efficient parsing.

WillAbides 47 Jun 15, 2022
Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang

kazaam Description Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang. This functionality provides

Qntfy 205 Sep 17, 2021
Fast json for go. Lightweight fork of jsoniter.

jx Fast json for go. Lightweight fork of jsoniter. Features Reduced scope (no reflection or encoding/json adapter) Fuzzing, improved test coverage Dra

ogen 61 Jun 25, 2022
library for working amorphous data (as when you decode json into an interface{})

Introduction Decoding json into an interface{} produces an hierarchical arrangement of four data types: float64, string are 'primative types' and form

Chuck Luciano 9 Jul 5, 2021
A library to query the godoc.org JSON API.

gopkg This repository provides minimal Go package that makes queries against the godoc.org JSON API. Since that site has mostly been subsumed by pkg.g

M. J. Fromberger 2 Nov 26, 2021
Copy of Golang's json library with IsZero feature

json Copy of Golang's json library with IsZero feature from CL13977 Disclaimer It is a package primary used for my own projects, I will keep it up-to-

Ferenc Fabian 4 Oct 9, 2021
Fork of Go's standard library json encoder

A fork of the Go standard library's json encoder Why? https://github.com/golang/go/issues/6213 was proposed in 2013 but was never accepted. Difference

unchain.io 0 Nov 25, 2021
JSON Unmarshalling Library

JSAWN (JAY-sawn) This is a JSON library to add to the capabilities of the standard 'encoding/json' library. Unmarshalling The first enhancement is to

null 1 Feb 16, 2022
Abstract JSON for golang with JSONPath support

Abstract JSON Abstract JSON is a small golang package provides a parser for JSON with support of JSONPath, in case when you are not sure in its struct

Stepan Pyzhov 119 Jun 24, 2022
Small utility to create JSON objects

gjo Small utility to create JSON objects. This was inspired by jpmens/jo. Support OS Mac Linux Windows Requirements Go 1.1.14~ Git Installtion Build $

skanehira 108 Apr 27, 2022