Fast JSON encoder/decoder compatible with encoding/json for Go

Overview

go-json

Go GoDoc codecov

Fast JSON encoder/decoder compatible with encoding/json for Go

Installation

go get github.com/goccy/go-json

How to use

Replace import statement from encoding/json to github.com/goccy/go-json

-import "encoding/json"
+import "github.com/goccy/go-json"

JSON library comparison

name encoder decoder compatible with encoding/json
encoding/json yes yes N/A
json-iterator/go yes yes partial
easyjson yes yes no
gojay yes yes no
segmentio/encoding/json yes yes yes
jettison yes no no
simdjson-go no yes no
goccy/go-json yes yes yes
  • json-iterator/go isn't compatible with encoding/json in many ways, but it hasn't been supported for a long time.

Benchmarks

$ cd benchmarks
$ go test -bench .

Encode

Decode

How it works

go-json is very fast in both encoding and decoding compared to other libraries. It's easier to implement by using automatic code generation for performance or by using a dedicated interface, but go-json dares to stick to compatibility with encoding/json and is the simple interface. Despite this, we are developing with the aim of being the fastest library.

Here, we explain the various speed-up techniques implemented by go-json.

Basic technique

The techniques listed here are the ones used by most of the libraries listed above.

Buffer reuse

Since the only value required for the result of json.Marshal(interface{}) ([]byte, error) is []byte, the only value that must be allocated during encoding is the return value []byte .

Also, as the number of allocations increases, the performance will be affected, so the number of allocations should be kept as low as possible when creating []byte.

Therefore, there is a technique to reduce the number of times a new buffer must be allocated by reusing the buffer used for the previous encoding by using sync.Pool.

Finally, you allocate a buffer that is as long as the resulting buffer and copy the contents into it, you only need to allocate the buffer once in theory.

type buffer struct {
    data []byte
}

var bufPool = sync.Pool{
    New: func() interface{} {
        return &buffer{data: make([]byte, 0, 1024)}
    },
}

buf := bufPool.Get().(*buffer)
data := encode(buf.data) // reuse buf.data

newBuf := make([]byte, len(data))
copy(newBuf, buf)

buf.data = data
bufPool.Put(buf)

Elimination of reflection

As you know, the reflection operation is very slow.

Therefore, using the fact that the address position where the type information is stored is fixed for each binary ( we call this typeptr ), we can use the address in the type information to call a pre-built optimized process.

For example, you can get the address to the type information from interface{} as follows and you can use that information to call a process that does not have reflection.

To process without reflection, pass a pointer (unsafe.Pointer) to the value is stored.

type emptyInterface struct {
    typ unsafe.Pointer
    ptr unsafe.Pointer
}

var typeToEncoder = map[uintptr]func(unsafe.Pointer)([]byte, error){}

func Marshal(v interface{}) ([]byte, error) {
    iface := (*emptyInterface)(unsafe.Pointer(&v)
    typeptr := uintptr(iface.typ)
    if enc, exists := typeToEncoder[typeptr]; exists {
        return enc(iface.ptr)
    }
    ...
}

※ In reality, typeToEncoder can be referenced by multiple goroutines, so exclusive control is required.

Unique speed-up technique

Encoder

Do not escape arguments of Marshal

json.Marshal and json.Unmarshal receive interface{} value and they perform type determination dynamically to process. In normal case, you need to use the reflect library to determine the type dynamically, but since reflect.Type is defined as interface, when you call the method of reflect.Type, The reflect's argument is escaped.

Therefore, the arguments for Marshal and Unmarshal are always escape to the heap. However, go-json can use the feature of reflect.Type while avoiding escaping.

reflect.Type is defined as interface, but in reality reflect.Type is implemented only by the structure rtype defined in the reflect package. For this reason, to date reflect.Type is the same as *reflect.rtype.

Therefore, by directly handling *reflect.rtype, which is an implementation of reflect.Type, it is possible to avoid escaping because it changes from interface to using struct.

The technique for working with *reflect.rtype directly from go-json is implemented at https://github.com/goccy/go-json/blob/master/rtype.go.

Also, the same technique is cut out as a library ( https://github.com/goccy/go-reflect )

Initially this feature was the default behavior of go-json. But after careful testing, I found that I passed a large value to json.Marshal() and if the argument could not be assigned to the stack, it could not be properly escaped to the heap (a bug in the Go compiler).

Therefore, this feature will be provided as an optional until this issue is resolved.

To use it, add NoEscape like MarshalNoEscape()

Encoding using opcode sequence

I explained that you can use typeptr to call a pre-built process from type information.

In other libraries, this dedicated process is processed by making it an function calling like anonymous function, but function calls are inherently slow processes and should be avoided as much as possible.

Therefore, go-json adopted the Instruction-based execution processing system, which is also used to implement virtual machines for programming language.

If it is the first type to encode, create the opcode ( instruction ) sequence required for encoding. From the second time onward, use typeptr to get the cached pre-built opcode sequence and encode it based on it. An example of the opcode sequence is shown below.

json.Marshal(struct{
    X int `json:"x"`
    Y string `json:"y"`
}{X: 1, Y: "hello"})

When encoding a structure like the one above, create a sequence of opcodes like this:

- opStructFieldHead ( `{` )
- opStructFieldInt ( `"x": 1,` )
- opStructFieldString ( `"y": "hello"` )
- opStructEnd ( `}` )
- opEnd

※ When processing each operation, write the letters on the right.

In addition, each opcode is managed by the following structure ( Pseudo code ).

type opType int
const (
    opStructFieldHead opType = iota
    opStructFieldInt
    opStructFieldStirng
    opStructEnd
    opEnd
)
type opcode struct {
    op opType
    key []byte
    next *opcode
}

The process of encoding using the opcode sequence is roughly implemented as follows.

func encode(code *opcode, b []byte, p unsafe.Pointer) ([]byte, error) {
    for {
        switch code.op {
        case opStructFieldHead:
            b = append(b, '{')
            code = code.next
        case opStructFieldInt:
            b = append(b, code.key...)
            b = appendInt((*int)(unsafe.Pointer(uintptr(p)+code.offset)))
            code = code.next
        case opStructFieldString:
            b = append(b, code.key...)
            b = appendString((*string)(unsafe.Pointer(uintptr(p)+code.offset)))
            code = code.next
        case opStructEnd:
            b = append(b, '}')
            code = code.next
        case opEnd:
            goto END
        }
    }
END:
    return b, nil
}

In this way, the huge switch-case is used to encode by manipulating the linked list opcodes to avoid unnecessary function calls.

Opcode sequence optimization

One of the advantages of encoding using the opcode sequence is the ease of optimization. The opcode sequence mentioned above is actually converted into the following optimized operations and used.

- opStructFieldHeadInt ( `{"x": 1,` )
- opStructEndString ( `"y": "hello"}` )
- opEnd

It has been reduced from 5 opcodes to 3 opcodes ! Reducing the number of opcodees means reducing the number of branches with switch-case. In other words, the closer the number of operations is to 1, the faster the processing can be performed.

In go-json, optimization to reduce the number of opcodes itself like the above and it speeds up by preparing opcodes with optimized paths.

Change recursive call from CALL to JMP

Recursive processing is required during encoding if the type is defined recursively as follows:

type T struct {
    X int
    U *U
}

type U struct {
    T *T
}

b, err := json.Marshal(&T{
    X: 1,
    U: &U{
        T: &T{
            X: 2,
        },
    },
})
fmt.Println(string(b)) // {"X":1,"U":{"T":{"X":2,"U":null}}}

In go-json, recursive processing is processed by the operation type of opStructFieldRecursive.

In this operation, after acquiring the opcode sequence used for recursive processing, the function is not called recursively as it is, but the necessary values ​​are saved by itself and implemented by moving to the next operation.

The technique of implementing recursive processing with the JMP operation while avoiding the CALL operation is a famous technique for implementing a high-speed virtual machine.

For more details, please refer to the article ( but Japanese only ).

Dispatch by typeptr from map to slice

When retrieving the data cached from the type information by typeptr, we usually use map. Map requires exclusive control, so use sync.Map for a naive implementation.

However, this is slow, so it's a good idea to use the atomic package for exclusive control as implemented by segmentio/encoding/json ( https://github.com/segmentio/encoding/blob/master/json/codec.go#L41-L55 ).

This implementation slows down the set instead of speeding up the get, but it works well because of the nature of the library, it encodes much more for the same type.

However, as a result of profiling, I noticed that runtime.mapaccess2 accounts for a significant percentage of the execution time. So I thought if I could change the lookup from map to slice.

There is an API named typelinks defined in the runtime package that the reflect package uses internally. This allows you to get all the type information defined in the binary at runtime.

The fact that all type information can be acquired means that by constructing slices in advance with the acquired total number of type information, it is possible to look up with the value of typeptr without worrying about out-of-range access.

However, if there is too much type information, it will use a lot of memory, so by default we will only use this optimization if the slice size fits within 2Mib .

If this approach is not available, it will fall back to the atomic based process described above.

If you want to know more, please refer to the implementation here

Decoder

Dispatch by typeptr from map to slice

Like the encoder, the decoder also uses typeptr to call the dedicated process.

Faster termination character inspection using NUL character

In order to decode, you have to traverse the input buffer character by position. At that time, if you check whether the buffer has reached the end, it will be very slow.

buf : []byte type variable. holds the string passed to the decoder cursor : int64 type variable. holds the current read position

buflen := len(buf)
for ; cursor < buflen; cursor++ { // compare cursor and buflen at all times, it is so slow.
    switch buf[cursor] {
    case ' ', '\n', '\r', '\t':
    }
}

Therefore, by adding the NUL (\000) character to the end of the read buffer as shown below, it is possible to check the termination character at the same time as other characters.

for {
    switch buf[cursor] {
    case ' ', '\n', '\r', '\t':
    case '\000':
        return nil
    }
    cursor++
}

Use Boundary Check Elimination

Due to the NUL character optimization, the Go compiler does a boundary check every time, even though buf[cursor] does not cause out-of-range access.

Therefore, go-json eliminates boundary check by fetching characters for hotspot by pointer operation. For example, the following code.

func char(ptr unsafe.Pointer, offset int64) byte {
	return *(*byte)(unsafe.Pointer(uintptr(ptr) + uintptr(offset)))
}

p := (*sliceHeader)(&unsafe.Pointer(buf)).data
for {
    switch char(p, cursor) {
    case ' ', '\n', '\r', '\t':
    case '\000':
        return nil
    }
    cursor++
}

Checking the existence of fields of struct using Bitmaps

I found by the profiling result, in the struct decode, lookup process for field was taking a long time.

For example, consider decoding a string like {"a":1,"b":2,"c":3} into the following structure:

type T struct {
    A int `json:"a"`
    B int `json:"b"`
    C int `json:"c"`
}

At this time, it was found that it takes a lot of time to acquire the decoding process corresponding to the field from the field name as shown below during the decoding process.

fieldName := decodeKey(buf, cursor) // "a" or "b" or "c"
decoder, exists := fieldToDecoderMap[fieldName] // so slow
if exists {
    decoder(buf, cursor)
} else {
    skipValue(buf, cursor)
}

To improve this process, json-iterator/go is optimized so that it can be branched by switch-case when the number of fields in the structure is 10 or less (switch-case is faster than map). However, there is a risk of hash collision because the value hashed by the FNV algorithm is used for conditional branching. Also, gojay processes this part at high speed by letting the library user yourself write switch-case.

go-json considers and implements a new approach that is different from these. I call this bitmap field optimization.

The range of values ​​per character can be represented by [256]byte. Also, if the number of fields in the structure is 8 or less, int8 type can represent the state of each field. In other words, it has the following structure.

  • Base ( 8bit ): 00000000
  • Key "a": 00000001 ( assign key "a" to the first bit )
  • Key "b": 00000010 ( assign key "b" to the second bit )
  • Key "c": 00000100 ( assign key "c" to the third bit )

Bitmap structure is the following

        | key index(0) |
------------------------
 0      | 00000000     |
 1      | 00000000     |
~~      |              |
97 (a)  | 00000001     |
98 (b)  | 00000010     |
99 (c)  | 00000100     |
~~      |              |
255     | 00000000     |

You can think of this as a Bitmap with a height of 256 and a width of the maximum string length in the field name. In other words, it can be represented by the following type .

[maxFieldKeyLength][256]int8`

When decoding a field character, check whether the corresponding character exists by referring to the pre-built bitmap like the following.

var curBit int8 = math.MaxInt8 // 11111111

c := char(buf, cursor)
bit := bitmap[keyIdx][c]
curBit &= bit
if curBit == 0 {
    // not found field
}

If curBit is not 0 until the end of the field string, then the string is You may have hit one of the fields. But the possibility is that if the decoded string is shorter than the field string, you will get a false hit.

  • input: {"a":1}
type T struct {
    X int `json:"abc"`
}

※ Since a is shorter than abc, it can decode to the end of the field character without curBit being 0.

Rest assured. In this case, it doesn't matter because you can tell if you hit by comparing the string length of a with the string length of abc.

Finally, calculate the position of the bit where 1 is set and get the corresponding value, and you're done.

Using this technique, field lookups are possible with only bitwise operations and access to slices.

go-json uses a similar technique for fields with 9 or more and 16 or less fields. At this time, Bitmap is constructed as [maxKeyLen][256]int16 type.

Currently, this optimization is not performed when the maximum length of the field name is long (specifically, 64 bytes or more) in addition to the limitation of the number of fields from the viewpoint of saving memory usage.

Others

I have done a lot of other optimizations. I will find time to write about them. If you have any questions about what's written here or other optimizations, please visit the #go-json channel on gophers.slack.com .

Reference

Regarding the story of go-json, there are the following articles in Japanese only.

License

MIT

Comments
  • Panic when marshaling a map with custom marshaler

    Panic when marshaling a map with custom marshaler

    I just switched some working code from encoding/json to github.com/goccy/go-json and this caused a panic. Sometimes it panics with fatal error: invalid pointer found on stack and sometimes with runtime error: invalid memory address or nil pointer dereference.

    The error occurs when using a map, where the values are a custom map type which implements the json.Marshaler interface.

    Here's some code to reproduce the error. It works fine if you change github.com/goccy/go-json back to encoding/json:

    package main
    
    import (
    	"log"
    
    	"github.com/goccy/go-json"
    )
    
    type Set map[string]struct{}
    
    func NewSet(items ...string) Set {
    	s := make(Set, len(items))
    	for _, v := range items {
    		s[v] = struct{}{}
    	}
    	return s
    }
    
    func (s Set) MarshalJSON() ([]byte, error) {
    	if s == nil {
    		return []byte("null"), nil
    	}
    
    	if len(s) == 0 {
    		return []byte("[]"), nil
    	}
    
    	size := 1
    	for symbol := range s {
    		size += len(symbol) + 3
    	}
    
    	b := make([]byte, 0, size)
    	b = append(b, '[')
    	for symbol := range s {
    		b = append(b, '"')
    		b = append(b, symbol...)
    		b = append(b, '"', ',')
    	}
    	b[len(b)-1] = ']'
    
    	return b, nil
    }
    
    func main() {
    	m := map[string]Set{"foo": NewSet("foo", "bar")}
    	b, err := json.Marshal(m)
    	if err != nil {
    		log.Fatal(err)
    	}
    	log.Println(string(b))
    }
    
    opened by abemedia 1
  • Panic

    Panic

    I cut structures for readability. It works with "encoding/json", but panics with "github.com/goccy/go-json"

    package main
    
    import (
    	"github.com/goccy/go-json"
    )
    
    // "encoding/json"
    // "github.com/goccy/go-json"
    
    type Auth struct {
    	Id                  int                  `json:"id"`
    	Login               string               `json:"login"`
    	CurrentOrganization *CurrentOrganization `json:"organization,omitempty" db:"organization"`
    }
    
    type CurrentOrganization struct {
    	Id int `json:"id"`
    }
    
    type User struct {
    	Auth
    }
    
    type AddUser struct {
    	User
    }
    
    func main() {
    	user := new(AddUser)
    	user.Login = "user756"
    
    	jsonUser, err := json.Marshal(user)
    	if err != nil {
    		println("error:", err.Error())
    	}
    	println("user:", jsonUser)
    }
    
    bug 
    opened by kaibox-git 1
  • Support for custom tags

    Support for custom tags

    Can support for custom tag while marshalling and unmarshalling struct be added to the library ?

    Usecase - We receive json response in wierd key name but while storing and forwarding the same response I want to fix the key names in response.

    Current solution -

    1. I am aware that I can create 2 struct like below and typecast them
    type Incoming struct {
     Name string `cust:"n"`
     Age string `cust:"a"`
    }
    
    type Out struct {
     Name string `json:"name"`
     Age string `json:"age"`
    }
    
    incoming := Incoming{Name: "name", age: "25"}
    out := Out(incoming)
    

    This would give exactly the same result but would require me to maintain 2 exactly same structs.

    Therefore I was wondering if we can directly add the support for custom tag which would be ewxposed with Options flag I have implemented the same in my forked version - https://github.com/goccy/go-json/compare/master...jsjain:go-json:master

    Usage -

    json.UnmarshalWithOption(responseBytes, &responseStruct, json.DecodeWithTagName("cust"))
    

    PS: Currently i have not added support for stream decoder

    feature request 
    opened by jsjain 0
  • Large memory consumption/slow when using decoder.token()

    Large memory consumption/slow when using decoder.token()

    I want to write a program that is able to parse a few specific entries in a json file without needing to load the complete file into memory, in order to minimize both CPU and memory consumption for a docker container. I have been using encoding/json decoder.token() for this. It works but is kinda slow, so I have tried the same with the go-json library. While I get a significant speedup for some of my test files (factor 2-4 faster), the memory consumption is extremely large, needing around 90 MB to parse a 50 MB file. For some files, decoder.Token() of go-json is extremely slow, about 50 times slower than encoding/json.

    Here is a minimal example for showcasing the issue, I simply open the file and then do a for loop getting tokens until I reach the end of the file:

    package main
    
    import (
    	"fmt"
    	"io"
    	"os"
    
    	json "github.com/goccy/go-json"
    )
    
    func main() {
    	jsonFile, err := os.Open("./large-file.json")
    	if err != nil {
    		fmt.Printf("Error opening file: %s\n", err)
    		return
    	}
    	defer func() {
    		if err = jsonFile.Close(); err != nil {
    			fmt.Printf("Error closing file:%s\n", err)
    		}
    	}()
    	decoder := json.NewDecoder(jsonFile)
    	continueParse := true
    	for continueParse {
    		_, err := decoder.Token()
    		if err == io.EOF {
    			continueParse = false
    		}
    		if err != nil {
    			fmt.Printf("Error while parsing: %s\n", err)
    			continueParse = false
    		}
    	}
    }
    

    As an example, I used the 25 MB large file from https://github.com/json-iterator/test-data This file causes both the large memory consumption issue and the slow CPU time issue.

    To make replication easier, I also used a docker container with the following Dockerfile:

    FROM golang:alpine
    WORKDIR /app
    COPY . .
    RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -trimpath -ldflags="-w -s" .
    

    I run the container with limited ressources, e.g.

    docker build -t test:gojson .
    docker run -ti --memory="100m" --cpus="0.5" --memory-swap 100m test:gojson
    ./largeFiles
    

    Observing the container with docker stats $container_name, I saw that the memory consumption goes up to 75 MB and going through the whole file takes about 3 minutes. Doing the exact same thing, but importing encoding/json instead of github.com/goccy/go-json decreases memory consumption to about 8 MB and the time to parse the file to about 4 seconds.

    The slowness seems to be caused by decodeEscapeString, 99.79% of cpu time is spent by runtime memove. Perhaps there is something wrong with line 179 in internal/decoder/string.go?

    The large memory consumption might be caused by internal/decoder/stream.go readBuf(), I did not find anything equivalent to encoding/json, where the data is first slided down before it is checked if the buffer size needs to be doubled:

    func (dec *Decoder) refill() error {
    	// Make room to read more into the buffer.
    	// First slide down data already consumed.
    	if dec.scanp > 0 {
    		dec.scanned += int64(dec.scanp)
    		n := copy(dec.buf, dec.buf[dec.scanp:])
    		dec.buf = dec.buf[:n]
    		dec.scanp = 0
    	}
    
    enhancement performance 
    opened by Maydaer 0
  • Support Decoder options for Unmarshal

    Support Decoder options for Unmarshal

    It would be nice if this library could support Decoder.UseNumber and Decoder.DisallowUnknownFields on the Unmarshal method as well. Currently, as far as I can tell, using these options requires using Decoder, which has two disadvantages:

    • If you have data that is already in a byte array then Decoder is slower according to the benchmarks
    • As far as I know Decoder has no reliable method to ensure that there is no extra data after the read json object (so for example {"value": 323};,fdaslfdaksf is considered valid by the decoder, if you only decode one object).
    feature request 
    opened by FeldrinH 0
Releases(v0.9.11)
  • v0.9.11(Aug 18, 2022)

    What's Changed

    Fix bugs

    • Fix unexpected behavior when buffer ends with backslash ( #383 ) @KimHyeonwoo
    • Fix stream decoding of escaped character ( #387 ) @orisano

    New Contributors

    • @KimHyeonwoo made their first contribution in https://github.com/goccy/go-json/pull/383

    Full Changelog: https://github.com/goccy/go-json/compare/v0.9.10...v0.9.11

    Source code(tar.gz)
    Source code(zip)
  • v0.9.10(Jul 15, 2022)

    Fix bugs

    • Fix boundary exception of type caching ( #382 ) @Trim21

    New Contributors

    • @Trim21 made their first contribution in https://github.com/goccy/go-json/pull/382

    Full Changelog: https://github.com/goccy/go-json/compare/v0.9.9...v0.9.10

    Source code(tar.gz)
    Source code(zip)
  • v0.9.9(Jul 15, 2022)

    Fix bugs

    • Fix encoding of directed interface with typed nil ( #377 ) @orisano
    • Fix embedded primitive type encoding using alias ( #378 ) @orisano
    • Fix slice/array type encoding with types implementing MarshalJSON ( #379 ) @orisano
    • Fix unicode decoding when the expected buffer state is not met after reading ( #380 ) @orisano

    Full Changelog: https://github.com/goccy/go-json/compare/v0.9.8...v0.9.9

    Source code(tar.gz)
    Source code(zip)
  • v0.9.8(Jun 29, 2022)

    Fix bugs

    • Fix decoding of surrogate-pair ( #365 ) by @orisano
    • Fix handling of embedded primitive type ( #366 ) by @orisano
    • Add validation of escape sequence for decoder ( #367 ) by @orisano
    • Fix stream tokenizing respecting UseNumber ( #369 ) by @zeroshade
    • Fix encoding when struct pointer type that implements Marshal JSON is embedded ( #375 ) by @orisano

    Improve performance

    • Improve performance of linkRecursiveCode ( #368 ) by @orisano

    New Contributors

    • @zeroshade made their first contribution in https://github.com/goccy/go-json/pull/369

    Full Changelog: https://github.com/goccy/go-json/compare/v0.9.7...v0.9.8

    Source code(tar.gz)
    Source code(zip)
  • v0.9.7(Apr 21, 2022)

    What's Changed

    • fix: an incompatible behavior on map key decoder by @orisano in https://github.com/goccy/go-json/pull/353
    • fix: add filtering on slow path by @orisano in https://github.com/goccy/go-json/pull/355
    • feat: add DebugWith option by @orisano in https://github.com/goccy/go-json/pull/356
    • fix: add a fallback uint8 sliceDecoder to bytesDecoder by @orisano in https://github.com/goccy/go-json/pull/361
    • fix: to care about the case of OpInterfacePtr by @orisano in https://github.com/goccy/go-json/pull/363

    Full Changelog: https://github.com/goccy/go-json/compare/v0.9.6...v0.9.7

    Source code(tar.gz)
    Source code(zip)
  • v0.9.6(Mar 22, 2022)

    What's Changed

    • fix: to care ints minimum values by @orisano in https://github.com/goccy/go-json/pull/344
    • Update go version by @goccy in https://github.com/goccy/go-json/pull/347
    • feat: improves escapeString's performance by @orisano in https://github.com/goccy/go-json/pull/345
    • fix: to safe when got unexpected typeptr by @orisano in https://github.com/goccy/go-json/pull/351
    • fix: mismatched between len(s.buf) and s.bufSize by @orisano in https://github.com/goccy/go-json/pull/349

    Full Changelog: https://github.com/goccy/go-json/compare/v0.9.5...v0.9.6

    Source code(tar.gz)
    Source code(zip)
  • v0.9.5(Mar 4, 2022)

    What's Changed

    • fix: panic when decoding time.Time with context by @orisano in https://github.com/goccy/go-json/pull/328
    • fix: incorrect handling on skipValue by @orisano in https://github.com/goccy/go-json/pull/341
    • fix: avoid reading the next character in buffer to nul consideration by @orisano in https://github.com/goccy/go-json/pull/338
    • feat: improve performance when a payload contains escape sequence by @orisano in https://github.com/goccy/go-json/pull/334

    Full Changelog: https://github.com/goccy/go-json/compare/v0.9.4...v0.9.5

    Source code(tar.gz)
    Source code(zip)
  • v0.9.4(Jan 20, 2022)

    What's Changed

    • Fix IsNilForMarshaler for string type with omitempty by @IncSW in https://github.com/goccy/go-json/pull/323
    • Fix the case where the embedded field is at the end by @goccy in https://github.com/goccy/go-json/pull/326

    Full Changelog: https://github.com/goccy/go-json/compare/v0.9.3...v0.9.4

    Source code(tar.gz)
    Source code(zip)
  • v0.9.3(Jan 14, 2022)

    What's Changed

    • Fix logic of removing struct field for decoder by @goccy in https://github.com/goccy/go-json/pull/322

    Full Changelog: https://github.com/goccy/go-json/compare/v0.9.2...v0.9.3

    Source code(tar.gz)
    Source code(zip)
  • v0.9.2(Jan 14, 2022)

    What's Changed

    • Add invalid decoder to delay type error judgment at decode by @goccy in https://github.com/goccy/go-json/pull/321

    Full Changelog: https://github.com/goccy/go-json/compare/v0.9.1...v0.9.2

    Source code(tar.gz)
    Source code(zip)
  • v0.9.1(Jan 11, 2022)

    What's Changed

    • Fix encoding of MarshalText/MarshalJSON operation with head offset by @goccy in https://github.com/goccy/go-json/pull/319

    Full Changelog: https://github.com/goccy/go-json/compare/v0.9.0...v0.9.1

    Source code(tar.gz)
    Source code(zip)
  • v0.9.0(Jan 5, 2022)

    New feature

    • Supports dynamic filtering of struct fields ( #314 )

    Improve encoding performance

    • Improve map encoding performance ( #310 )
    • Optimize encoding path for escaped string ( #311 )
    • Add encoding option for performance ( #312 )

    Fix bugs

    • Fix panic at encoding map value on 1.18 ( #310 )
    • Fix MarshalIndent for interface type ( #317 )
    Source code(tar.gz)
    Source code(zip)
  • v0.8.1(Dec 5, 2021)

    What's Changed

    • Fix operation conversion for PtrHead to Head in Recursive type by @goccy in https://github.com/goccy/go-json/pull/305

    Full Changelog: https://github.com/goccy/go-json/compare/v0.8.0...v0.8.1

    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Dec 2, 2021)

    What's Changed

    • Fix embedded field conflict behavior by @goccy in https://github.com/goccy/go-json/pull/300
    • Refactor compiler for encoder by @goccy in https://github.com/goccy/go-json/pull/301
    • Refactor vm code for encoder by @goccy in https://github.com/goccy/go-json/pull/302

    Full Changelog: https://github.com/goccy/go-json/compare/v0.7.10...v0.8.0

    Source code(tar.gz)
    Source code(zip)
  • v0.7.10(Oct 16, 2021)

    What's Changed

    • Fix conversion from pointer to uint64 by @goccy in https://github.com/goccy/go-json/pull/294

    Full Changelog: https://github.com/goccy/go-json/compare/v0.7.9...v0.7.10

    Source code(tar.gz)
    Source code(zip)
  • v0.7.9(Sep 28, 2021)

  • v0.7.8(Sep 1, 2021)

    • Fix mapassign_faststr for indirect struct type ( #283 )
    • Fix encoding of not empty interface type ( #284 )
    • Fix encoding of empty struct interface type ( #286 )
    Source code(tar.gz)
    Source code(zip)
  • v0.7.7(Aug 25, 2021)

  • v0.7.6(Aug 13, 2021)

  • v0.7.5(Aug 12, 2021)

    • Fix encoding of embedded struct with tags ( #265 )
    • Fix encoding of embedded struct that isn't first field ( #272 )
    • Fix decoding of binary type with escaped char ( #273 )
    Source code(tar.gz)
    Source code(zip)
  • v0.7.4(Jul 6, 2021)

  • v0.7.3(Jun 29, 2021)

  • v0.7.2(Jun 26, 2021)

    Fix decoder

    • Add decoder for func type to fix decoding of nil function value ( #257 )
    • Fix stream decoding of []byte type ( #258 )

    Performance

    • Improve decoding performance of map[string]interface{} type ( use mapassign_faststr ) ( #256 )
    • Improve encoding performance of empty interface type ( remove recursive calling of vm.Run ) ( #259 )

    Benchmark

    • Add bytedance/sonic as benchmark target ( #254 )
    Source code(tar.gz)
    Source code(zip)
  • v0.7.1(Jun 17, 2021)

  • v0.7.0(Jun 12, 2021)

    Support context for MarshalJSON and UnmarshalJSON ( #248 )

    • json.MarshalContext(context.Context, interface{}, ...json.EncodeOption) ([]byte, error)
    • json.NewEncoder(io.Writer).EncodeContext(context.Context, interface{}, ...json.EncodeOption) error
    • json.UnmarshalContext(context.Context, []byte, interface{}, ...json.DecodeOption) error
    • json.NewDecoder(io.Reader).DecodeContext(context.Context, interface{}) error
    type MarshalerContext interface {
      MarshalJSON(context.Context) ([]byte, error)
    }
    
    type UnmarshalerContext interface {
      UnmarshalJSON(context.Context, []byte) error
    }
    

    Add DecodeFieldPriorityFirstWin option ( #242 )

    In the default behavior, go-json, like encoding/json, will reflect the result of the last evaluation when a field with the same name exists. I've added new options to allow you to change this behavior. json.DecodeFieldPriorityFirstWin option reflects the result of the first evaluation if a field with the same name exists. This behavior has a performance advantage as it allows the subsequent strings to be skipped if all fields have been evaluated.

    Fix encoder

    • Fix indent number contains recursive type ( #249 )
    • Fix encoding of using empty interface as map key ( #244 )

    Fix decoder

    • Fix decoding fields containing escaped characters ( #237 )

    Refactor

    • Move some tests to subdirectory ( #243 )
    • Refactor package layout for decoder ( #238 )
    Source code(tar.gz)
    Source code(zip)
  • v0.6.1(Jun 2, 2021)

  • v0.6.0(Jun 1, 2021)

    Support Colorize option for encoding (#233)

    b, err := json.MarshalWithOption(v, json.Colorize(json.DefaultColorScheme))
    if err != nil {
      ...
    }
    fmt.Println(string(b)) // print colored json
    

    screenshot

    Refactor

    • Fix opcode layout - Adjust memory layout of the opcode to 128 bytes in a 64-bit environment ( #230 )
    • Refactor encode option ( #231 )
    • Refactor escape string ( #232 )
    Source code(tar.gz)
    Source code(zip)
  • v0.5.1(May 20, 2021)

    Optimization

    • Add type addrShift to enable bigger encoder/decoder cache ( #213 )

    Fix decoder

    • Keep original reference of slice element ( #229 )

    Refactor

    • Refactor Debug mode for encoding ( #226 )
    • Generate VM sources for encoding ( #227 )
    • Refactor validator for null/true/false for decoding ( #221 )
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(May 8, 2021)

    Supports using omitempty and string tags at the same time ( #216 )

    Fix decoder

    • Fix stream decoder for unicode char ( #215 )
    • Fix decoding of slice element ( #219 )
    • Fix calculating of buffer length for stream decoder ( #220 )

    Refactor

    • replace skipWhiteSpace goto by loop ( #212 )
    Source code(tar.gz)
    Source code(zip)
  • v0.4.14(May 3, 2021)

    Benchmark

    • Add valyala/fastjson to benchmark ( #193 )
    • Add benchmark task for CI ( #211 )

    Fix decoder

    • Fix decoding of slice with unmarshal json type ( #198 )
    • Fix decoding of null value for interface type that does not implement Unmarshaler ( #205 )
    • Fix decoding of null value to []byte by json.Unmarshal ( #206 )
    • Fix decoding of backslash char at the end of string ( #207 )
    • Fix stream decoder for null/true/false value ( #208 )
    • Fix stream decoder for slow reader ( #211 )

    Performance

    • If cap of slice is enough, reuse slice data for compatibility with encoding/json ( #200 )
    Source code(tar.gz)
    Source code(zip)
Owner
Masaaki Goshima
Masaaki Goshima
Go encoder and decoder for the NetBPM/PNM image formats. Compatible with Go's image packages.

gpnm This package implements an encoder and decoder for PNM image formats. It can be used with Go's image library. It covers all formats as defined by

null 0 Nov 26, 2021
goi - The “Quite OK Image” format encoder / decoder for Go.

goi - The “Quite OK Image” format encoder / decoder for Go. QOI - The “Quite OK Image” - is losslessly image format that offering speedup both compres

neguse 14 Mar 3, 2022
Decoder/Encoder for GhostControls Gate Remotes

ghostcontrols Decoder/Encoder for GhostControls Gate Remotes GhostControls makes a variety of automatic gate operators, transmitters and keypads & rec

null 1 Sep 5, 2022
A high-performance 100% compatible drop-in replacement of "encoding/json"

A high-performance 100% compatible drop-in replacement of "encoding/json" You can also use thrift like JSON using thrift-iterator Benchmark Source cod

Jsoniter 11.3k Sep 17, 2022
Fast and flexible JSON encoder for Go

Jettison Jettison is a fast and flexible JSON encoder for the Go programming language, inspired by bet365/jingo, with a richer features set, aiming at

William Poussier 135 Sep 15, 2022
Package json implements encoding and decoding of JSON as defined in RFC 7159

Package json implements encoding and decoding of JSON as defined in RFC 7159. The mapping between JSON and Go values is described in the documentation for the Marshal and Unmarshal functions

High Performance, Kubernetes Native Object Storage 2 Jun 26, 2022
Fork of Go's standard library json encoder

A fork of the Go standard library's json encoder Why? https://github.com/golang/go/issues/6213 was proposed in 2013 but was never accepted. Difference

unchain.io 0 Nov 25, 2021
An efficient JSON decoder

pkg/json An alternative JSON decoder for Go. Features pkg/json aims to be a drop in replacement for encoding/json. It features: json.Scanner which, wh

null 288 Sep 8, 2022
Golang JSON decoder supporting case-sensitive, number-preserving, and strict decoding use cases

Golang JSON decoder supporting case-sensitive, number-preserving, and strict decoding use cases

Kubernetes SIGs 14 Sep 11, 2022
json encoding and decoding

jx Package jx implements encoding and decoding of json [RFC 7159]. Lightweight fork of jsoniter. go get github.com/go-faster/jx Usage and examples Roa

go faster 68 Sep 15, 2022
JSON:API compatible query string parser

QParser The package helps to parse part of the URL path and its parameters string to a handy structure. The structure format is compatible with the JS

Velmie 1 Dec 21, 2021
JSON Spanner - A Go package that provides a fast and simple way to filter or transform a json document

JSON SPANNER JSON Spanner is a Go package that provides a fast and simple way to

null 6 Sep 14, 2022
Fast JSON parser and validator for Go. No custom structs, no code generation, no reflection

fastjson - fast JSON parser and validator for Go Features Fast. As usual, up to 15x faster than the standard encoding/json. See benchmarks. Parses arb

Aliaksandr Valialkin 1.7k Sep 16, 2022
Fast JSON serializer for golang.

easyjson Package easyjson provides a fast and easy way to marshal/unmarshal Go structs to/from JSON without the use of reflection. In performance test

Free and open source software developed at Mail.Ru 3.9k Sep 17, 2022
Fast Color JSON Marshaller + Pretty Printer for Golang

ColorJSON: The Fast Color JSON Marshaller for Go What is this? This package is based heavily on hokaccha/go-prettyjson but has some noticible differen

Tyler Brock 115 Sep 15, 2022
A fast json parser for go

rjson rjson is a json parser that relies on Ragel-generated state machines for most parsing. rjson's api is minimal and focussed on efficient parsing.

WillAbides 49 Aug 28, 2022
A blazingly fast JSON serializing & deserializing library

Sonic A blazingly fast JSON serializing & deserializing library, accelerated by JIT(just-in-time compiling) and SIMD(single-instruction-multi-data). B

Bytedance Inc. 3.5k Sep 20, 2022
Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang

kazaam Description Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang. This functionality provides

Qntfy 205 Sep 17, 2021
Fast json for go. Lightweight fork of jsoniter.

jx Fast json for go. Lightweight fork of jsoniter. Features Reduced scope (no reflection or encoding/json adapter) Fuzzing, improved test coverage Dra

ogen 68 Sep 15, 2022