Golang JSON decoder supporting case-sensitive, number-preserving, and strict decoding use cases

Overview

sigs.k8s.io/json

Go Reference

Introduction

This library is a subproject of sig-api-machinery. It provides case-sensitive, integer-preserving JSON unmarshaling functions based on encoding/json Unmarshal().

Compatibility

The UnmarshalCaseSensitivePreserveInts() function behaves like encoding/json#Unmarshal() with the following differences:

  • JSON object keys are treated case-sensitively. Object keys must exactly match json tag names (for tagged struct fields) or struct field names (for untagged struct fields).
  • JSON integers are unmarshaled into interface{} fields as an int64 instead of a float64 when possible, falling back to float64 on any parse or overflow error.
  • Syntax errors do not return an encoding/json *SyntaxError error. Instead, they return an error which can be passed to SyntaxErrorOffset() to obtain an offset.

Additional capabilities

The UnmarshalStrict() function decodes identically to UnmarshalCaseSensitivePreserveInts(), and also returns non-fatal strict errors encountered while decoding:

  • Duplicate fields encountered
  • Unknown fields encountered

Community, discussion, contribution, and support

You can reach the maintainers of this project via the sig-api-machinery mailing list / channels.

Code of conduct

Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.

Issues
  • Please use a regular Apache 2.0 LICENSE

    Please use a regular Apache 2.0 LICENSE

    Adding exceptions to the LICENSE file makes it difficult for consumers to determine whether this component can be considered released under Apache 2.0. The assumption then becomes that it isn't, which probably wasn't the intention when adding the exception?

    opened by thallgren 5
  • Output field path in strict errors

    Output field path in strict errors

    Fixes https://github.com/kubernetes-sigs/json/issues/4

    When decoding strictly, includes path context in errors for unknown and duplicate fields.

    This is needed for parity with the client-side validation this is replacing.

    benchmark                                       old allocs     new allocs     delta
    BenchmarkUnmarshal/typed_stdlib-12              161            161            +0.00%
    BenchmarkUnmarshal/typed_unmarshal-12           161            161            +0.00%
    BenchmarkUnmarshal/typed_strict-12              167            171            +2.40%
    BenchmarkUnmarshal/typed_strict-custom-12       168            172            +2.38%
    BenchmarkUnmarshal/untyped_stdlib-12            517            517            +0.00%
    BenchmarkUnmarshal/untyped_unmarshal-12         513            521            +1.56%
    BenchmarkUnmarshal/untyped_strict-12            523            535            +2.29%
    BenchmarkUnmarshal/untyped_strict-custom-12     524            536            +2.29%
    
    benchmark                                       old bytes     new bytes     delta
    BenchmarkUnmarshal/typed_stdlib-12              13240         13240         +0.00%
    BenchmarkUnmarshal/typed_unmarshal-12           13272         13304         +0.24%
    BenchmarkUnmarshal/typed_strict-12              13784         14168         +2.79%
    BenchmarkUnmarshal/typed_strict-custom-12       13816         14200         +2.78%
    BenchmarkUnmarshal/untyped_stdlib-12            28213         28203         -0.04%
    BenchmarkUnmarshal/untyped_unmarshal-12         28190         28252         +0.22%
    BenchmarkUnmarshal/untyped_strict-12            28411         28849         +1.54%
    BenchmarkUnmarshal/untyped_strict-custom-12     28448         28880         +1.52%
    

    running benchmarks with bigger/deeper objects (e.g. medium-size serialized pods):

    benchmark                                      old allocs     new allocs     delta
    BenchmarkUnmarshalTyped/std-unmarshal-12       161            161            +0.00%
    BenchmarkUnmarshalTyped/std-decode-12          171            171            +0.00%
    BenchmarkUnmarshalTyped/utiljson-12            161            161            +0.00%
    BenchmarkUnmarshalTyped/kjson-12               161            164            +1.86%
    BenchmarkUnmarshalTyped/kjson-strict-12        168            176            +4.76%
    BenchmarkUnmarshalUntyped/std-unmarshal-12     521            521            +0.00%
    BenchmarkUnmarshalUntyped/std-decode-12        531            531            +0.00%
    BenchmarkUnmarshalUntyped/utiljson-12          536            536            +0.00%
    BenchmarkUnmarshalUntyped/kjson-12             519            524            +0.96%
    BenchmarkUnmarshalUntyped/kjson-strict-12      526            537            +2.09%
    
    benchmark                                      old bytes     new bytes     delta
    BenchmarkUnmarshalTyped/std-unmarshal-12       12672         12672         +0.00%
    BenchmarkUnmarshalTyped/std-decode-12          27472         27472         +0.00%
    BenchmarkUnmarshalTyped/utiljson-12            12672         12672         +0.00%
    BenchmarkUnmarshalTyped/kjson-12               12704         12760         +0.44%
    BenchmarkUnmarshalTyped/kjson-strict-12        13112         13664         +4.21%
    BenchmarkUnmarshalUntyped/std-unmarshal-12     27033         27032         -0.00%
    BenchmarkUnmarshalUntyped/std-decode-12        41847         41850         +0.01%
    BenchmarkUnmarshalUntyped/utiljson-12          41945         41944         -0.00%
    BenchmarkUnmarshalUntyped/kjson-12             27064         27143         +0.29%
    BenchmarkUnmarshalUntyped/kjson-strict-12      27303         28375         +3.93%
    
    approved cncf-cla: yes lgtm size/L 
    opened by liggitt 5
  • Populate sigs.k8s.io/json

    Populate sigs.k8s.io/json

    Best reviewed commit by commit

    • First few commits update template content, initialize the module, and set up CI (example run at https://github.com/liggitt/json/runs/3842971665) to run unit tests, vet/unsafe/dependency checks, and catch go API incompatibilities
    • The next set of commits copy the stdlib encoding/json package to an internal package, get the existing unit tests running, and alias the types to be compatible with encoding/json
    • The next set make tweaks to
      • allow setting decoder options when unmarshaling
      • allow accumulating/distinguishing strict errors while maintaining identical decoding behavior
      • add case-sensitive decoder option
      • add int preserving decoder option
      • add disallow duplicate fields decoder option
    • The final commit exposes UnmarshalCaseSensitivePreserveInts, UnmarshalStrict, and NewDecoderCaseSensitivePreserveInts functions

    benchmarks vs stdlib with strict error checking additions:

    BenchmarkUnmarshal/typed_stdlib-12            25560     45859 ns/op   13240 B/op     161 allocs/op
    BenchmarkUnmarshal/typed_unmarshal-12         26212     45366 ns/op   13272 B/op     161 allocs/op
    BenchmarkUnmarshal/typed_strict-12            25543     47007 ns/op   13784 B/op     167 allocs/op
    
    BenchmarkUnmarshal/untyped_stdlib-12          23452     50936 ns/op   28214 B/op     517 allocs/op
    BenchmarkUnmarshal/untyped_unmarshal-12       23703     51473 ns/op   28186 B/op     513 allocs/op
    BenchmarkUnmarshal/untyped_strict-12          21214     55943 ns/op   28412 B/op     523 allocs/op
    
    approved cncf-cla: yes lgtm size/XXL 
    opened by liggitt 5
  • Project support for Go 1.16

    Project support for Go 1.16

    Currently, this project is advertised as supporting Go 1.16 in the project go.mod. However, the project uses functionality introduced in Go 1.17 (1, 2) that does not exist in Go 1.16. This is resulting in downstream packages breaking when trying to build with Go 1.16 and the Go module system not detecting the incompatibility.

    Ideally this project would support Go 1.16 as it is still a supported version by the Go language maintainers. If that is not possible, it makes sense to bump the minimum version of Go supported in the main go.mod to allow these errors to be caught by go tooling.

    opened by MrAlias 4
  • Create a SECURITY_CONTACTS file.

    Create a SECURITY_CONTACTS file.

    As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS file.

    The template for the file can be found in the kubernetes-template repository[2]. A description for the file is in the steering-committee docs[3], you might need to search that page for "Security Contacts".

    Please feel free to ping me on the PR when you make it, otherwise I will see when you close this issue. :)

    Thanks so much, let me know if you have any questions.

    (This issue was generated from a tool, apologies for any weirdness.)

    [1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE [2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS [3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md

    opened by k8s-triage-robot 3
  • Create a SECURITY_CONTACTS file.

    Create a SECURITY_CONTACTS file.

    As per the email sent to kubernetes-dev[1], please create a SECURITY_CONTACTS file.

    The template for the file can be found in the kubernetes-template repository[2]. A description for the file is in the steering-committee docs[3], you might need to search that page for "Security Contacts".

    Please feel free to ping me on the PR when you make it, otherwise I will see when you close this issue. :)

    Thanks so much, let me know if you have any questions.

    (This issue was generated from a tool, apologies for any weirdness.)

    [1] https://groups.google.com/forum/#!topic/kubernetes-dev/codeiIoQ6QE [2] https://github.com/kubernetes/kubernetes-template-project/blob/master/SECURITY_CONTACTS [3] https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-governance-template-short.md

    opened by k8s-triage-robot 2
  • Provide the location of unmarshal errors within the JSON object

    Provide the location of unmarshal errors within the JSON object

    When an unmarshal fails, such as when disallowUnknownFields is set, it returns an error such as json: unknown field "foo".

    We would like some sort of context on where within the JSON object the unknown field exists. There are many instances where the same field name exists multiple times in an object and so without that context it can be hard to tell where the error came from. For example, large CRDs can have many “description” fields. If one of those “descriptions” is misplaced within the object it will be really difficult to track it down.

    Simple playground example: https://play.golang.org/p/je8-WmSD1vE

    opened by kevindelgado 1
  • sync go1.18 changes from encoding/json

    sync go1.18 changes from encoding/json

    git log --format=oneline upstream/release-branch.go1.17..upstream/release-branch.go1.18 -- src/encoding/json/
    
    approved size/XL cncf-cla: yes 
    opened by liggitt 2
Owner
Kubernetes SIGs
Org for Kubernetes SIG-related work
Kubernetes SIGs
COBS implementation in Go (Decoder) and C (Encoder & Decoder) with tests.

COBS Table of Contents About The project COBS Specification Getting Started 3.1. Prerequisites 3.2. Installation 3.3. Roadmap Contributing License Con

Thomas Höhenleitner 2 May 22, 2022
Package json implements encoding and decoding of JSON as defined in RFC 7159

Package json implements encoding and decoding of JSON as defined in RFC 7159. The mapping between JSON and Go values is described in the documentation for the Marshal and Unmarshal functions

High Performance, Kubernetes Native Object Storage 3 May 10, 2022
Fast JSON encoder/decoder compatible with encoding/json for Go

Fast JSON encoder/decoder compatible with encoding/json for Go

Masaaki Goshima 1.6k Jun 26, 2022
Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang

kazaam Description Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang. This functionality provides

Qntfy 205 Sep 17, 2021
json encoding and decoding

jx Package jx implements encoding and decoding of json [RFC 7159]. Lightweight fork of jsoniter. go get github.com/go-faster/jx Usage and examples Roa

go faster 59 Jun 19, 2022
An efficient JSON decoder

pkg/json An alternative JSON decoder for Go. Features pkg/json aims to be a drop in replacement for encoding/json. It features: json.Scanner which, wh

null 286 Jun 6, 2022
Go encoder and decoder for the NetBPM/PNM image formats. Compatible with Go's image packages.

gpnm This package implements an encoder and decoder for PNM image formats. It can be used with Go's image library. It covers all formats as defined by

null 0 Nov 26, 2021
goi - The “Quite OK Image” format encoder / decoder for Go.

goi - The “Quite OK Image” format encoder / decoder for Go. QOI - The “Quite OK Image” - is losslessly image format that offering speedup both compres

neguse 14 Mar 3, 2022
Decoder/Encoder for GhostControls Gate Remotes

ghostcontrols Decoder/Encoder for GhostControls Gate Remotes GhostControls makes a variety of automatic gate operators, transmitters and keypads & rec

null 0 Jan 1, 2022
Json-go - CLI to convert JSON to go and vice versa

Json To Go Struct CLI Install Go version 1.17 go install github.com/samit22/js

Samit Ghimire 5 Mar 3, 2022
JSON Spanner - A Go package that provides a fast and simple way to filter or transform a json document

JSON SPANNER JSON Spanner is a Go package that provides a fast and simple way to

null 2 May 20, 2022
Get JSON values quickly - JSON parser for Go

get json values quickly GJSON is a Go package that provides a fast and simple way to get values from a json document. It has features such as one line

Josh Baker 10.5k Jun 30, 2022
JSON diff library for Go based on RFC6902 (JSON Patch)

jsondiff jsondiff is a Go package for computing the diff between two JSON documents as a series of RFC6902 (JSON Patch) operations, which is particula

William Poussier 172 Jun 23, 2022
Abstract JSON for golang with JSONPath support

Abstract JSON Abstract JSON is a small golang package provides a parser for JSON with support of JSONPath, in case when you are not sure in its struct

Stepan Pyzhov 119 Jun 24, 2022
JSON query in Golang

gojq JSON query in Golang. Install go get -u github.com/elgs/gojq This library serves three purposes: makes parsing JSON configuration file much easie

Qian Chen 182 Apr 27, 2022
Automatically generate Go (golang) struct definitions from example JSON

gojson gojson generates go struct definitions from json or yaml documents. Example $ curl -s https://api.github.com/repos/chimeracoder/gojson | gojson

Aditya Mukerjee 2.5k Jun 23, 2022
Arbitrary transformations of JSON in Golang

kazaam Description Kazaam was created with the goal of supporting easy and fast transformations of JSON data with Golang. This functionality provides

Qntfy 230 Jun 17, 2022
Parsing JSON is a hassle in golang

GoJSON Parsing JSON is a hassle in golang. This package will allow you to parse and search elements in a json without structs. Install gojson go get g

swaraj18 25 Nov 12, 2021
Fast JSON serializer for golang.

easyjson Package easyjson provides a fast and easy way to marshal/unmarshal Go structs to/from JSON without the use of reflection. In performance test

Free and open source software developed at Mail.Ru 3.8k Jun 22, 2022