JSON diff library for Go based on RFC6902 (JSON Patch)

Overview

jsondiff


jsondiff is a Go package for computing the diff between two JSON documents as a series of RFC6902 (JSON Patch) operations, which is particularly suitable to create the patch response of a Kubernetes Mutating Webhook for example.


Usage

First, get the latest version of the library using the following command:

$ go get github.com/wI2L/[email protected]

⚠️ Requires Go1.14+, due to the usage of the package hash/maphash.

Example use cases

Kubernetes Dynamic Admission Controller

The typical use case within an application is to compare two values of the same type that represents the source and desired target of a JSON document. A concrete application of that would be to generate the patch returned by a Kubernetes dynamic admission controller to mutate a resource. Thereby, instead of generating the operations, just copy the source in order to apply the required changes and delegate the patch generation to the library.

For example, given the following corev1.Pod value that represents a Kubernetes demo pod containing a single container:

import corev1 "k8s.io/api/core/v1"

pod := corev1.Pod{
    Spec: corev1.PodSpec{
        Containers: []corev1.Container{{
            Name:  "webserver",
            Image: "nginx:latest",
            VolumeMounts: []corev1.VolumeMount{{
                Name:      "shared-data",
                MountPath: "/usr/share/nginx/html",
            }},
        }},
        Volumes: []corev1.Volume{{
            Name: "shared-data",
            VolumeSource: corev1.VolumeSource{
                EmptyDir: &corev1.EmptyDirVolumeSource{
                    Medium: corev1.StorageMediumMemory,
                },
            },
        }},
    },
}

The first step is to copy the original pod value. The corev1.Pod type defines a DeepCopy method, which is handy, but for other types, a shallow copy is discouraged, instead use a specific library, such as ulule/deepcopier. Alternatively, if you don't require to keep the original value, you can marshal it to JSON using json.Marshal to store a pre-encoded copy of the document, and mutate the value.

newPod := pod.DeepCopy()
// or
podBytes, err := json.Marshal(pod)
if err != nil {
    // handle error
}

Secondly, make some changes to the pod spec. Here we modify the image and the storage medium used by the pod's volume shared-data.

// Update the image of the webserver container.
newPod.Spec.Containers[0].Image = "nginx:1.19.5-alpine"

// Switch storage medium from memory to default.
newPod.Spec.Volumes[0].EmptyDir.Medium = corev1.StorageMediumDefault

Finally, generate the patch that represents the changes relative to the original value. Note that when the Compare or CompareOpts functions are used, the source and target parameters are first marshaled using the encoding/json package in order to obtain their final JSON representation, prior to comparing them.

import "github.com/wI2L/jsondiff"

patch, err := jsondiff.Compare(pod, newPod)
if err != nil {
    // handle error
}
b, err := json.MarshalIndent(patch, "", "    ")
if err != nil {
    // handle error
}
os.Stdout.Write(b)

The output is similar to the following:

[{
    "op": "replace",
    "path": "/spec/containers/0/image",
    "value": "nginx:1.19.5-alpine"
}, {
    "op": "remove",
    "path": "/spec/volumes/0/emptyDir/medium"
}]

The JSON patch can then be used in the response payload of you Kubernetes webhook.

Optional fields gotcha

Note that the above example is used for simplicity, but in a real-world admission controller, you should create the diff from the raw bytes of the AdmissionReview.AdmissionRequest.Object.Raw field. As pointed out by user /u/terinjokes on Reddit, due to the nature of Go structs, the "hydrated" corev1.Pod object may contain "optional fields", resulting in a patch that state added/changed values that the Kubernetes API server doesn't know about. Below is a quote of the original comment:

Optional fields being ones that are a struct type, but are not pointers to those structs. These will exist when you unmarshal from JSON, because of how Go structs work, but are not in the original JSON. Comparing between the unmarshaled and copied versions can generate add and change patches below a path not in the original JSON, and the API server will reject your patch.

A realistic usage would be similar to the following snippet:

podBytes, err := json.Marshal(pod)
if err != nil {
    // handle error
}
// req is a k8s.io/api/admission/v1.AdmissionRequest object
jsondiff.CompareJSON(req.AdmissionRequest.Object.Raw, podBytes)

Mutating the original pod object or a copy is up to you, as long as you use the raw bytes of the AdmissionReview object to generate the patch.

You can find a detailed description of that problem and its resolution in this GitHub issue.

Outdated package version

There's also one other downside to the above example. If your webhook does not have the latest version of the client-go package, or whatever package that contains the types for the resource your manipulating, all fields not known in that version will be deleted.

For example, if your webhook mutate Service resources, a user could set the field .spec.allocateLoadBalancerNodePort in Kubernetes 1.20 to disable allocating a node port for services with Type=LoadBalancer. However, if the webhook is still using the v1.19.x version of the k8s.io/api/core/v1 package that define the Service type, instead of simply ignoring this field, a remove operation will be generated for it.

Diff options

If more control over the diff behaviour is required, use the CompareOpts or CompareJSONOpts function instead. The third parameter is variadic and accept a list of functional opt-in options described below.

Note that any combination of options can be used without issues.

Operations factorization

By default, when computing the difference between two JSON documents, the package does not produce move or copy operations. To enable the factorization of value removals and additions as moves and copies, you should use the functional option Factorize(). Factorization reduces the number of operations generated, which inevitably reduce the size of the patch once it is marshaled as JSON.

For instance, given the following document:

{
    "a": [1, 2, 3],
    "b": { "foo": "bar" }
}

In order to obtain this updated version:

{
    "a": [1, 2, 3],
    "c": [1, 2, 3],
    "d": { "foo": "bar" }
}

The package generates the following patch:

[
    { "op": "remove", "path": "/b" },
    { "op": "add", "path": "/c", "value": [1, 2, 3] },
    { "op": "add", "path": "/d", "value": { "foo": "bar" } }
]

If we take the previous example and generate the patch with factorization enabled, we then get a different patch, containing copy and move operations instead:

[
    { "op": "copy", "from": "/a", "path": "/c" },
    { "op": "move", "from": "/b", "path": "/d" }
]

Operations rationalization

The default method used to compare two JSON documents is a recursive comparison. This produce one or more operations for each difference found. On the other hand, in certain situations, it might be beneficial to replace a set of operations representing several changes inside a JSON node by a single replace operation targeting the parent node.

For that purpose, you can use the Rationalize() option. It uses a simple weight function to decide which patch is best (it marshals both sets of operations to JSON and looks at the length of bytes to keep the smaller footprint).

Let's illustrate that with the following document:

{
    "a": { "b": { "c": { "1": 1, "2": 2, "3": 3 } } }
}

In order to obtain this updated version:

{
    "a": { "b": { "c": { "x": 1, "y": 2, "z": 3 } } }
}

The expected output is one remove/add operation combo for each children field of the object located at path a.b.c:

[
    { "op": "remove", "path": "/a/b/c/1" },
    { "op": "remove", "path": "/a/b/c/2" },
    { "op": "remove", "path": "/a/b/c/3" },
    { "op": "add", "path": "/a/b/c/x", "value": 1 },
    { "op": "add", "path": "/a/b/c/y", "value": 2 },
    { "op": "add", "path": "/a/b/c/z", "value": 3 }
]

If we also enable factorization, as seen above, we can reduce the number of operations by half:

[
    { "op": "move", "from": "/a/b/c/1", "path": "/a/b/c/x" },
    { "op": "move", "from": "/a/b/c/2", "path": "/a/b/c/y" },
    { "op": "move", "from": "/a/b/c/3", "path": "/a/b/c/z" }
]

And finally, with rationalization enabled, those operations are replaced with a single replace of the parent object:

[
    { "op": "replace", "path": "/a/b/c", "value": { "x": 1, "y": 2, "z": 3 } }
]

Invertible patch

Using the functional option Invertible(), it is possible to instruct the diff generator to precede each remove and replace operation with a test operation. Such patches can be inverted to return a patched document to its original form.

However, note that it comes with one limitation. copy operations cannot be inverted, as they are ambiguous (the reverse of a copy is a remove, which could then become either an add or a copy). As such, using this option disable the generation of copy operations (if option Factorize() is used) and replace them with add, albeit potentially at the cost of increased patch size.

For example, let's generate the diff between those two JSON documents:

{
    "a": "1",
    "b": "2"
}
{
    "a": "3",
    "c": "4"
}

The patch is similar to the following:

[
    { "op": "test", "path": "/a", "value": "1" },
    { "op": "replace", "path": "/a", "value": "3" },
    { "op": "test", "path": "/b", "value": "2" },
    { "op": "remove", "path": "/b" },
    { "op": "add", "path": "/c", "value": "4" }
]

As you can see, the remove and replace operations are preceded with a test operation which assert/verify the value of the previous path. On the other hand, the add operation can be reverted to a remove operation directly and doesn't need to be preceded by a test.

Finally, as a side example, if we were to use the Rationalize() option in the context of the previous example, the output would be shorter, but the generated patch would still remain invertible:

[
    { "op": "test", "path": "", "value": { "a": "1", "b": "2" } },
    { "op": "replace", "path": "", "value": { "a": "3", "c": "4" } }
]

Benchmarks

Performance is not the primary target of the package, instead it strives for correctness. A simple benchmark that compare the performance of available options is provided to give a rough estimate of the cost of each option. You can find the JSON documents used by this benchmark in the directory testdata/bench.

If you'd like to run the benchmark yourself, use the following command:

go get github.com/cespare/prettybench
go test -bench=. | prettybench

Results

The benchmark was run 10x (statistics computed with benchstat) on a MacBook Pro 13", with the following specs:

OS : macOS Catalina (10.15.7)
CPU: 1.4 GHz Intel Core i5
Mem: 16GB 2133 MHz
Go : go version go1.15.5 darwin/amd64
Tag: v0.1.0
Output
name                            time/op
CompareJSONOpts/default-8       26.5µs ± 1%
CompareJSONOpts/invertible-8    27.4µs ± 1%
CompareJSONOpts/factorize-8     31.6µs ± 1%
CompareJSONOpts/rationalize-8   90.3µs ± 0%
CompareJSONOpts/factor+ratio-8  95.5µs ± 1%
CompareJSONOpts/all-options-8    126µs ± 1%

Credits

This package has been inspired by existing implementations of JSON Patch for various languages:

License

jsondiff is licensed under the MIT license. See the LICENSE file.

Issues
  • [Feature] An option to avoid generation of diff in re-ordered Slices?

    [Feature] An option to avoid generation of diff in re-ordered Slices?

    The jsondiff library works well for us in terms of diff generations between two structs. Our problem starts to appear when the changes are made in a slice field of a struct that is they are re-ordered or any element is added/removed in the start/middle of the slice, this causes all of the elements to be shifted which is desired use-case as per the RFC-6902 and so the library generates a series of diff's. This causes problems in our use case as we don't really care about ordering and so the question:

    • Is there an option/flag to avoid the generation of several diff's in case of re-ordering or addition/removal of elements in the start/middle of a slice? or we can override this behavior of index-based comparison to content-based comparison in slices?
    enhancement 
    opened by prashjai94 11
  • I am looking for a Stronger jsonPatch apply lib

    I am looking for a Stronger jsonPatch apply lib

    usefor Kubernetes Dynamic Admission Controller

    package main
    
    import (
    	"encoding/json"
    	"fmt"
    
    	jsonpatch "github.com/xxxx/json-patch"
    	corev1 "k8s.io/api/core/v1"
    
    )
    
    func main()  {
    
    	var pod = corev1.Pod{
    		Spec:       corev1.PodSpec{
    			Volumes: []corev1.Volume{
    				{
    					Name: "default-token-lsh6v",
    					VolumeSource: corev1.VolumeSource{
    						Secret: &corev1.SecretVolumeSource{
    							SecretName: "default-token-lsh6v",
    						},
    					},
    				},
    			},
    
    		},
    	}
    
    	podBytes, _ := json.Marshal(&pod)
    
    	fmt.Printf("old document: %s\n", podBytes)
    
    	patchJSON := []byte(`[
       {
          "op":"add",
          "path":"/spec",
          "value":{
             "dnsConfig":{
                "options":[
                   {
                      "name":"single-request-reopen"
                   }
                ]
             }
          }
       },
       {
          "op":"add",
          "path":"/spec/volumes/-",
          "value":{
             "name":"default-token-lsh6v1111",
             "secret":{
                "secretName":"default-token-lsh6v1111"
             }
          }
       }
    ]`)
    	patch, err := jsonpatch.DecodePatch(patchJSON)
    	if err != nil {
    		panic(err)
    	}
    	modified, err := patch.Apply(podBytes)
    	if err != nil {
    		panic(err)
    	}
    	fmt.Printf("Modified document: %s\n", modified)
    
    }
    

    expect result json is:

    {
       "metadata":{
          "creationTimestamp":null
       },
       "spec":{
          "dnsConfig":{
             "options":[
                {
                   "name":"single-request-reopen"
                }
             ]
          },
          "containers":null,
          "volumes":[
             {
                "name":"default-token-lsh6v",
                "secret":{
                   "secretName":"default-token-lsh6v"
                }
             },
             {
                "name":"default-token-lsh6v1111",
                "secret":{
                   "secretName":"default-token-lsh6v1111"
                }
             }
          ]
       }
    }
    

    also, I expect:

       {
          "op":"append",
          "path":"/spec/containers/*/env",
          "value":{
             "name":"KEY",
             "value": "VALUE"
          }
       }
    

    this will append a env to all containers ,not override

    opened by zhangguanzhang 8
  • the diff for arrays does not reference array indexes

    the diff for arrays does not reference array indexes

    the resultant json-patch document after comparing two JSONs does not have the right patch operations for arrays based on indexes.

    the json-patch for an array should look like this { "op": "replace", "path": "/options/1/value", "value": "soln-v1 1" }, { "op": "replace", "path": "/options/1/label", "value": "item 1_1" }, { "op": "add", "path": "/options/2", "value": { "label": "item 1_2", "value": "soln-v1 2" } }

    I have an array containing in the source JSON {"options": [ {"label": "item 1", "value": "item 1"},{"label": "item 2", "value": "item 2"}]}

    and an modified array in the target JSON {"options": [ {"label": "item 1", "value": "item 1"},{"label": "item 1_1", "value": "soln-v1 1"},{"label": "item 1_2", "value": "soln-v1 2"}]}

    the output should look like the example provided

    and not a replace of the entire source array to the target array.

    HTH.

    NOTE: This is for the CompareJSON() method. The Compare() method works very well.

    Thanks and Regards, Srinivas

    question 
    opened by srinivasj-sys 3
  • adding Apply methods?

    adding Apply methods?

    It appears that https://github.com/evanphx/json-patch doesn't support the patches you are generating. I really like the terse results of your patches and have validated that they are valid. Given you are already parsing and looking for deltas, would it be possible to make some Apply and Revert functions in addition to patch generation?

    opened by delaneyj 3
  • [feature request]: Show old value for replace operations

    [feature request]: Show old value for replace operations

    Factorize() only shows from on move and copy operations. In our use case we are not generating the diffs for use with a PATCH but to show changes. We'd like to be able to generate the diff so that it shows both the old and new values of the changed fields.

    We tried using Factorize() but that doesn't change the output for replace operations. It appears the append method explicitly sets the from value to emptyPtr. It looks like there may be a few other places that changes would be needed to set the old value.

    Ideally we'd like to see a replace able to be renders as

    {"op":"replace","from":"oldValue","path":"/key","value":"newValue"}
    
    opened by tjdavis3 1
Releases(v0.2.0)
  • v0.2.0(Apr 4, 2022)

    This release brings various performance improvements for all options. The Differ type is also now exported and provide a Reset method, to allow reuse of underlying storage.

    A playground is also now available, if you want to try the package directly in your browser.

    Changelog

    • 55ad2c4 docs: add badge for playground in README header
    • 213ca8c refactor: export Differ type to allow reuse of resources
    • c04fcf3 feat: add new Equivalent option
    • 291eb7c ci: add go1.18 to tests matrix
    • 344b4be docs: update benchmark results with latest improvements
    • 7ba8329 perf: various performance improvements for all options
    • 5aa8829 test: add benchmark run for differ.diff method only
    • b393df1 test: benchmark CompareJSON against Compare
    • fdd6214 chore: add goreleaser Github workflow

    Commits List: https://github.com/wI2L/jsondiff/compare/v0.1.1...v0.2.0

    Source code(tar.gz)
    Source code(zip)
  • v0.1.1(Aug 30, 2021)

    Changelog

    1d70c73 chore: cleanup module dependencies and update CI workflow 56b34ca docs: describe gotchas of Kubernetes admission controller example 0c0c14c docs: add comment to patch operation constants block

    Source code(tar.gz)
    Source code(zip)
/ˈdʏf/ - diff tool for YAML files, and sometimes JSON

dyff is inspired by the way the old BOSH v1 deployment output reported changes from one version to another by only showing the parts of a YAML file that change.

null 638 Jun 23, 2022
Get JSON values quickly - JSON parser for Go

get json values quickly GJSON is a Go package that provides a fast and simple way to get values from a json document. It has features such as one line

Josh Baker 10.5k Jun 30, 2022
Fast JSON encoder/decoder compatible with encoding/json for Go

Fast JSON encoder/decoder compatible with encoding/json for Go

Masaaki Goshima 1.6k Jun 26, 2022
Package json implements encoding and decoding of JSON as defined in RFC 7159

Package json implements encoding and decoding of JSON as defined in RFC 7159. The mapping between JSON and Go values is described in the documentation for the Marshal and Unmarshal functions

High Performance, Kubernetes Native Object Storage 3 May 10, 2022
Json-go - CLI to convert JSON to go and vice versa

Json To Go Struct CLI Install Go version 1.17 go install github.com/samit22/js

Samit Ghimire 5 Mar 3, 2022
JSON Spanner - A Go package that provides a fast and simple way to filter or transform a json document

JSON SPANNER JSON Spanner is a Go package that provides a fast and simple way to

null 2 May 20, 2022
Simple json based db, if you call db xd.

Golang-json-db Simple json based db, if you call db xd. Basics //First of all you have to config main json file then config backup json file. Backup i

null 3 Sep 27, 2021
Senml-go - a Golang module for the JSON-based SenML sensor data format

ThingWave SenML module for Golang This is a Golang module for the JSON-based Sen

ThingWave 0 Jan 2, 2022
library for working amorphous data (as when you decode json into an interface{})

Introduction Decoding json into an interface{} produces an hierarchical arrangement of four data types: float64, string are 'primative types' and form

Chuck Luciano 9 Jul 5, 2021
A blazingly fast JSON serializing & deserializing library

Sonic A blazingly fast JSON serializing & deserializing library, accelerated by JIT(just-in-time compiling) and SIMD(single-instruction-multi-data). B

Bytedance Inc. 3k Jun 24, 2022
A library to query the godoc.org JSON API.

gopkg This repository provides minimal Go package that makes queries against the godoc.org JSON API. Since that site has mostly been subsumed by pkg.g

M. J. Fromberger 2 Nov 26, 2021
Copy of Golang's json library with IsZero feature

json Copy of Golang's json library with IsZero feature from CL13977 Disclaimer It is a package primary used for my own projects, I will keep it up-to-

Ferenc Fabian 4 Oct 9, 2021
Fork of Go's standard library json encoder

A fork of the Go standard library's json encoder Why? https://github.com/golang/go/issues/6213 was proposed in 2013 but was never accepted. Difference

unchain.io 0 Nov 25, 2021
JSON Unmarshalling Library

JSAWN (JAY-sawn) This is a JSON library to add to the capabilities of the standard 'encoding/json' library. Unmarshalling The first enhancement is to

null 1 Feb 16, 2022
Abstract JSON for golang with JSONPath support

Abstract JSON Abstract JSON is a small golang package provides a parser for JSON with support of JSONPath, in case when you are not sure in its struct

Stepan Pyzhov 119 Jun 24, 2022
Fast JSON parser and validator for Go. No custom structs, no code generation, no reflection

fastjson - fast JSON parser and validator for Go Features Fast. As usual, up to 15x faster than the standard encoding/json. See benchmarks. Parses arb

Aliaksandr Valialkin 1.6k Jun 21, 2022
Small utility to create JSON objects

gjo Small utility to create JSON objects. This was inspired by jpmens/jo. Support OS Mac Linux Windows Requirements Go 1.1.14~ Git Installtion Build $

skanehira 108 Apr 27, 2022
A Go package for handling common HTTP JSON responses.

go-respond A Go package for handling common HTTP JSON responses. Installation go get github.com/nicklaw5/go-respond Usage The goal of go-respond is to

Nick Law 47 May 18, 2022
JSON query in Golang

gojq JSON query in Golang. Install go get -u github.com/elgs/gojq This library serves three purposes: makes parsing JSON configuration file much easie

Qian Chen 182 Apr 27, 2022