/ˈdʏf/ - diff tool for YAML files, and sometimes JSON

Overview

δyƒƒ /ˈdʏf/ License Go Report Card Build and Tests Codecov Go Reference Release

dyff

A diff tool for YAML files, and sometimes JSON

Description

dyff is inspired by the way the old BOSH v1 deployment output reported changes from one version to another by only showing the parts of a YAML file that change.

Each difference is referenced by its location in the YAML document by using either the Spruce or go-patch path syntax. The output report aims to be as compact as possible to give a clear and simple overview of the change.

Similar to the standard diff tool, it follows the principle of describing the change by going from the from input file to the target to input file.

Input files can be local files (filesystem path), remote files (URI), or the standard input stream (using -).

All orders of keys in hashes are preserved during processing and output to the terminal, most notably in the sub-commands to convert YAML to JSON and vice versa.

Installation

On macOS, dyff is available via Homebrew:

brew install homeport/tap/dyff

Prebuilt binaries for a lot of operating systems and architectures can be downloaded from the releases section.

There is a convenience script to download the latest release for Linux or macOS if you want to keep it simple (you need curl and jq installed on your machine):

curl --silent --location https://tinyurl.com/y4qvdl4d | bash

You can download and build dyff from source using go get:

GO111MODULE=on go get github.com/homeport/dyff/cmd/dyff

Use cases and examples

  • Show the differences between two versions of cf-deployment YAMLs:

    dyff between \
      https://raw.githubusercontent.com/cloudfoundry/cf-deployment/v1.10.0/cf-deployment.yml \
      https://raw.githubusercontent.com/cloudfoundry/cf-deployment/v1.20.0/cf-deployment.yml

    dyff between example

  • Embed dyff into Git for better understandable differences

    # Setup...
    git config --local diff.dyff.command 'dyff_between() { dyff --color on between --omit-header "$2" "$5"; }; dyff_between'
    echo '*.yml diff=dyff' >> .gitattributes
    
    # And have fun, e.g.:
    git log --ext-diff -u
    git show --ext-diff HEAD

    dyff between example of a Git commit

  • Convert a JSON stream to YAML

    sometool --json | jq --raw-output '.data' | dyff yaml -
  • Sometimes you end up with YAML or JSON files, where the order of the keys in maps was sorted alphabetically. With dyff you can restructure keys in maps to a more human appealing order:

    sometool --export --json | dyff yaml --restructure -

    Or, rewrite a file in place with the restructured order of keys.

    dyff yaml --restructure --in-place somefile.yml
  • Just print a YAML (or JSON) file to the terminal to look at it. By default, dyff will use a neat output schema which includes different colors and indent helper lines to improve readability. The colors are roughly based on the default Atom schema and work best on dark terminal backgrounds. The neat output is disabled if the output of dyff is redirected into a pipe, or you can disable it explicitly using the --plain flag.

    dyff yaml somefile.yml
  • Convert a YAML file to JSON and vice versa:

    dyff json https://raw.githubusercontent.com/cloudfoundry/cf-deployment/v1.19.0/cf-deployment.yml

    The dyff sub-command (yaml, or json) defines the output format, the tool automatically detects the input format itself.

    dyff yaml https://raw.githubusercontent.com/homeport/dyff/develop/assets/bosh-yaml/manifest.json
Issues
  • Support different document ordering

    Support different document ordering

    Diffing two files that contain the same set of documents, but in different orders shows a diff. At least with e.g. Kubernetes resources we can identify the document uniquely using kind and metadata.name, which would allow dyff to pair up documents.

    I realise this is probably related to #16 as it's dealing with multiple documents.

    a.yaml:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: x
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: y
    

    b.yaml:

    apiVersion: v1
    kind: Service
    metadata:
      name: y
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: x
    

    dyff between --ignore-order-changes --detect-kubernetes a.yaml b.yaml:

         _        __  __
       _| |_   _ / _|/ _|  between a.yaml, two documents
     / _' | | | | |_| |_       and b.yaml, two documents
    | (_| | |_| |  _|  _|
     \__,_|\__, |_| |_|   returned six differences
            |___/
    
    apiVersion  (document #1)
      ± value change
        - apps/v1
        + v1
    
    kind  (document #1)
      ± value change
        - Deployment
        + Service
    
    metadata.name  (document #1)
      ± value change
        - x
        + y
    
    apiVersion  (document #2)
      ± value change
        - v1
        + apps/v1
    
    kind  (document #2)
      ± value change
        - Service
        + Deployment
    
    metadata.name  (document #2)
      ± value change
        - y
        + x
    
    
    enhancement 
    opened by jsok 7
  • arm64 support

    arm64 support

    I have some issue installing on macOS with m1:

    $ curl --silent --location https://git.io/JYfAY | sudo bash
    Unsupported operating system darwin or machine type arm64: Please check https://github.com/homeport/dyff/releases manually.
    

    arm64 is supported in goreleaser https://github.com/goreleaser/goreleaser/pull/1956 with go 1.16 version

    enhancement 
    opened by maciej-wilczynski 7
  • Support for diffing multilines string content

    Support for diffing multilines string content

    fixes #171

    If implemented, an inline diff will be processed for multilines strings, and diff hunks will be highlighted.

    Signed-off-by: Soule BA [email protected]

    opened by souleb 6
  • error with kubectl diff compatibility

    error with kubectl diff compatibility

    seems like kubectl is expecting the external diff variable to be just the executable name? is there a workaround? thanks

    % export KUBECTL_EXTERNAL_DIFF="dyff between --omit-header --set-exit-code"
    % kubectl diff -f foo.yaml
    error: failed to run "dyff between --omit-header --set-exit-code": executable file not found in $PATH

    % kubectl version Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.17-gke.4900", GitCommit:"2812f9fb0003709fc44fc34166701b377020f1c9", GitTreeState:"clean", BuildDate:"2021-03-19T09:19:27Z", GoVersion:"go1.13.15b4", Compiler:"gc", Platform:"linux/amd64"}

    opened by onching 6
  • Option to filter results

    Option to filter results

    I'm using dyff for comparing k8s manifests which can be quite huge. So I think it'll be useful if we can filter out results using some pattern like below.

    dyff between file1.json file2.json --filter deployment.spec.template.spec.containers.env
    

    PS: Great job with the visualisation 🥇

    opened by rvignesh89 6
  • HumanReport output with additions including multiple '_' treats them as formatting directives in output

    HumanReport output with additions including multiple '_' treats them as formatting directives in output

    See for example:

    Screen Shot 2022-05-24 at 3 42 04 PM

    in actuality, those fields are like:

      string x_forwarded_host = 53;
      string worker_status = 54;
      uint64 worker_cpu_time_micro = 55;
    

    I'll take a look at raising a fix myself, but just wanted to raise the issue :)

    opened by dlmather 5
  • Support Yaml Anchors and Aliases.

    Support Yaml Anchors and Aliases.

    Hi,

    This tool is really good and it helped me a lot. My request is to actually run the diff logic on the "expanded' version after expanding all the anchors and aliases.

    Thx Gil

    bug 
    opened by gfliker-emx 5
  • Support for diffing kubernetes configmap content

    Support for diffing kubernetes configmap content

    Right now, dyff treats configmaps like every other resource, which means that if there's a one-line difference in a hundred-line file, dyff will print all 100 lines of both old and new content.

    It'd be nice if dyff detected that a resource was a configmap and diffed the content instead of treating values in the configmap as atomic. (I hope that makes sense.)

    opened by SrslyJosh 4
  • Curl convenience installation script results in a 404

    Curl convenience installation script results in a 404

    Roughly two weeks ago I was able to use the convenience script to install dyff as described on https://github.com/homeport/dyff :

    This is the script: curl --silent --location https://goo.gl/DRXDVN | bash

    The shorthand link resolves to https://raw.githubusercontent.com/HeavyWombat/dyff/master/scripts/download-latest.sh which gives me a 404.

    opened by MartinKanters 4
  • Example of embedding dyff into git

    Example of embedding dyff into git

    Works rather well but unfortunately dyff does not allow hiding header with ASCII graphics dyff logo. This header is displayed each change when browsing git log --ext-diff -u and takes quite much space.

    opened by Self-Perfection 4
  • dyff between -o headless

    dyff between -o headless

    dyff outputs a nice header on between command, but for post process it could be nice to removed it. With files fom.yaml

    a: 1
    b: 2
    c: 3
    

    to.yaml

    a: 1
    b: 22
    c: 3
    

    The command

    dyff -o headless from.yaml to.yaml
    

    Will outputs

    b
      ± value change
        - 2
        - 22
    
    opened by abourree 4
  • Allow omitting whitespace only changes

    Allow omitting whitespace only changes

    Hey all, thanks a lot for maintaining this great tool!

    This is a feature request/suggestion for isWhitespaceOnlyChanges: https://github.com/homeport/dyff/blob/416d58b8e8ce6f9f208a662c4b3a6e2a9bb142ee/pkg/dyff/output_human.go#L337-L342

    It would be amazing to have a --omit-whitespace flag on dyff between. WDYT?

    opened by clux 2
  • simuluate git diff output

    simuluate git diff output

    Code mostly copied from dyff between , trying to resolve #189

    • accepts all arguments passed from git
    • generate similar diff context like built-in git diff
    • omit dyff header when running as external git diff tool
    • set GIT_CONFIG_PARAMETERS="'color.ui=always'" to enable color output

    Tested with GIT_CONFIG_PARAMETERS="'color.ui=always'" git log --ext-diff -p '**/*.yml' in current repo:

    Example output (and limitations)

    commit c45a66f9da22f6676b31767dca2a9e68dd62a8a3
    Author: Matthias Diester <[email protected]>
    Date:   Sun Mar 27 22:32:28 2022 +0200
    
        Fix timestamp parse issue in JSON command
        
        Fixes #217
        
        Bump `github.com/gonvenience/neat` to pick up timestamp parsing fix.
        
        Add test case based on user feedback.
    
    dyff --git a/assets/issues/issue-217/datestring.yml b/assets/issues/issue-217/datestring.yml
    new file mode 100644
    index 0000000..71d2450
    --- /dev/null
    +++ b/assets/issues/issue-217/datestring.yml
    
    (root level)
    + ten map entries added:
      Datestring: 2033-12-20
      ThirteenthMonth: 2033-13-20
      FortyDays: 2033-13-40
      TheYear9999: 9999-11-20
      OneShortDatestring: 999-99-99
      ExtDatestring: 2021-01-01-04-05
      DatestringFake: 9999-99-99
      DatestringNonHyphenated: 99999999
      DatestringOneHyphen: 9999-9999
      DatestringSlashes: 2022/01/01
    
    
    commit 9f044ad62aaa0354937be279cb7e815011c673b8
    Author: Matthias Diester <[email protected]>
    Date:   Mon Dec 6 20:45:04 2021 +0100
    
        Add Kubernetes specific document compare
        
        Introduce function to compare Kubernetes resources by their name.
    
    dyff --git a/assets/issues/issue-184/from.yml b/assets/issues/issue-184/from.yml
    new file mode 100644
    index 0000000..cfaa8a4
    --- /dev/null
    +++ b/assets/issues/issue-184/from.yml
    ╭ Error: failed to compare input files
    │ comparing YAMLs with a different number of documents is
    │ currently not supported
    ╵
    fatal: external diff died, stopping at assets/issues/issue-184/from.yml
    
    opened by williamjoy 2
  • Add git diff support

    Add git diff support

    It would be quite nice to have a dyff command called "git" which would accept command line arguments in the way that:

    GIT_EXTERNAL_DIFF="dyff git" git diff
    

    would accept. It's documented in the git(1) man page.

    opened by elric1 0
  • Question: How to handle

    Question: How to handle "sensitive" diff ?

    Any idea how to handle sensitive values in output ? Its easy to determine sensitive values based on Kind type for Kubernetes for example, but wondering what can be used for docker-compose spec.

    Thinking about some argument which can help mask whole environment array/map: ( instead of values, some hash can be outputed ) dyff between docker-compose-old.yaml docker-compose-new.yaml --mask services.*.environment

    or mask specific keys: dyff between docker-compose-old.yaml docker-compose-new.yaml --mask services.*.environment.APP_TOKEN --mask services.*.environment.DATABASE_PASSWORD

    with file input: dyff between ... --mask-from-file denylist.json

    or with regular support: dyff between ... --mask-keys-regex '.*(PASSWORD|TOKEN|CERTIFICATE)'

    environments spec in docker-compose can be either map or list, ^ would probably work only for the map variant.

    Any ideas?

    Thanks

    opened by lukasmrtvy 1
Releases(v1.5.4)
JSON diff library for Go based on RFC6902 (JSON Patch)

jsondiff jsondiff is a Go package for computing the diff between two JSON documents as a series of RFC6902 (JSON Patch) operations, which is particula

William Poussier 172 Jun 23, 2022
A JSON diff utility

JayDiff A JSON diff utility. Install Downloading the compiled binary Download the latest version of the binary: releases extract the archive and place

Guillaume de Sagazan 86 May 31, 2022
A simple Go package to Query over JSON/YAML/XML/CSV Data

A simple Go package to Query over JSON Data. It provides simple, elegant and fast ODM like API to access, query JSON document Installation Install the

Saddam H 1.9k Jun 22, 2022
A convenient syntax to generate JSON (or YAML) for commandline

clon A convenient syntax to generate JSON (or YAML) for commandline "mumbo-jumbo". Syntax Overview Syntax resembles that of JSON with a few caveats: a

Luca Sepe 9 May 14, 2021
Package json implements encoding and decoding of JSON as defined in RFC 7159

Package json implements encoding and decoding of JSON as defined in RFC 7159. The mapping between JSON and Go values is described in the documentation for the Marshal and Unmarshal functions

High Performance, Kubernetes Native Object Storage 3 May 10, 2022
Json-go - CLI to convert JSON to go and vice versa

Json To Go Struct CLI Install Go version 1.17 go install github.com/samit22/js

Samit Ghimire 5 Mar 3, 2022
JSON Spanner - A Go package that provides a fast and simple way to filter or transform a json document

JSON SPANNER JSON Spanner is a Go package that provides a fast and simple way to

null 2 May 20, 2022
Get JSON values quickly - JSON parser for Go

get json values quickly GJSON is a Go package that provides a fast and simple way to get values from a json document. It has features such as one line

Josh Baker 10.5k Jun 20, 2022
Fast JSON encoder/decoder compatible with encoding/json for Go

Fast JSON encoder/decoder compatible with encoding/json for Go

Masaaki Goshima 1.6k Jun 26, 2022
Devto-json2md - Convert the dev.to blog JSON export into Markdown files

devto-json2md Convert dev.to blog JSON export into Markdown files Features Exports Markdown compatible files using Hugo-compatible front-matter fields

Thomas Strömberg 0 Jan 8, 2022
A tool to aggregate and mine data from JSON reports of Go tests.

teststat A tool to aggregate and mine data from JSON reports of Go tests. Why? Mature Go projects often have a lot of tests, and not all of them are i

Viacheslav Poturaev 3 Jan 3, 2022
CLI tool to handle JSON logs

logit is a CLI tool that consumes logs in JSON and feeds them into a logs storage like file, Amazon CloudWatch, or Slack.

Life4 22 Nov 9, 2021
Online tool that convert JSON to Go.

json-to-go 将json生成go的数据结构。Online tool that convert JSON to Go. Online Link | 地址 Link1 | 国内 Link2 | 国外 Local Develope | 本地开发 suggest use(强烈建议使用) pnpm !

misaka full stack 15 Dec 18, 2021
Fast JSON parser and validator for Go. No custom structs, no code generation, no reflection

fastjson - fast JSON parser and validator for Go Features Fast. As usual, up to 15x faster than the standard encoding/json. See benchmarks. Parses arb

Aliaksandr Valialkin 1.6k Jun 21, 2022
Fast and flexible JSON encoder for Go

Jettison Jettison is a fast and flexible JSON encoder for the Go programming language, inspired by bet365/jingo, with a richer features set, aiming at

William Poussier 128 Jun 1, 2022
HuJSON: JSON for Humans (comments and trailing commas)

HuJSON - Human JSON The HuJSON decoder is a JSON decoder that also allows comments, both /* ... */ and // to end of line trailing commas on arrays and

Tailscale 394 Jun 28, 2022
tson is JSON viewer and editor written in Go

tson tson is JSON viewer and editor written in Go. This tool displays JSON as a tree and you can search and edit key or values. Support OS Mac Linux I

skanehira 127 Mar 17, 2022
JSONata in Go Package jsonata is a query and transformation language for JSON

JSONata in Go Package jsonata is a query and transformation language for JSON. It's a Go port of the JavaScript library JSONata.

Blues Inc 25 May 27, 2022
Slow and unreliable JSON parser generator (in progress)

VivaceJSON Fast and reliable JSON parser generator Todo List parse fields parse types generate struct generate (keypath+key) to struct Value Mapping F

null 7 Mar 3, 2022