Library providing routines to merge and validate JSON, YAML and/or TOML files

Overview

gophers

CONFLATE

Library providing routines to merge and validate JSON, YAML, TOML files and/or structs (godoc)

Typical use case: Make your application configuration files multi-format, modular, templated, sparse, location-independent and validated

Build Status Coverage Status Go Report Card

Description

Conflate is a library and cli-tool, that provides the following features :

  • merge data from multiple formats (JSON/YAML/TOML/go structs) and multiple locations (filesystem paths and urls)
  • validate the merged data against a JSON schema
  • apply any default values defined in a JSON schema to the merged data
  • expand environment variables inside the data
  • marshal merged data to multiple formats (JSON/YAML/TOML/go structs)

Improvements, ideas and bug fixes are welcomed.

Background

This project is a hard fork of the original that I developed at https://github.com/miracl/conflate. However, I will be actively supporting and extending this project independently here.

Getting started

Run the following command, which will build and install the latest binary in $GOPATH/bin

go get github.com/the4thamigo-uk/conflate/...

Alternatively, you can install one of the pre-built release binaries from https://github.com/the4thamigo-uk/conflate/releases

Usage of Library

Please refer to the godoc and the example code

Usage of CLI Tool

Help can be obtained in the usual way :

$conflate --help
Usage of conflate:
  -data value
    	The path/url of JSON/YAML/TOML data, or 'stdin' to read from standard input
  -defaults
    	Apply defaults from schema to data
  -expand
    	Expand environment variables in files
  -format string
    	Output format of the data JSON/YAML/TOML
  -includes string
    	Name of includes array. Blank string suppresses expansion of includes arrays (default "includes")
  -noincludes
    	Switches off conflation of includes. Overrides any --includes setting.
  -schema string
    	The path/url of a JSON v4 schema file
  -validate
    	Validate the data against the schema
  -version
    	Display the version number

Basic Merging of Files by Inclusion

To conflate the following file :

$cat ./testdata/valid_parent.json
{
  "includes": [
    "valid_child.json",
    "valid_sibling.json"
  ],
  "parent_only" : "parent", 
  "parent_child" : "parent", 
  "parent_sibling" : "parent", 
  "all": "parent"
}

...run the following command, which will merge valid_parent.json, valid_child.json, valid_sibling.json, resulting in the file :

$conflate -data ./testdata/valid_parent.json -format JSON
{
  "all": "parent",
  "child_only": "child",
  "parent_child": "parent",
  "parent_only": "parent",
  "parent_sibling": "parent",
  "sibling_child": "sibling",
  "sibling_only": "sibling"
}

Note how the includes, are loaded as paths relative to the including file.

Also, note that where the same named value occurs in multiple files, the resulting value follows the following priority rules :

  • the values defined in a file, override the values that are imported from the included file(s)
  • the values defined in an included file, override the values imported from any included file(s) occurring before it, in the includes list

Furthermore, included files can include other files to any depth.

Choosing an Output Format

To output in a different format use the -format option, e.g. TOML :

$conflate -data ./testdata/valid_parent.json -format TOML
all = "parent"
child_only = "child"
parent_child = "parent"
parent_only = "parent"
parent_sibling = "parent"
sibling_child = "sibling"
sibling_only = "sibling"

Basic Merging of Files Without Inclusion

If you don't want to intrusively embed an "includes" array inside your JSON, you can instead provide multiple files on the command line, which are prioritised from left-to-right, with the right taking highest priority :

$conflate -data ./testdata/valid_child.json -data ./testdata/valid_sibling.json -format JSON
{
  "all": "sibling",
  "child_only": "child",
  "parent_child": "child",
  "parent_sibling": "sibling",
  "sibling_child": "sibling",
  "sibling_only": "sibling"
}

Or alternatively, you can create a top-level file containing only the includes array. For fun, lets choose to use YAML for the top-level file, and output TOML :

$cat toplevel.yaml 
includes:
  - testdata/valid_child.json
  - testdata/valid_sibling.json

$conflate -data toplevel.yaml -format TOML
all = "sibling"
child_only = "child"
parent_child = "child"
parent_sibling = "sibling"
sibling_child = "sibling"
sibling_only = "sibling"

Merging Data from STDIN

If you want to read a file from stdin you use -data stdin. For example, here we pipe in some TOML to override a single value in a JSON file :

$echo 'all="MY OVERRIDDEN VALUE"' |  conflate -data ./testdata/valid_parent.json -data stdin -format JSON
{
  "all": "MY OVERRIDDEN VALUE",
  "child_only": "child",
  "parent_child": "parent",
  "parent_only": "parent",
  "parent_sibling": "parent",
  "sibling_child": "sibling",
  "sibling_only": "sibling"
}

Note that in all cases -data sources are processed from left-to-right, with values in right files overriding values in left files, so the following doesnt work, since the key "all", is present in ./testdata/valid_parent.json:

$echo 'all="MY OVERRIDDEN VALUE"' |  conflate -data stdin -data ./testdata/valid_parent.json  -format JSON
{
  "all": "parent",
  "child_only": "child",
  "parent_child": "parent",
  "parent_only": "parent",
  "parent_sibling": "parent",
  "sibling_child": "sibling",
  "sibling_only": "sibling"
}

Merging of Remote files by Inclusion

If you instead host a file somewhere else, then just use a URL :

$conflate -data https://raw.githubusercontent.com/the4thamigo-uk/conflate/master/testdata/valid_parent.json -format JSON
{
  "all": "parent",
  "child_only": "child",
  "parent_child": "parent",
  "parent_only": "parent",
  "parent_sibling": "parent",
  "sibling_child": "sibling",
  "sibling_only": "sibling"
}

The includes here are also loaded as urls relative the the url of the including file, and follow exactly the same merging rules as for local files.

Validation Against a JSON Schema

To validate your data against a JSON schema, use -schema in combination with -validate :

$cat ./testdata/blank.yaml

$conflate -data ./testdata/blank.yaml -schema ./testdata/test.schema.json -validate -format YAML
Schema validation failed : The document is not valid against the schema : Invalid type. Expected: object, given: null (#)

Note that you can validate any conflated JSON, TOML or JSON files, against the JSON v4 schema, it doesnt have to be pure JSON.

Also, note that the schema, is itself, validated for correctness against the JSON v4 meta-schema.

Use for Simplifying Boiler Plate Validation

One typical use-case for conflate, is to use JSON schema validation to avoid having to write lots of boiler plate GO code in order to validate a user's configuration file. You can perform quite extensive validation using only JSON schemas (even v4 schemas), so this can greatly simplify this kind of work. One neat approach is to embed your schema into your code as a const string, and load it using NewSchemaData, when your app starts up, so you can validate your configuration files against a schema that is directly compiled into your binary.

Applying Default Values from a JSON Schema

To apply default values defined in a JSON schema use -schema in combination with -defaults:

$conflate -data ./testdata/blank.yaml -schema ./testdata/test.schema.json -defaults -validate -format YAML
all: parent
child_only: child
parent_child: parent
parent_only: parent
parent_sibling: parent
sibling_child: sibling
sibling_only: sibling

Note if you also specify -validate, the defaults are applied before validation is performed, as you would expect.

Expansion of Environment Variables

You can optionally expand environment variables in the files like this :

$export echo MYVALUE="some value"
$export echo MYJSONMAP='{ "item1" : "value1" }'
$echo '{ "my_value": "$MYVALUE", "my_map": $MYJSONMAP }' | conflate -data stdin -expand -format JSON
{
  "my_map": {
    "item1": "value1"
  },
  "my_value": "some value"
}

Acknowledgements

Images derived from originals by Renee French https://golang.org/doc/gopher/

Owner
Andy
Related accounts : https://github.com/andy-miracl
Andy
Tmpl - A tool to apply variables from cli, env, JSON/TOML/YAML files to templates

tmpl allows to apply variables from JSON/TOML/YAML files, environment variables or CLI arguments to template files using Golang text/template and functions from the Sprig project.

krako 1 May 30, 2022
Light weight, extensible configuration management library for Go. Built in support for JSON, TOML, YAML, env, command line, file, S3 etc. Alternative to viper.

koanf (pronounced conf; a play on the Japanese Koan) is a library for reading configuration from different sources in different formats in Go applicat

Kailash Nadh 1k Jul 6, 2022
Generic templating tool with support of JSON, YAML and TOML data

gotempl Small binary used to generate files from Go Templates and data files. The following formats are supported: JSON YAML TOML Usage usage: gotempl

Link Society 7 Jun 15, 2022
Golang Configuration tool that support YAML, JSON, TOML, Shell Environment

Configor Golang Configuration tool that support YAML, JSON, TOML, Shell Environment (Supports Go 1.10+) Usage package main import ( "fmt" "github.c

Jinzhu 1.4k Jun 27, 2022
Golang library for reading properties from configuration files in JSON and YAML format or from environment variables.

go-config Golang library for reading properties from configuration files in JSON and YAML format or from environment variables. Usage Create config in

null 3 Feb 22, 2022
Go-yaml - Yaml parsing Toolkit For Golang

go-yaml 介绍 gopkg.in/yaml.v3 已经是个非常好用的包,但是在实际开发中总有类型转换带来的麻烦,go-yaml只是在它的基础上,简单的一层

xiaowu 4 Jan 13, 2022
TOML parser and encoder library for Golang

TOML parser and encoder library for Golang TOML parser and encoder library for Golang. This library is compatible with TOML version v0.4.0. Installati

Naoya Inada 288 Jun 9, 2022
Go library for the TOML language

go-toml Go library for the TOML format. This library supports TOML version v1.0.0-rc.3 Features Go-toml provides the following features for using data

Thomas Pelletier 1.3k Jun 29, 2022
JSON or YAML configuration wrapper with convenient access methods.

Config Package config provides convenient access methods to configuration stored as JSON or YAML. This is a fork of the original version. This version

Oleg Lebedev 250 May 5, 2022
TOML parser for Golang with reflection.

THIS PROJECT IS UNMAINTAINED The last commit to this repo before writing this message occurred over two years ago. While it was never my intention to

Andrew Gallant 3.9k Jul 4, 2022
Go-config - Config parser for go that supports environment vars and multiple yaml files

go-multiconfig This package is able to parse yaml config files. It supports gett

Dimitris Tassopoulos 1 Jun 23, 2022
🛠 A configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP

config A small configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP. Example func main() {

Josh Betz 210 Apr 10, 2022
A better way to marshal and unmarshal YAML in Golang

YAML marshaling and unmarshaling support for Go Introduction A wrapper around go-yaml designed to enable a better way of handling YAML when marshaling

Sam Ghods 923 Jul 6, 2022
YAML support for the Go language.

YAML support for the Go language Introduction The yaml package enables Go programs to comfortably encode and decode YAML values. It was developed with

null 5.8k Jul 3, 2022
create a bootable disk image from Docker image or a yaml config

docker2boot docker2boot creates a bootable disk from either a Docker image or a config yaml file Features status dns Y cloud-init Y network Y ssh TODO

Bin Chen 6 Dec 21, 2021
It syncronizes the configuration described in a YAML file against your GitHub Organization

It syncronizes the configuration described in a YAML file against your GitHub Organization. Combined with a CI system, it can be used to implement GitOps for GitHub.

Carlos Tadeu Panato Junior 6 Jul 19, 2021
YAML support for the Go language

YAML support for the Go language

Masaaki Goshima 725 Jul 2, 2022
Golang config.yaml loader

Description goconfig is a configuration library designed using the following pri

null 2 May 31, 2022
SmartYAML - Go package to handle YAML

SmartYAML - Go package to handle YAML The smartyaml is a go package to handle parsed YAML files more confortable. This package is not a parser, it use

Péter Deák 0 Feb 25, 2022