A better way to marshal and unmarshal YAML in Golang

Related tags

Configuration yaml
Overview

YAML marshaling and unmarshaling support for Go

Build Status

Introduction

A wrapper around go-yaml designed to enable a better way of handling YAML when marshaling to and from structs.

In short, this library first converts YAML to JSON using go-yaml and then uses json.Marshal and json.Unmarshal to convert to or from the struct. This means that it effectively reuses the JSON struct tags as well as the custom JSON methods MarshalJSON and UnmarshalJSON unlike go-yaml. For a detailed overview of the rationale behind this method, see this blog post.

Compatibility

This package uses go-yaml and therefore supports everything go-yaml supports.

Caveats

Caveat #1: When using yaml.Marshal and yaml.Unmarshal, binary data should NOT be preceded with the !!binary YAML tag. If you do, go-yaml will convert the binary data from base64 to native binary data, which is not compatible with JSON. You can still use binary in your YAML files though - just store them without the !!binary tag and decode the base64 in your code (e.g. in the custom JSON methods MarshalJSON and UnmarshalJSON). This also has the benefit that your YAML and your JSON binary data will be decoded exactly the same way. As an example:

BAD:
	exampleKey: !!binary gIGC

GOOD:
	exampleKey: gIGC
... and decode the base64 data in your code.

Caveat #2: When using YAMLToJSON directly, maps with keys that are maps will result in an error since this is not supported by JSON. This error will occur in Unmarshal as well since you can't unmarshal map keys anyways since struct fields can't be keys.

Installation and usage

To install, run:

$ go get github.com/ghodss/yaml

And import using:

import "github.com/ghodss/yaml"

Usage is very similar to the JSON library:

package main

import (
	"fmt"

	"github.com/ghodss/yaml"
)

type Person struct {
	Name string `json:"name"` // Affects YAML field names too.
	Age  int    `json:"age"`
}

func main() {
	// Marshal a Person struct to YAML.
	p := Person{"John", 30}
	y, err := yaml.Marshal(p)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(y))
	/* Output:
	age: 30
	name: John
	*/

	// Unmarshal the YAML back into a Person struct.
	var p2 Person
	err = yaml.Unmarshal(y, &p2)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(p2)
	/* Output:
	{John 30}
	*/
}

yaml.YAMLToJSON and yaml.JSONToYAML methods are also available:

package main

import (
	"fmt"

	"github.com/ghodss/yaml"
)

func main() {
	j := []byte(`{"name": "John", "age": 30}`)
	y, err := yaml.JSONToYAML(j)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(y))
	/* Output:
	name: John
	age: 30
	*/
	j2, err := yaml.YAMLToJSON(y)
	if err != nil {
		fmt.Printf("err: %v\n", err)
		return
	}
	fmt.Println(string(j2))
	/* Output:
	{"age":30,"name":"John"}
	*/
}
Issues
  • Please tag a new release after #49

    Please tag a new release after #49

    #49 adds basic support for Go modules, but the latest current release tag (v1.0.0) is dated March 2017. Could you please find a suitable point at or after #49 to add a newer tag? (I think v1.1.0 is a reasonable choice, since there don't seem to be any API changes, but I didn't check exhaustively).

    opened by creachadair 22
  • Panic when trying to unmarshal specific yaml

    Panic when trying to unmarshal specific yaml

    This is a pretty bizarre one. When trying to unmarshal this yaml:

    Port: 7746
    Processes:
        Something:
            Command: some command
    

    I get this panic:

    panic: reflect: reflect.Value.Set using unaddressable value
    
    goroutine 1 [running]:
    panic(0x3cdb80, 0xc820132680)
        /usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:481 +0x3e6
    reflect.flag.mustBeAssignable(0x16)
        /usr/local/Cellar/go/1.6.2/libexec/src/reflect/value.go:229 +0x221
    reflect.Value.Set(0x3a98e0, 0x0, 0x16, 0x3a98e0, 0xc82016e2c0, 0x16)
        /usr/local/Cellar/go/1.6.2/libexec/src/reflect/value.go:1328 +0x25
    dj/vendor/github.com/ghodss/yaml.indirect(0x3a98e0, 0x0, 0x16, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /Users/craig/go-clean/src/dj/vendor/github.com/ghodss/yaml/fields.go:48 +0x528
    dj/vendor/github.com/ghodss/yaml.convertToJSONableObject(0x3ca880, 0xc820136630, 0xc8201346a0, 0x0, 0x0, 0x0, 0x0)
        /Users/craig/go-clean/src/dj/vendor/github.com/ghodss/yaml/yaml.go:106 +0xc9
    dj/vendor/github.com/ghodss/yaml.convertToJSONableObject(0x3ca880, 0xc8201365d0, 0xc8200d7170, 0x0, 0x0, 0x0, 0x0)
        /Users/craig/go-clean/src/dj/vendor/github.com/ghodss/yaml/yaml.go:205 +0xf3a
    dj/vendor/github.com/ghodss/yaml.convertToJSONableObject(0x3ca880, 0xc8201365a0, 0xc8200d7628, 0x0, 0x0, 0x0, 0x0)
        /Users/craig/go-clean/src/dj/vendor/github.com/ghodss/yaml/yaml.go:195 +0xb15
    dj/vendor/github.com/ghodss/yaml.yamlToJSON(0xc820162000, 0x43, 0x243, 0xc8200d78a0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/craig/go-clean/src/dj/vendor/github.com/ghodss/yaml/yaml.go:89 +0x110
    dj/vendor/github.com/ghodss/yaml.Unmarshal(0xc820162000, 0x43, 0x243, 0x3a9880, 0x78c480, 0x0, 0x0)
        /Users/craig/go-clean/src/dj/vendor/github.com/ghodss/yaml/yaml.go:32 +0xa8
    

    Interestingly it seems like it might be something specific to the word Processes cause if I change that to Process or really any other word it works fine. I was previously using an older version of your library c3eb24aeea63668ebdac08d2e252f20df8b6b1ae and it works fine there.

    opened by ccampbell 9
  • Return typed errors from exported funcs

    Return typed errors from exported funcs

    By adding a YAMLError type and returning typed errors from exported functions, downstream users will be able to easily distinguish yaml-specific errors.

    Related downstream PR: https://github.com/kubernetes/kubernetes/pull/38905

    cc @deads2k @fabianofranz

    opened by juanvallejo 6
  • "y" argument not parsed

    After parsing the following yaml :

    x: 10
    y: 20
    

    x is being extracted, but y is not. Changing 'y' to anything else works.

    Reproducible sample:

    package main
    
    import (
        "fmt"
        "github.com/ghodss/yaml"
    )
    
    type Position struct {
        X int `json:"x"`
        Y int `json:"y"`
    }
    
    func main() {
        s := []byte(`
    x: 10
    y: 20
    `)
    
        var p Position
        err := yaml.Unmarshal(s, &p)
        if err != nil {
            fmt.Printf("err: %v\n", err)
            return
        }
        fmt.Println(p) // Should be {10 20}, but prints {10 0}
    }
    
    
    opened by pavel-georgiev 6
  • Add UnmarshalStrict for strict yaml unmarshaling

    Add UnmarshalStrict for strict yaml unmarshaling

    Reopening a fresh PR with code from @tanner-bruce in: https://github.com/ghodss/yaml/pull/36

    Please note that i had another one similar to Tanner's in: https://github.com/ghodss/yaml/pull/37

    This is NOT the same as in: https://github.com/ghodss/yaml/pull/33

    and we need it to fix Kubernetes issue: https://github.com/kubernetes/kubernetes/issues/67477

    opened by dims 4
  • Adds a hook to specify unmarshal options.

    Adds a hook to specify unmarshal options.

    This makes it possible to configure the parser to error out if an unknown tagged field has been found in the byte stream that is being unmarshaled to a go object.

    For example like this (also see ExampleUnknown in yaml_test.go):

    func ExampleUnknown() {
    	type WithTaggedField struct {
    		Field string `json:"field"`
    	}
    	y := []byte(`unknown: "hello"`)
    	v := WithTaggedField{}
    	fmt.Printf("%v\n", Unmarshal(y, &v, DisallowUnknownFields))
    	// Ouptut:
    	// unmarshaling JSON: while decoding JSON: json: unknown field "unknown"
    }
    

    If you want to manipulate the decoder used in the unmarshal function, you can define a custom option to your liking and apply it in the Unmarshal call.

    The way I found out about this is kind of comical. Some nonprintable characters made its way into my YAML file. You wouldn't see them in the editor, but they'd be there. Now when the YAML file is parsed, the nonprintable chars together with the key characters were made into a weird UTF-8 key (e.g. like "\u00c2some_key"), which would then become a key unknown to the tags of my struct, and would silently get dropped in the parsing steps. So as result, you would get a perfectly normal parse, except the data that you wanted to be in the struct is not there.

    This meant that silently dropping unknown keys is not always a good idea, so I went to fix it, while retaining the same interface. JsonOpt type was fairly easy to add in the Unmarshal because it can be silently ignored by the old code.

    opened by filmil 4
  • Add UnmarshalStrict.

    Add UnmarshalStrict.

    UnmarshalStrict is like Unmarshal except that any mapping keys that are duplicates, will result in an error.

    Since https://godoc.org/gopkg.in/yaml.v2 provides both Unmarshal and UnmarshalStrict, this library should provide the same interface.

    opened by diekmann 3
  • Use pkg/errors to wrap extended errors

    Use pkg/errors to wrap extended errors

    Creating a new error with the content from the previous one loses context. By wrapping the error, we can still extend it while allowing to retrieve the base error.

    For example, the json package defines SyntaxError which carries more than just the message (by being it's own struct, we can do a switch on it's type). Wrapping the error avoids losing that information.

    opened by dmathieu 3
  • Difference in behaviour parsing numbers compared to gopkg.in/yaml.v2

    Difference in behaviour parsing numbers compared to gopkg.in/yaml.v2

    gopkg.in/yaml.v2 parses yaml integers into ints and yaml floats into float64s, which I believe is correct. github.com/ghodss/yaml parses both into float64s, which is a difference in behaviour, and which I believe is wrong.

    opened by jim-minter 2
  • How to decode yaml with ---

    How to decode yaml with ---

    kind: Namespace
    metadata:
       name: test
    ---
    name: aaa
    
    type Namespace struct {
        Kind     string `json:"kind"`
        Metadata struct {
            Name string `json:"name"`
        } `json:"metadata"`
    }
    
    type Name struct {
        Name string `json:"name"`
    }
    

    Can't decode the name struct, How?

    opened by fanux 2
  • Travis doesn't allow testing with go 1.10

    Travis doesn't allow testing with go 1.10

    Time passes and go 1.10 is now the "official" current go version to use.

    Sadly, .travis.yaml considers go 1.10 to be go 1.1, so doesn't build and test properly.

    See here: https://travis-ci.org/ghodss/yaml/builds/363368191?utm_source=github_status&utm_medium=notification

    The .travis.yml file used is this one: https://github.com/ghodss/yaml/pull/27/files#diff-354f30a63fb0907d4ad57269548329e3

    note how tests are performed for go versions 1.3, 1.4 and 1.1.

    I would expect 1.3, 1.4 and 1.10 instead.

    opened by filmil 2
  • Keys like

    Keys like "yes" and "on" are read as true

    template = `
    on:
     - RUNNING
    `
    template, _ = yaml.YAMLToJSON(template)
    fmt.Println(string(template))
    // => {"true":["DELETED"]}
    
    opened by slntopp 3
  • How about cutting a new release?

    How about cutting a new release?

    The latest release is v1.0.0 which is released in the year 2017, I think it's time to cut a new release, e.g. v1.0.1 or v1.1.0(only if there are new features).

    cc @bradfitz @ghodss

    opened by RainbowMango 0
  • Add support for UnmarshalStrict

    Add support for UnmarshalStrict

    Hi,

    I'm wondering if there is a plan to add support for the UnmarshalStrict function of go-yaml? It's not in the current release, are there any plans on releasing soon?

    opened by ipapast 0
  • unable to convert zero prepended integer from YAML file to YAML object using “yaml.Unmarshal”

    unable to convert zero prepended integer from YAML file to YAML object using “yaml.Unmarshal”

    Hello experts,

    Would like to know how to convert zero prepended integer( i.e octal integer) number from YAML file to YAML Object. Tried with yaml.v3 also but could see same results.

    Any help would be appreciates

    zero prepended integer examples:

    01 001 00002 and so

    Sample program ============== package main

    import "fmt" import "gopkg.in/yaml.v2"

    var data = a: 01 b: 1

    func main() { var obj interface{} _ = yaml.Unmarshal([]byte(data), &obj) fmt.Println("YAML data:", data) fmt.Println("OBJ:", obj) }_

    O/p of above program:

    =================== YAML data: a: 01 b: 1 OBJ: map[a:1 b:1]

    opened by AnilPetnaktoa 0
  • Add YAMLToJSONCustom to accept custom unmarshal func

    Add YAMLToJSONCustom to accept custom unmarshal func

    We want to use YAMLToJSON with yaml.v3 and this is the most minimal change that allows us to use yaml.v3 without breaking the interface of the rest of the library.

    Why v3? Because v3 changed how it treats boolean values:

    YAML 1.1 bools (yes/no, on/off) are supported as long as they are being decoded into a typed bool value. Otherwise they behave as a string. Booleans in YAML 1.2 are true/false only. (https://github.com/go-yaml/yaml/blob/v3/README.md#compatibility)

    opened by mrnugget 2
Releases(v1.0.0)
Owner
Sam Ghods
Sam Ghods
Go-yaml - Yaml parsing Toolkit For Golang

go-yaml 介绍 gopkg.in/yaml.v3 已经是个非常好用的包,但是在实际开发中总有类型转换带来的麻烦,go-yaml只是在它的基础上,简单的一层

xiaowu 4 Jan 13, 2022
Golang library for reading properties from configuration files in JSON and YAML format or from environment variables.

go-config Golang library for reading properties from configuration files in JSON and YAML format or from environment variables. Usage Create config in

null 3 Feb 22, 2022
Library providing routines to merge and validate JSON, YAML and/or TOML files

CONFLATE Library providing routines to merge and validate JSON, YAML, TOML files and/or structs (godoc) Typical use case: Make your application config

Andy 25 May 5, 2022
Golang Configuration tool that support YAML, JSON, TOML, Shell Environment

Configor Golang Configuration tool that support YAML, JSON, TOML, Shell Environment (Supports Go 1.10+) Usage package main import ( "fmt" "github.c

Jinzhu 1.4k May 16, 2022
Golang config.yaml loader

Description goconfig is a configuration library designed using the following pri

null 1 Dec 21, 2021
Generic templating tool with support of JSON, YAML and TOML data

gotempl Small binary used to generate files from Go Templates and data files. The following formats are supported: JSON YAML TOML Usage usage: gotempl

Link Society 6 Mar 23, 2022
Go-config - Config parser for go that supports environment vars and multiple yaml files

go-multiconfig This package is able to parse yaml config files. It supports gett

Dimitris Tassopoulos 0 Jan 31, 2022
JSON or YAML configuration wrapper with convenient access methods.

Config Package config provides convenient access methods to configuration stored as JSON or YAML. This is a fork of the original version. This version

Oleg Lebedev 250 May 5, 2022
Light weight, extensible configuration management library for Go. Built in support for JSON, TOML, YAML, env, command line, file, S3 etc. Alternative to viper.

koanf (pronounced conf; a play on the Japanese Koan) is a library for reading configuration from different sources in different formats in Go applicat

Kailash Nadh 981 May 14, 2022
YAML support for the Go language.

YAML support for the Go language Introduction The yaml package enables Go programs to comfortably encode and decode YAML values. It was developed with

null 5.6k May 11, 2022
create a bootable disk image from Docker image or a yaml config

docker2boot docker2boot creates a bootable disk from either a Docker image or a config yaml file Features status dns Y cloud-init Y network Y ssh TODO

Bin Chen 6 Dec 21, 2021
It syncronizes the configuration described in a YAML file against your GitHub Organization

It syncronizes the configuration described in a YAML file against your GitHub Organization. Combined with a CI system, it can be used to implement GitOps for GitHub.

Carlos Tadeu Panato Junior 6 Jul 19, 2021
YAML support for the Go language

YAML support for the Go language

Masaaki Goshima 689 May 9, 2022
Tmpl - A tool to apply variables from cli, env, JSON/TOML/YAML files to templates

tmpl allows to apply variables from JSON/TOML/YAML files, environment variables or CLI arguments to template files using Golang text/template and functions from the Sprig project.

krako 1 Jan 31, 2022
SmartYAML - Go package to handle YAML

SmartYAML - Go package to handle YAML The smartyaml is a go package to handle parsed YAML files more confortable. This package is not a parser, it use

Péter Deák 0 Feb 25, 2022
VINYL Inscribes Nettlesome YAML Legibly

VINYL Inscribes Nettlesome YAML Legibly VINYL formats yaml files into a canonical format retaining comments and setting the indentation by default to

Jesse Hathaway 1 Jan 18, 2022
Quick and easy way to load config files based on a simple set of rules.

config Quick and easy way to load config files based on a simple set of rules. Project inspired by https://github.com/lorenwest/node-config Important

Tarcisio Gruppi 1 Apr 9, 2021
Noel 2 Feb 13, 2022