Go library for decoding generic map values into native Go structures and vice versa.

Overview

mapstructure Godoc

mapstructure is a Go library for decoding generic map values to structures and vice versa, while providing helpful error handling.

This library is most useful when decoding values from some data stream (JSON, Gob, etc.) where you don't quite know the structure of the underlying data until you read a part of it. You can therefore read a map[string]interface{} and use this library to decode it into the proper underlying native Go structure.

Installation

Standard go get:

$ go get github.com/mitchellh/mapstructure

Usage & Example

For usage and examples see the Godoc.

The Decode function has examples associated with it there.

But Why?!

Go offers fantastic standard libraries for decoding formats such as JSON. The standard method is to have a struct pre-created, and populate that struct from the bytes of the encoded format. This is great, but the problem is if you have configuration or an encoding that changes slightly depending on specific fields. For example, consider this JSON:

{
  "type": "person",
  "name": "Mitchell"
}

Perhaps we can't populate a specific structure without first reading the "type" field from the JSON. We could always do two passes over the decoding of the JSON (reading the "type" first, and the rest later). However, it is much simpler to just decode this into a map[string]interface{} structure, read the "type" key, then use something like this library to decode it into the proper structure.

Comments
  • add support for 'required' tag value

    add support for 'required' tag value

    Implementation for checking that required fields got filled in map -> struct decoding direction.

    A basic test has been added as well.

    type Required struct {
    	RequiredBar string `mapstructure:"bar,required"`
    	Value       string `mapstructure:"foo"`
    }
    

    Fixes: #7

    opened by bonifaido 16
  • Create a tag

    Create a tag

    Can a tag be created of this so that dependent packages can pull a specific version rather than having to use current sources? That would be much appreciated!

    While this shows a 0.1, I made that version up. You can see in the ebuild it pulls the master.zip file rather than a tagged version.

    https://github.com/Obsidian-StudiosInc/os-xtoo/blob/master/dev-go/mapstructure/mapstructure-0.1.ebuild

    opened by wltjr 13
  • time.Time encoded as map[string]interface{}{}

    time.Time encoded as map[string]interface{}{}

    Noticed that if encoding a struct into a map, time.Time entries are passed into empty maps, which doesn't seam very useful.

    I could add a hook to encode it into a string instance, but because hooks run as pre-processors, it's not possible to preserve it as a time.Time.

    Example code:

    package main
    
    import (
    	"fmt"
    	"time"
    
    	"github.com/mitchellh/mapstructure"
    )
    
    func PayloadEncoder(target *map[string]interface{}) (*mapstructure.Decoder, error) {
    	return mapstructure.NewDecoder(&mapstructure.DecoderConfig{
    		//DecodeHook: mapstructure.ComposeDecodeHookFunc(
    		// <- no hook can be added here to solve this.
    		//),
    		Result: target,
    	})
    }
    
    type MyStruct struct {
    	CreatedAt time.Time
    }
    
    func main() {
    	p1 := MyStruct{
    		CreatedAt: time.Now(),
    	}
    	m1 := make(map[string]interface{})
    	d1, err := PayloadEncoder(&m1)
    	fmt.Println("\nerr:", err, "\ndec:", d1, "\nmap:", m1)
    	err2 := d1.Decode(p1)
    	fmt.Printf("\nerr: %v\noutput: %#v\n", err2, m1)
    }
    

    Output:

    
    err: <nil> 
    dec: &{0xc0001be000} 
    map: map[]
    
    err: <nil>
    output: map[string]interface {}{"CreatedAt":map[string]interface {}{}}
    
    opened by smyrman 12
  • Decode to int slice but input is string slice

    Decode to int slice but input is string slice

    I have specific case that my struct have []int32 , but values for it goes from url via fields=xxx&fields=yyy so input map have map['fields']=[]string{'xxx','yyy'}

    to decode it i need to get map with names and ints. How ca i do that via this package? P.S. Now i have handmade stuff that uses reflection and get values via proto.EnumValueMap via passing to it slice element type name

    opened by vtolstov 12
  • Feature: Decode structs to maps

    Feature: Decode structs to maps

    This feature adds the ability to decode structs to maps (essentially allowing the converse of the conventional use of this library). I've tried to be as complete in the test cases that I could add.

    One thing this doesn't do (yet) is map structs recursively. I'm not sure if this library should even do that, the containers for those inner structs can't really be supplied elegantly. If this was to be done - and i'd be happy to do it if that's the case, all structs would be initialized and decoded to map[string]interface{}. What do you think @mitchellh?

    Closes #53

    opened by mcos 12
  • No encoding ?

    No encoding ?

    mapstructure is a Go library for decoding generic map values to structures and vice versa

    At first glance, there is no way to encode a struct to a map with this library ..?

    opened by n00dl3 8
  • Accept string to time.Time as weak input

    Accept string to time.Time as weak input

    It would be nice if the import can handle strings with a timestamp or a date time format as string. Something like that: "2015-09-30T01:18:56.096224525+02:00"

    opened by mantenie 8
  • failed to decode config

    failed to decode config

    Use below code to reproduce:

    package main
    
    import (
    	"fmt"
    	"github.com/mitchellh/mapstructure"
    )
    
    type Config struct {
    	ClientConfig  clientConfig   `yaml:"client,omitempty"`
    	ClientConfigs []clientConfig `yaml:"clients,omitempty"`
    }
    
    type clientConfig struct {
    	URL string `"yaml:url,omitempty"`
    }
    
    func main() {
    
    	input := map[string]interface{}{
    		"client": map[string]string{
    			"url": "http://198.168.181.61:3100/api/prom/push",
    		},
    	}
    
    	var result Config
    	err := mapstructure.Decode(input, &result)
    	if err != nil {
    		panic(err)
    	}
    
    	fmt.Printf("%#v", result)
    
    }
    

    output:

    main.Config{ClientConfig:main.clientConfig{URL:""}, ClientConfigs:[]main.clientConfig(nil)}
    
    opened by daixiang0 7
  • Would you be open to support array types?

    Would you be open to support array types?

    I have added in a case to handle array type because I need to deal with uuid.UUID type

    	var err error
    	dataKind := getKind(val)
    	fmt.Println(name, val, dataKind)
    	switch dataKind {
    	case reflect.Bool:
    		err = d.decodeBool(name, data, val)
    	case reflect.Interface:
    		err = d.decodeBasic(name, data, val)
    	case reflect.String:
    		err = d.decodeString(name, data, val)
    	case reflect.Int:
    		err = d.decodeInt(name, data, val)
    	case reflect.Uint:
    		err = d.decodeUint(name, data, val)
    	case reflect.Float32:
    		err = d.decodeFloat(name, data, val)
    	case reflect.Struct:
    		err = d.decodeStruct(name, data, val)
    	case reflect.Map:
    		err = d.decodeMap(name, data, val)
    	case reflect.Ptr:
    		err = d.decodePtr(name, data, val)
    	case reflect.Slice:
    		err = d.decodeSlice(name, data, val)
    	case reflect.Array: // required to handle uuid.UUID type
    		err = d.decodeArray(name, data, val)
    	default:
    		// If we reached this point then we weren't able to decode it
    		return fmt.Errorf("%s: unsupported type: %s", name, dataKind)
    	}
    

    Would you accept this new feature?

    opened by calvinchengx 7
  • Support struct wrapped in interface as target for decoding

    Support struct wrapped in interface as target for decoding

    Similar to #186, mapstructure is not handling structs assigned to interfaces:

    var cc interface{}
    cc = struct{ Foo, Bar string }{}
    
    decoderConfig := &mapstructure.DecoderConfig{Result: &cc, ErrorUnused: true}
    decoder, err := mapstructure.NewDecoder(decoderConfig)
    if err != nil {
    	log.Fatal(err)
    }
    
    conf := map[string]string{
    	"foo": "bar",
    	"bar": "baz",
    }
    
    if err := decoder.Decode(conf); err != nil {
    	log.Fatal(err)
    }
    

    Will give

    * '' has invalid keys: bar, foo
    

    Would this be something worthwile to address with a PR or are there constraints that would prevent handling this case?

    opened by andig 6
  • Fix data race in decodeStruct

    Fix data race in decodeStruct

    Using *reflect.StructField as map key results in a data race. I honestly do not understand why yet and can only reproduce this in the consul codebase with high concurrency but this patch fixes it.

    The same problem also exists in hashicorp/hcl.

    https://github.com/hashicorp/hcl/pull/213

    opened by magiconair 6
  • Decode into map with case insensitive keys

    Decode into map with case insensitive keys

    I have a situation where I don't know the target struct and hence decode into a map. This will create duplicate keys if input has multiple types of key case. It would be helpful if I could deduplicate the keys in a case insensitive way.

    opened by andig 0
  • Add a recovering decode hook

    Add a recovering decode hook

    Hooks are difficult to get 100% correct. When there is an error in the code using reflection, we are likely to trigger a panic. While a panic is helpful to the developer, an end-user has no clue where the problem is and they don't know what part of the configuration would need to be tweaked to work around the issue.

    The proposed recovering decode hook will turn the panic into an error, which is a more user-friendly way to report the error to an end user.

    opened by vincentbernat 0
  • Proposal: add generic Unmarshaler interface

    Proposal: add generic Unmarshaler interface

    I've just had the requirement, to provide specific unmarshaling of some structures without modifying the unmarshaler configuration. I already have WeaklyTypedInput and DecodeHook/ComposeDecodeHookFunc.

    I'd like to propose implementing an UnmarshallerHookFunc DecodeHookFunc. UnmarshallerHookFunc would be called whenever the target implements mapstructure.Unmarshaler:

    type Unmarshaler interface {
    	UnmarshalMapStructure(f reflect.Type, data interface{}) error
    }
    

    Here's an example usage case. ValueDefinition can be supplied as simple string or as map and both will be decoded into an ValueDefinition.

    type ValueDefinition struct {
    	Value string
    	Scale float64
    }
    
    var _ util.MapStructureUnmarshaler = (*ValueDefinition)(nil)
    
    func (v *ValueDefinition) UnmarshalMapStructure(f reflect.Type, data interface{}) error {
    	switch f.Kind() {
    	case reflect.String:
    		*v = ValueDefinition{Value: data.(string)}
    		return nil
    
    	case reflect.Map:
    		var cc struct {
    			Value string
    			Scale float64
    		}
    
    		err := util.DecodeOther(data, &cc)
    		if err == nil {
    			*v = ValueDefinition{
    				Value: cc.Value,
    				Scale: cc.Scale,
    			}
    		}
    
    		return err
    
    	default:
    		return fmt.Errorf("invalid kind: %s", f.Kind())
    	}
    }
    

    If that sounds interesting I'd be happy to provide a PR.

    opened by andig 1
  • Decode underscore value

    Decode underscore value

    Hi I have Jason field k8s_version which contain underscore. However I have noticed that I am not able to decode values with _. Just wondering if its a bug or required me to add anything

    opened by mussa572 1
  • decode do not zero the field

    decode do not zero the field

    func TestIssue299(t *testing.T) { var a []interface{} inputStringSlice := map[string]interface{}{ "vfoo": "foo", "vbar": a, } var result = Slice{ Vfoo: "a", Vbar: []string{"a", "b"}, } config := &DecoderConfig{ Metadata: nil, ZeroFields: true, Result: &result, WeaklyTypedInput: true, }

    decoder, err := NewDecoder(config)
    if err != nil {
    	t.Fatalf("got error: %s", err)
    }
    
    err = decoder.Decode(inputStringSlice)
    if err != nil {
    	t.Fatalf("got error: %s", err)
    }
    if result.Vfoo != "foo" {
    	t.Errorf("want want get %s", result.Vfoo)
    }
    if len(result.Vbar) != 0 {
    	t.Errorf("want len(result.Vbar) == 0, get %d", len(result.Vbar))
    }
    

    }

    this will failed

    opened by cuiweixie 0
Owner
Mitchell Hashimoto
Mitchell Hashimoto
GED - Global-purpose Encoding / Decoding library

GED - Global-purpose Encoding / Decoding library This library lets you use common encoding/decoding schemes and allows you to define custom ones. Use

Noémien Kocher 0 Nov 28, 2021
An optimal, byte-aligned, LZ+RLE hybrid encoder, designed to maximize decoding speed on NMOS 6502 and derived CPUs

TSCrunch TSCrunch is an optimal, byte-aligned, LZ+RLE hybrid encoder, designed to maximize decoding speed on NMOS 6502 and derived CPUs, while keeping

null 26 Jun 10, 2022
Easily and dynamically generate maps from Go static structures

structomap This package helps you to transform your struct into map easily. It provides a structomap.Serializer interface implemented by the structoma

Daniel Perez 137 Sep 8, 2022
csvutil provides fast and idiomatic mapping between CSV and Go (golang) values.

csvutil Package csvutil provides fast and idiomatic mapping between CSV and Go (golang) values. This package does not provide a CSV parser itself, it

Jacek Szwec 753 Sep 28, 2022
generic sort for slices in golang

slices generic sort for slices in golang basic API func BinarySearch[E constraints.Ordered](list []E, x E) int func IsSorted[E constraints.Ordered](li

阮坤良 16 Aug 25, 2022
Golang binary decoder for mapping data into the structure

binstruct Golang binary decoder to structure Install go get -u github.com/ghostiam/binstruct Examples ZIP decoder PNG decoder Use For struct From file

Vladislav Fursov 60 Sep 20, 2022
Cap'n Proto library and parser for go. This is go-capnproto-1.0, and does not have rpc. See https://github.com/zombiezen/go-capnproto2 for 2.0 which has rpc and capabilities.

Version 1.0 vs 2.0 Update 2015 Sept 20: Big news! Version 2.0 of the go-bindings, authored by Ross Light, is now released and newly available! It feat

Jason E. Aten, Ph.D. 283 Sep 2, 2022
Asn.1 BER and DER encoding library for golang.

WARNING This repo has been archived! NO further developement will be made in the foreseen future. asn1 -- import "github.com/PromonLogicalis/asn1" Pac

Logicalis 51 Aug 3, 2022
A go library that facilitates the implementation of decorator pattern.

go-decorator go-decorator is a library that facilitates the implementation of decorator pattern. Installation To install go-decorator, use go get: go

Garen Chan 1 Nov 25, 2021
A library that provides dynamic features of Go language.

go-dynamic go-dynamic is a library that provides dynamic features of Go language. Installation To install go-dynamic, use go get: go get -u github.com

Garen Chan 1 Dec 8, 2021
Encode and decode binary message and file formats in Go

Encode and Decode Binary Formats in Go This module wraps the package encoding/binary of the Go standard library and provides the missing Marshal() and

Joel Ling 7 Aug 31, 2022
idiomatic codec and rpc lib for msgpack, cbor, json, etc. msgpack.org[Go]

go-codec This repository contains the go-codec library, the codecgen tool and benchmarks for comparing against other libraries. This is a High Perform

Ugorji Nwoke 1.7k Sep 12, 2022
Go package for dealing with maps, slices, JSON and other data.

Objx Objx - Go package for dealing with maps, slices, JSON and other data. Get started: Install Objx with one line of code, or update it with another

Stretchr, Inc. 545 Sep 23, 2022
Encode and decode Go (golang) struct types via protocol buffers.

protostructure protostructure is a Go library for encoding and decoding a struct type over the wire. This library is useful when you want to send arbi

Mitchell Hashimoto 172 Sep 9, 2022
Simple, specialised, and efficient binary marshaling

?? surge Documentation A library for fast binary (un)marshaling. Designed to be used in Byzantine networks, ?? surge never explicitly panics, protects

Ren 38 Mar 3, 2022
Go library for decoding generic map values into native Go structures and vice versa.

mapstructure mapstructure is a Go library for decoding generic map values to structures and vice versa, while providing helpful error handling. This l

Mitchell Hashimoto 6.2k Sep 27, 2022
Go library for encoding native Go structures into generic map values.

wstructs origin: github.com/things-go/structs Go library for encoding native Go structures into generic map values. Installation Use go get. go ge

null 0 Jan 10, 2022
:steam_locomotive: Decodes url.Values into Go value(s) and Encodes Go value(s) into url.Values. Dual Array and Full map support.

Package form Package form Decodes url.Values into Go value(s) and Encodes Go value(s) into url.Values. It has the following features: Supports map of

Go Playgound 556 Sep 21, 2022
Simple Relay between a Unix socket and a TCP socket, and vice versa.

Simple TCP <-> Unix Relay simpletcpunixrelay is a program which exposes a TCP endpoint as a Unix socket and vice versa. Usecase Let's say you are runn

Antoine Catton 0 Dec 4, 2021
Lookup or replace AWS account IDs with their names and vice versa

awsacc A trusty helper for working with AWS account IDs. Working with AWS account IDs often involves more manual effort than necessary. Often account

Christian Bargmann 3 Oct 14, 2021