Light weight, extensible configuration management library for Go. Built in support for JSON, TOML, YAML, env, command line, file, S3 etc. Alternative to viper.

Overview

koanf

koanf (pronounced conf; a play on the Japanese Koan) is a library for reading configuration from different sources in different formats in Go applications. It is a cleaner, lighter alternative to spf13/viper with better abstractions and extensibility and fewer dependencies.

koanf comes with built in support for reading configuration from files, command line flags, and environment variables, and can parse JSON, YAML, TOML, and Hashicorp HCL.

Build Status GoDoc

Installation

go get -u github.com/knadh/koanf

Contents

Concepts

  • koanf.Provider is a generic interface that provides configuration, for example, from files, environment variables, HTTP sources, or anywhere. The configuration can either be raw bytes that a parser can parse, or it can be a nested map[string]interface{} that can be directly loaded.
  • koanf.Parser is a generic interface that takes raw bytes, parses, and returns a nested map[string]interface{} representation. For example, JSON and YAML parsers.
  • Once loaded into koanf, configuration are values queried by a delimited key path syntax. eg: app.server.port. Any delimiter can be chosen.
  • Configuration from multiple sources can be loaded and merged into a koanf instance, for example, load from a file first and override certain values with flags from the command line.

With these two interface implementations, koanf can obtain a configuration from multiple sources and parse any format and make it available to an application.

Reading config from files

package main

import (
	"fmt"
	"log"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/parsers/yaml"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

func main() {
	// Load JSON config.
	if err := k.Load(file.Provider("mock/mock.json"), json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Load YAML config and merge into the previously loaded config (because we can).
	k.Load(file.Provider("mock/mock.yaml"), yaml.Parser())

	fmt.Println("parent's name is = ", k.String("parent1.name"))
	fmt.Println("parent's ID is = ", k.Int("parent1.id"))
}

Watching files for changes

The koanf.Provider interface has a Watch(cb) method that asks a provider to watch for changes and trigger the given callback that can live reload the configuration.

Currently, file.Provider supports this.

package main

import (
	"fmt"
	"log"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/parsers/yaml"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

func main() {
	// Load JSON config.
	f := file.Provider("mock/mock.json")
	if err := k.Load(f, json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Load YAML config and merge into the previously loaded config (because we can).
	k.Load(file.Provider("mock/mock.yaml"), yaml.Parser())

	fmt.Println("parent's name is = ", k.String("parent1.name"))
	fmt.Println("parent's ID is = ", k.Int("parent1.id"))

	// Watch the file and get a callback on change. The callback can do whatever,
	// like re-load the configuration.
	// File provider always returns a nil `event`.
	f.Watch(func(event interface{}, err error) {
		if err != nil {
			log.Printf("watch error: %v", err)
			return
		}

		log.Println("config changed. Reloading ...")
		k.Load(f, json.Parser())
		k.Print()
	})

	// Block forever (and manually make a change to mock/mock.json) to
	// reload the config.
	log.Println("waiting forever. Try making a change to mock/mock.json to live reload")
	<-make(chan bool)
}

Reading from command line

The following example shows the use of posflag.Provider, a wrapper over spf13/pflag library, an advanced commandline lib. For Go's built in flag package, use basicflag.Provider.

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/toml"
	"github.com/knadh/koanf/providers/file"
	"github.com/knadh/koanf/providers/posflag"
	flag "github.com/spf13/pflag"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

func main() {
	// Use the POSIX compliant pflag lib instead of Go's flag lib.
	f := flag.NewFlagSet("config", flag.ContinueOnError)
	f.Usage = func() {
		fmt.Println(f.FlagUsages())
		os.Exit(0)
	}
	// Path to one or more config files to load into koanf along with some config params.
	f.StringSlice("conf", []string{"mock/mock.toml"}, "path to one or more .toml config files")
	f.String("time", "2020-01-01", "a time string")
	f.String("type", "xxx", "type of the app")
	f.Parse(os.Args[1:])

	// Load the config files provided in the commandline.
	cFiles, _ := f.GetStringSlice("conf")
	for _, c := range cFiles {
		if err := k.Load(file.Provider(c), toml.Parser()); err != nil {
			log.Fatalf("error loading file: %v", err)
		}
	}

	// "time" and "type" may have been loaded from the config file, but
	// they can still be overridden with the values from the command line.
	// The bundled posflag.Provider takes a flagset from the spf13/pflag lib.
	// Passing the Koanf instance to posflag helps it deal with default command
	// line flag values that are not present in conf maps from previously loaded
	// providers.
	if err := k.Load(posflag.Provider(f, ".", k), nil); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	fmt.Println("time is = ", k.String("time"))
}

Reading environment variables

package main

import (
	"fmt"
	"log"
	"strings"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/providers/env"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use . as the key path delimiter. This can be / or anything.
var k = koanf.New(".")

func main() {
	// Load JSON config.
	if err := k.Load(file.Provider("mock/mock.json"), json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Load environment variables and merge into the loaded config.
	// "MYVAR" is the prefix to filter the env vars by.
	// "." is the delimiter used to represent the key hierarchy in env vars.
	// The (optional, or can be nil) function can be used to transform
	// the env var names, for instance, to lowercase them.
	//
	// For example, env vars: MYVAR_TYPE and MYVAR_PARENT1_CHILD1_NAME
	// will be merged into the "type" and the nested "parent1.child1.name"
	// keys in the config file here as we lowercase the key, 
	// replace `_` with `.` and strip the MYVAR_ prefix so that 
	// only "parent1.child1.name" remains.
	k.Load(env.Provider("MYVAR_", ".", func(s string) string {
		return strings.Replace(strings.ToLower(
			strings.TrimPrefix(s, "MYVAR_")), "_", ".", -1)
	}), nil)

	fmt.Println("name is = ", k.String("parent1.child1.name"))
}

You can also use the env.ProviderWithValue with a callback that supports mutating both the key and value to return types other than a string.

	// Load environment variables and merge into the loaded config.
	// "MYVAR" is the prefix to filter the env vars by just as the example above but
	// in this case we check the existing type of the value (if it exists)
	// If it's a slice, we convert the environment variable to a string slice
	k.Load(env.ProviderWithValue("MYVAR_", ".", func(s string, v string) (string, interface{}) {
		// Convert the environment variable key to the koanf key format
		key := strings.Replace(strings.ToLower(strings.TrimPrefix(s, "MYVAR_")), "_", ".", -1)
		// Check to exist if we have a configuration option already and see if it's a slice
		switch k.Get(key).(type) {
		case []interface{}, []string:
			// Convert our environment variable to a slice by splitting on space
			return key, strings.Split(v, " ")
		}
		return key, s // Otherwise return the new key with the unaltered value
	}), nil)
}

Reading from an S3 bucket

// Load JSON config from s3.
if err := k.Load(s3.Provider(s3.Config{
	AccessKey: os.Getenv("AWS_S3_ACCESS_KEY"),
	SecretKey: os.Getenv("AWS_S3_SECRET_KEY"),
	Region:    os.Getenv("AWS_S3_REGION"),
	Bucket:    os.Getenv("AWS_S3_BUCKET"),
	ObjectKey: "dir/config.json",
}), json.Parser()); err != nil {
	log.Fatalf("error loading config: %v", err)
}

Reading raw bytes

The bundled rawbytes Provider can be used to read arbitrary bytes from a source, like a database or an HTTP call.

package main

import (
	"fmt"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/providers/rawbytes"
)

// Global koanf instance. Use . as the key path delimiter. This can be / or anything.
var k = koanf.New(".")

func main() {
	b := []byte(`{"type": "rawbytes", "parent1": {"child1": {"type": "rawbytes"}}}`)
	k.Load(rawbytes.Provider(b), json.Parser())
	fmt.Println("type is = ", k.String("parent1.child1.type"))
}

Unmarshalling and marshalling

Parsers can be used to unmarshal and scan the values in a Koanf instance into a struct based on the field tags, and to marshal a Koanf instance back into serialized bytes, for example, back to JSON or YAML, to write back to files.

package main

import (
	"fmt"
	"log"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use . as the key path delimiter. This can be / or anything.
var (
	k      = koanf.New(".")
	parser = json.Parser()
)

func main() {
	// Load JSON config.
	if err := k.Load(file.Provider("mock/mock.json"), parser); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Structure to unmarshal nested conf to.
	type childStruct struct {
		Name       string            `koanf:"name"`
		Type       string            `koanf:"type"`
		Empty      map[string]string `koanf:"empty"`
		GrandChild struct {
			Ids []int `koanf:"ids"`
			On  bool  `koanf:"on"`
		} `koanf:"grandchild1"`
	}

	var out childStruct

	// Quick unmarshal.
	k.Unmarshal("parent1.child1", &out)
	fmt.Println(out)

	// Unmarshal with advanced config.
	out = childStruct{}
	k.UnmarshalWithConf("parent1.child1", &out, koanf.UnmarshalConf{Tag: "koanf"})
	fmt.Println(out)

	// Marshal the instance back to JSON.
	// The paser instance can be anything, eg: json.Paser(), yaml.Parser() etc.
	b, _ := k.Marshal(parser)
	fmt.Println(string(b))
}

Unmarshalling with flat paths

Sometimes it is necessary to unmarshal an assortment of keys from various nested structures into a flat target structure. This is possible with the UnmarshalConf.FlatPaths flag.

package main

import (
	"fmt"
	"log"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/providers/file"
)

// Global koanf instance. Use . as the key path delimiter. This can be / or anything.
var k = koanf.New(".")

func main() {
	// Load JSON config.
	if err := k.Load(file.Provider("mock/mock.json"), json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	type rootFlat struct {
		Type                        string            `koanf:"type"`
		Empty                       map[string]string `koanf:"empty"`
		Parent1Name                 string            `koanf:"parent1.name"`
		Parent1ID                   int               `koanf:"parent1.id"`
		Parent1Child1Name           string            `koanf:"parent1.child1.name"`
		Parent1Child1Type           string            `koanf:"parent1.child1.type"`
		Parent1Child1Empty          map[string]string `koanf:"parent1.child1.empty"`
		Parent1Child1Grandchild1IDs []int             `koanf:"parent1.child1.grandchild1.ids"`
		Parent1Child1Grandchild1On  bool              `koanf:"parent1.child1.grandchild1.on"`
	}

	// Unmarshal the whole root with FlatPaths: True.
	var o1 rootFlat
	k.UnmarshalWithConf("", &o1, koanf.UnmarshalConf{Tag: "koanf", FlatPaths: true})
	fmt.Println(o1)

	// Unmarshal a child structure of "parent1".
	type subFlat struct {
		Name                 string            `koanf:"name"`
		ID                   int               `koanf:"id"`
		Child1Name           string            `koanf:"child1.name"`
		Child1Type           string            `koanf:"child1.type"`
		Child1Empty          map[string]string `koanf:"child1.empty"`
		Child1Grandchild1IDs []int             `koanf:"child1.grandchild1.ids"`
		Child1Grandchild1On  bool              `koanf:"child1.grandchild1.on"`
	}

	var o2 subFlat
	k.UnmarshalWithConf("parent1", &o2, koanf.UnmarshalConf{Tag: "koanf", FlatPaths: true})
	fmt.Println(o2)
}

Marshalling and writing config

It is possible to marshal and serialize the conf map into TOML, YAML etc.

Setting default values.

koanf does not provide any special functions to set default values but uses the Provider interface to enable it.

From a map

The bundled confmap provider takes a map[string]interface{} that can be loaded into a koanf instance.

package main

import (
	"fmt"
	"log"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/providers/confmap"
	"github.com/knadh/koanf/providers/file"
	"github.com/knadh/koanf/parsers/json"
	"github.com/knadh/koanf/parsers/yaml"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

func main() {
	// Load default values using the confmap provider.
	// We provide a flat map with the "." delimiter.
	// A nested map can be loaded by setting the delimiter to an empty string "".
	k.Load(confmap.Provider(map[string]interface{}{
		"parent1.name": "Default Name",
		"parent3.name": "New name here",
	}, "."), nil)

	// Load JSON config on top of the default values.
	if err := k.Load(file.Provider("mock/mock.json"), json.Parser()); err != nil {
		log.Fatalf("error loading config: %v", err)
	}

	// Load YAML config and merge into the previously loaded config (because we can).
	k.Load(file.Provider("mock/mock.yaml"), yaml.Parser())

	fmt.Println("parent's name is = ", k.String("parent1.name"))
	fmt.Println("parent's ID is = ", k.Int("parent1.id"))
}

From a struct

The bundled structs provider can be used to read data from a struct to load into a koanf instance.

package main

import (
	"fmt"

	"github.com/knadh/koanf"
	"github.com/knadh/koanf/providers/structs"
)

// Global koanf instance. Use "." as the key path delimiter. This can be "/" or any character.
var k = koanf.New(".")

type parentStruct struct {
	Name   string      `koanf:"name"`
	ID     int         `koanf:"id"`
	Child1 childStruct `koanf:"child1"`
}
type childStruct struct {
	Name        string            `koanf:"name"`
	Type        string            `koanf:"type"`
	Empty       map[string]string `koanf:"empty"`
	Grandchild1 grandchildStruct  `koanf:"grandchild1"`
}
type grandchildStruct struct {
	Ids []int `koanf:"ids"`
	On  bool  `koanf:"on"`
}
type sampleStruct struct {
	Type    string            `koanf:"type"`
	Empty   map[string]string `koanf:"empty"`
	Parent1 parentStruct      `koanf:"parent1"`
}

func main() {
	// Load default values using the structs provider.
	// We provide a struct along with the struct tag `koanf` to the
	// provider.
	k.Load(structs.Provider(sampleStruct{
		Type:  "json",
		Empty: make(map[string]string),
		Parent1: parentStruct{
			Name: "parent1",
			ID:   1234,
			Child1: childStruct{
				Name:  "child1",
				Type:  "json",
				Empty: make(map[string]string),
				Grandchild1: grandchildStruct{
					Ids: []int{1, 2, 3},
					On:  true,
				},
			},
		},
	}, "koanf"), nil)

	fmt.Printf("name is = `%s`\n", k.String("parent1.child1.name"))
}

Order of merge and key case senstivity

  • Config keys are case-sensitive in koanf. For example, app.server.port and APP.SERVER.port are not the same.
  • koanf does not impose any ordering on loading config from various providers. Every successive Load() or Load() merges new config into existing config. That means it is possible to load environment variables first, then files on top of it, and then command line variables on top of it, or any such order.

Custom Providers and Parsers

A Provider can provide a nested map[string]interface{} config that can be loaded into koanf with koanf.Load() or raw bytes that can be parsed with a Parser (loaded using koanf.Load().

Writing Providers and Parsers are easy. See the bundled implementations in the providers and parses directory.

API

Bundled providers

Package Provider Description
providers/file file.Provider(filepath string) Reads a file and returns the raw bytes to be parsed.
providers/basicflag basicflag.Provider(f *flag.FlagSet, delim string) Takes an stdlib flag.FlagSet
providers/posflag posflag.Provider(f *pflag.FlagSet, delim string) Takes an spft3/pflag.FlagSet (advanced POSIX compatible flags with multiple types) and provides a nested config map based on delim.
providers/env env.Provider(prefix, delim string, f func(s string) string) Takes an optional prefix to filter env variables by, an optional function that takes and returns a string to transform env variables, and returns a nested config map based on delim.
providers/confmap confmap.Provider(mp map[string]interface{}, delim string) Takes a premade map[string]interface{} conf map. If delim is provided, the keys are assumed to be flattened, thus unflattened using delim.
providers/structs structs.Provider(s interface{}, tag string) Takes a struct and struct tag.
providers/s3 s3.Provider(s3.S3Config{}) Takes a s3 config struct.
providers/rawbytes rawbytes.Provider(b []byte) Takes a raw []byte slice to be parsed with a koanf.Parser
providers/vault vault.Provider(vault.Config{}) Hashicorp Vault provider

Bundled parsers

Package Parser Description
parsers/json json.Parser() Parses JSON bytes into a nested map
parsers/yaml yaml.Parser() Parses YAML bytes into a nested map
parsers/toml toml.Parser() Parses TOML bytes into a nested map
parsers/dotenv dotenv.Parser() Parses DotEnv bytes into a flat map
parsers/hcl hcl.Parser(flattenSlices bool) Parses Hashicorp HCL bytes into a nested map. flattenSlices is recommended to be set to true. Read more.

Instance functions

Method Description
Load(p Provider, pa Parser) error Loads config from a Provider. If a koanf.Parser is provided, the config is assumed to be raw bytes that's then parsed with the Parser.
Keys() []string Returns the list of flattened key paths that can be used to access config values
KeyMap() map[string][]string Returns a map of all possible key path combinations possible in the loaded nested conf map
All() map[string]interface{} Returns a flat map of flattened key paths and their corresponding config values
Raw() map[string]interface{} Returns a copy of the raw nested conf map
Print() Prints a human readable copy of the flattened key paths and their values for debugging
Sprint() Returns a human readable copy of the flattened key paths and their values for debugging
Cut(path string) *Koanf Cuts the loaded nested conf map at the given path and returns a new Koanf instance with the children
Copy() *Koanf Returns a copy of the Koanf instance
Merge(*Koanf) Merges the config map of a Koanf instance into the current instance
Delete(path string) Delete the value at the given path, and does nothing if path doesn't exist.
MergeAt(in *Koanf, path string) Merges the config map of a Koanf instance into the current instance, at the given key path.
Unmarshal(path string, o interface{}) error Scans the given nested key path into a given struct (like json.Unmarshal) where fields are denoted by the koanf tag
UnmarshalWithConf(path string, o interface{}, c UnmarshalConf) error Like Unmarshal but with customizable options

Getter functions

Get(path string) interface{} Returns the value for the given key path, and if it doesn’t exist, returns nil
Exists(path string) bool Returns true if the given key path exists in the conf map
Int64(path string) int64
Int64s(path string) []int64
Int64Map(path string) map[string]int64
Int(path string) int
Ints(path string) []int
IntMap(path string) map[string]int
Float64(path string) float64
Float64s(path string) []float64
Float64Map(path string) map[string]float64
Duration(path string) time.Duration Returns the time.Duration value of the given key path if it’s numeric (attempts a parse+convert if string) or a string representation like "3s".
Time(path, layout string) time.Time Parses the string value of the the given key path with the given layout format and returns time.Time. If the key path is numeric, treats it as a UNIX timestamp and returns its time.Time.
String(path string) string
Strings(path string) []string
StringMap(path string) map[string]string
StringsMap(path string) map[string][]string
Byte(path string) []byte
Bool(path string) bool
Bools(path string) []bool
BoolMap(path string) map[string]bool
MapKeys(path string) []string Returns the list of keys in any map
Slices(path string) []Koanf Returns []map[string]interface{}, a slice of confmaps loaded into a slice of new Koanf instances.

Alternative to viper

koanf is a lightweight alternative to the popular spf13/viper. It was written as a result of multiple stumbling blocks encountered with some of viper's fundamental flaws.

  • viper breaks JSON, YAML, TOML, HCL language specs by forcibly lowercasing keys.
  • Significantly bloats build sizes.
  • Tightly couples config parsing with file extensions.
  • Has poor semantics and abstractions. Commandline, env, file etc. and various parses are hardcoded in the core. There are no primitives that can be extended.
  • Pulls a large number of third party dependencies into the core package. For instance, even if you do not use YAML or flags, the dependencies are still pulled as a result of the coupling.
  • Imposes arbitrary ordering conventions (eg: flag -> env -> config etc.)
  • Get() returns references to slices and maps. Mutations made outside change the underlying values inside the conf map.
  • Does non-idiomatic things such as throwing away O(1) on flat maps.
  • There are a large number of open issues.
Comments
  • Add ability to escape delimters.

    Add ability to escape delimters.

    Through this PR, you can escape the delim in the key, like this:

    map.Unflatten(map[string]interface{}{"this.is..an..escaped.key": true}, ".", "..") 
    

    will give you

    map[string]interface{}{
      "this": map[string]interface{}{
        "is.an.escaped": map[string]interface{}{
          "key": true,
        },
      },
    }
    
    opened by infalmo 19
  • Reconsider internal use of `keyMap`

    Reconsider internal use of `keyMap`

    We are using koanf in production for a multi-tenant system and are having serious issues with memory consumption when a number (~5000) of configs is loaded that have a medium amount of config values (~85 flattened key/value pairs) with memory spikes of up to 7 GB.

    Benchmarking and memory profiling shows that a lot of time is spent in maps.Flatten and koanf.populateKeyParts hence my two recent PRs:

    Bildschirmfoto 2021-06-07 um 13 34 44

    I think some of this boils down to KeyMap which contains the flattened keys in string and slice form. I looked through the code and it appears that the feature is not really used, or where it is used, strings.Split could be used instead which would probably reduce the memory count quite a bit because we avoid keeping these in memory:

    koanf.keyMap = map[string][]string{
      "foo": []string{"foo"},
      "foo.bar": []string{"foo", "bar"},
      "foo.bar.baz1": []string{"foo", "bar", "baz1"},
      "foo.bar.baz2": []string{"foo", "bar", "baz2"},
      "foo.bar.baz3": []string{"foo", "bar", "baz3"},
    }
    

    As you can see, the amount of memory explodes the more nested keys are used because the same keyparts are copied over and over. As Koanf strictly uses slice copying to avoid modification (I assume?) this leads to a lot of unnecessary memory use.

    The performance impact of using strings.Split("foo.bar.baz3", delimiter) should be negligible compared to the amount of data kept in memory.

    The only place where we would need to generate this keymap as it exists right now would be the KeyMap() function if we want to keep compatibility.

    opened by aeneasr 16
  • Question about race condition

    Question about race condition

    Hi,

    I am wondering whether we may have a race condition when the Watch method of a provider is called. I imagine that something wrong may happen when the configuration is read and reloaded at the same time. If it's possible, how would you advise me to make it thread safe?

    Thanks a lot

    question 
    opened by asiffer 14
  • posflag: add support for flags that implement the flag.Getter interface from stdlib

    posflag: add support for flags that implement the flag.Getter interface from stdlib

    A proposed implementation of

    • https://github.com/knadh/koanf/issues/102

    Let me know what you think of this approach. If it looks acceptable, I can add tests and docs.

    opened by e-nikolov 9
  • Remove keyMap internal from Koanf

    Remove keyMap internal from Koanf

    As outlined in #80, Koanf keeps track of KeyMap internally even though the representation is not really needed. KeyMaps use a significant amount of memory depending on the config layout which can lead to excessive memory consumptions for large or complex configs.

    This patch removes the need to keep keyMap around and therefore significantly improves performance:

    benchmark                old ns/op     new ns/op     delta
    BenchmarkLoadFile-16     146821        135027        -8.03%
    
    benchmark                old allocs     new allocs     delta
    BenchmarkLoadFile-16     679            657            -3.24%
    
    benchmark                old bytes     new bytes     delta
    BenchmarkLoadFile-16     46277         37019         -20.01%
    

    BREAKING CHANGE: This PR deprecates KeyMap function. It still works as before but is slower.

    It also removes the second return argument of maps.Flatten which was the KeyMap previously.

    opened by aeneasr 9
  • Default empty []string unmarshalls as a nil

    Default empty []string unmarshalls as a nil

    When I have a string slice that defaults (via flags) to empty slice ([]string) then it gets unmarshalled as nil.

    This isn't really a bug in koanf but in https://github.com/mitchellh/mapstructure/issues/146 Reporting here since koanf uses mapstructure directly. Is there any way to workaround this in koanf? mitchellh/mapstructure didn't get any commit in a while so looks like the issue might not get fixed, in that case is there an alternative koanf could use?

    opened by prymitive 9
  • Custom error handler

    Custom error handler

    When validating type of values passed via flags or config file I have the choice of either k.Int() or k.MustInt(), first one will default to 0 if the value is incorrect, second will panic on parse error. What I'm missing is a way of validating passed values and reporting an error to the user without the need to panic, there is no variant of a getter that returns an error from what I see. Is it possible to setup custom error handler? Perhaps one could register error handling function that's called on MustInt() to avoid polluting API with yet another version of a Int() getter?

    opened by prymitive 9
  • Watch example in README does not remove keys / values

    Watch example in README does not remove keys / values

    The current README example around file watching could be improved by making the limitations of that approach clear. Removing a value from the config file (e.g. parent1.id) will not remove the value from the config but keep the old value around. It would probably be expected for the removed value to be no longer available in the config as it was removed from the file.

    Example

    Run program and have mock.json with:

    {
      "parent1": {
        "name": "foo",
        "id": 34
      }
    }
    
    % go run .
    parent's name is =  bar
    parent's ID is =  34
    

    Remove all keys from mock.json

    {}
    

    Config reloads, but values are still present:

    2021/06/05 17:43:19 config changed. Reloading ...
    parent1.id -> 34
    parent1.name -> bar
    
    documentation 
    opened by aeneasr 8
  • How to provide string slice from environment variables

    How to provide string slice from environment variables

    Any thoughts on how to provide a string slice from an environment variable? Viper looks at the existing value and if the default is a slice it interprets the environment variable as a slice. It almost would have been better to return (string, interface{}) from the callback and allow the callback to manipulate the value as well.

    enhancement 
    opened by snowzach 8
  • Values not being loaded

    Values not being loaded

    I'm working with koanf and it looks like maps and some custom values are not being loaded properly. They appear to be empty even if the toml config file I'm using has values specified.

    Is there a delay or similar on reading values from a config file?

    bug 
    opened by mcrosson 8
  • unable to parse int64 with leading zeros

    unable to parse int64 with leading zeros

    Describe the bug Looks like koanf uses the standard go yaml.v2 library to parse YAML, but it is behaving differently than expected. Specifically, int64 values with leading zeros are not being parsed/converted correctly and it is throwing an error.

    To Reproduce sample code:

    package main
    
    import (
    	"log"
    
    	"github.com/davecgh/go-spew/spew"
    	"github.com/knadh/koanf"
    	"github.com/knadh/koanf/parsers/yaml"
    	"github.com/knadh/koanf/providers/rawbytes"
    	goyaml "gopkg.in/yaml.v2"
    )
    
    type Example struct {
    	Config map[string]*SubConfig `koanf:"Config" yaml:"Config"`
    }
    
    type SubConfig struct {
    	Accounts map[int64]string `koanf:"Accounts" yaml:"Accounts"`
    }
    
    var data = `
    Config:
      Foo:
        Accounts:
          0012345678912: FancyBear
    `
    
    func main() {
    	// plain yaml decode which works fine
    	example := Example{}
    	if err := goyaml.Unmarshal([]byte(data), &example); err != nil {
    		log.Fatalf("error: %v", err)
    	}
    	log.Printf("yaml decode: %s", spew.Sdump(example))
    
    	// koanf decode whieh is fatal
    	konf := koanf.New(".")
    	s := &Example{}
    	if err := konf.Load(rawbytes.Provider([]byte(data)), yaml.Parser()); err != nil {
    		log.Fatalf("Unable to load data: %s", err.Error())
    	}
    
    	if err := konf.Unmarshal("", s); err != nil {
    		log.Fatalf("Unable to unmarshal data: %s", err.Error())
    	}
    }
    
    $ go run .
    2022/02/23 20:04:38 yaml decode: (main.Example) {
     Config: (map[string]*main.SubConfig) (len=1) {
      (string) (len=3) "Foo": (*main.SubConfig)(0xc000010088)({
       Accounts: (map[int64]string) (len=1) {
        (int64) 12345678912: (string) (len=9) "FancyBear"
       }
      })
     }
    }
    2022/02/23 20:04:38 Unable to unmarshal data: 1 error(s) decoding:
    
    * cannot parse 'Config[Foo].Accounts[1.2345678912e+10]' as int: strconv.ParseInt: parsing "1.2345678912e+10": invalid syntax
    exit status 1
    

    Expected behavior Decode successfully and convert 0012345678912 => 12345678912

    Please provide the following information):

    • OS: macOS
    • Koanf Version 1.4.0

    Additional context

    I get a slightly different error if I simplify the Example struct above into just:

    type Example map[int64]string

    which generates:

    * cannot parse '[0012345678912]' as int: strconv.ParseInt: parsing "0012345678912": invalid syntax

    question 
    opened by synfinatic 7
  • fatih/structs is not maintained anymore

    fatih/structs is not maintained anymore

    The maintainer of fatih/structs has stopped maintaining the package.

    I think it would be beneficial to replace this package with a maintained alternative.

    • https://github.com/fatih/structs
    • https://arslan.io/2018/10/09/taking-an-indefinite-sabbatical-from-my-projects/
    enhancement 
    opened by stv0g 2
Releases(v1.4.2)
  • v1.4.2(Jun 17, 2022)

    What's Changed

    • Document another viper footgun by @albertvaka in https://github.com/knadh/koanf/pull/141
    • Add missing types to FlagVal by @muXxer in https://github.com/knadh/koanf/pull/147
    • Add github workflow status badge in Readme by @amarlearning in https://github.com/knadh/koanf/pull/148
    • Make copyable go command by @gozeloglu in https://github.com/knadh/koanf/pull/149
    • Add test for YAML parser by @amarlearning in https://github.com/knadh/koanf/pull/150
    • Add test for HCL parser by @amarlearning in https://github.com/knadh/koanf/pull/151
    • Add test for TOML parser by @amarlearning in https://github.com/knadh/koanf/pull/155
    • Update gopkg.in/yaml.v2 to gopkg.in/yaml.v3 by @1995parham in https://github.com/knadh/koanf/pull/156

    New Contributors

    • @albertvaka made their first contribution in https://github.com/knadh/koanf/pull/141
    • @muXxer made their first contribution in https://github.com/knadh/koanf/pull/147
    • @amarlearning made their first contribution in https://github.com/knadh/koanf/pull/148

    Full Changelog: https://github.com/knadh/koanf/compare/v1.4.1...v1.4.2

    Source code(tar.gz)
    Source code(zip)
  • v1.4.1(Apr 8, 2022)

    What's Changed

    • fix nil pointer dereference when no callback passed to provider by @robinbraemer in https://github.com/knadh/koanf/pull/129
    • eliminate a type assertion by @robinbraemer in https://github.com/knadh/koanf/pull/130
    • fix: highlight code in README by @gozeloglu in https://github.com/knadh/koanf/pull/131
    • Add tests for env.go by @gozeloglu in https://github.com/knadh/koanf/pull/132
    • Add tests for ProviderWithValue func in env.go by @gozeloglu in https://github.com/knadh/koanf/pull/133
    • Tests added for Read() in env by @gozeloglu in https://github.com/knadh/koanf/pull/134
    • Add tests for dotenv by @gozeloglu in https://github.com/knadh/koanf/pull/137
    • Add tests for parsers/json by @gozeloglu in https://github.com/knadh/koanf/pull/136
    • Fix StringMap and StringsMap by @hoshsadiq in https://github.com/knadh/koanf/pull/139

    New Contributors

    • @robinbraemer made their first contribution in https://github.com/knadh/koanf/pull/129
    • @gozeloglu made their first contribution in https://github.com/knadh/koanf/pull/131
    • @hoshsadiq made their first contribution in https://github.com/knadh/koanf/pull/139

    Full Changelog: https://github.com/knadh/koanf/compare/v1.4.0...v1.4.1

    Source code(tar.gz)
    Source code(zip)
  • v1.4.0(Jan 2, 2022)

    What's Changed

    • feat(appconfig): add new ProviderWithClient initialiser by @ollystephens in https://github.com/knadh/koanf/pull/124
    • Add variadic options parameter to Load by @aeneasr in https://github.com/knadh/koanf/pull/128

    New Contributors

    • @ollystephens made their first contribution in https://github.com/knadh/koanf/pull/124

    Full Changelog: https://github.com/knadh/koanf/compare/v1.3.3...v1.4.0

    Source code(tar.gz)
    Source code(zip)
  • v1.3.3(Nov 24, 2021)

    What's Changed

    • Add nestedtext parser by @npillmayer in https://github.com/knadh/koanf/pull/121
    • feat: add optional delim to structs provider by @rhnvrm in https://github.com/knadh/koanf/pull/123

    New Contributors

    • @npillmayer made their first contribution in https://github.com/knadh/koanf/pull/121

    Full Changelog: https://github.com/knadh/koanf/compare/v1.3.2...v1.3.3

    Source code(tar.gz)
    Source code(zip)
  • v1.3.2(Nov 3, 2021)

    What's Changed

    • Refactor the posflag provider to make pflag parsing pluggable. by @knadh in https://github.com/knadh/koanf/pull/117

    Full Changelog: https://github.com/knadh/koanf/compare/v1.3.1...v1.3.2

    Source code(tar.gz)
    Source code(zip)
  • v1.3.0(Oct 18, 2021)

    Add AWS AppConfig provider.

    What's Changed

    • feat: add provider for AWS AppConfig by @Thunderbottom in https://github.com/knadh/koanf/pull/112

    New Contributors

    • @Thunderbottom made their first contribution in https://github.com/knadh/koanf/pull/112

    Full Changelog: https://github.com/knadh/koanf/compare/v1.2.4...v1.3.0

    Source code(tar.gz)
    Source code(zip)
  • v1.2.1(Jul 28, 2021)

    changelog for v1.2.0 -> v1.2.1

    • 29cce50 Merge pull request #101 from e-nikolov/fix-pflag-map-types
    • 0202243 posflag: add support for pflag map types
    Source code(tar.gz)
    Source code(zip)
  • v1.2.0(Jul 27, 2021)

    • 5fc619c Add Delim()
    • f001ae3 Merge pull request #98 from knadh/fix-confmap-dep
    • bc5fe47 Remove superfluous dependency on providers/confmap in the core
    Source code(tar.gz)
    Source code(zip)
  • v1.1.1(Jun 22, 2021)

    • 57e89cf Merge pull request #94 from knadh/fix-posflag-callbacks
    • d3bad45 Merge pull request #91 from knadh/bug-report-template
    • b1985ad Merge pull request #93 from rhnvrm/unit-test-issue-90
    • a72baa3 Merge pull request #92 from knadh/fix-posflag-callbacks
    • adc0b54 tests: add unit test for #90
    • 9e87eba Fix unchanged default posflag value merging via incorrect callback.
    • 2f82a14 Merge pull request #87 from mvrahden/feature/filesystem
    • 6e9c0a7 Add github issue templates
    • 023ecc6 Update README with fs.FS provider
    • f7e28d0 fix call to Close on file descriptor
    • caeef66 Merge pull request #1 from rhnvrm/feature/filesystem
    • 02f42ea fix: add build constraints to maintain compat
    • 149144c add fs.FS provider
    • 139fe40 Merge pull request #85 from mvrahden/fix/base_test
    • 7cc3416 move error assertions and make them error-type dependent
    • 0a31536 fix race condition and reduce test time by half
    • af2e3f4 simplify and unify test arrangements
    • aa62c04 update testify to v1.7.0
    • 5234867 Merge pull request #84 from mr-karan/gh_actions
    • c79fbc5 feat: Replace Travis with GitHub Actions
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Jun 16, 2021)

    • 036e4bf Merge pull request #79 from aeneasr/fix-memory-keyparts
    • 6692d05 Merge pull request #78 from aeneasr/fix-memory-use
    • d174d00 Reduce memory consumption for populateKeyParts
    • 1095e5e Significantly reduce memory use of maps.Flatten
    • 7d983f3 Fix Watch() doc
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Jun 16, 2021)

Owner
Kailash Nadh
CTO @zerodhatech | Volunteer @fossunited
Kailash Nadh
🔥🔥 🌈 Golang configuration,use to Viper reading from remote Nacos config systems. Viper remote for Naocs.

Viper remote for Nacos Golang configuration,use to Viper reading from remote Nacos config systems. Viper remote for Naocs. runtime_viper := viper.New(

yoyofxteam 21 Sep 1, 2022
Tmpl - A tool to apply variables from cli, env, JSON/TOML/YAML files to templates

tmpl allows to apply variables from JSON/TOML/YAML files, environment variables or CLI arguments to template files using Golang text/template and functions from the Sprig project.

krako 1 May 30, 2022
Golang Configuration tool that support YAML, JSON, TOML, Shell Environment

Configor Golang Configuration tool that support YAML, JSON, TOML, Shell Environment (Supports Go 1.10+) Usage package main import ( "fmt" "github.c

Jinzhu 1.5k Sep 22, 2022
Generic templating tool with support of JSON, YAML and TOML data

gotempl Small binary used to generate files from Go Templates and data files. The following formats are supported: JSON YAML TOML Usage usage: gotempl

Link Society 7 Jun 15, 2022
A Go (golang) environment loader (which loads env vars from a .env file)

A Go (golang) environment loader (which loads env vars from a .env file)

Bendt Indonesia 0 Feb 8, 2022
Library providing routines to merge and validate JSON, YAML and/or TOML files

CONFLATE Library providing routines to merge and validate JSON, YAML, TOML files and/or structs (godoc) Typical use case: Make your application config

Andy 26 Sep 26, 2022
Jacket of spf13/viper: Simplified go configuration for wire-jacket.

Viper-Jacket: viper config for Wire-Jacket Jacket of spf13/viper: config for wire-jacket. Simplified env-based go configuration package using viper. b

Youngtak Han 1 Nov 18, 2021
Golang library for reading properties from configuration files in JSON and YAML format or from environment variables.

go-config Golang library for reading properties from configuration files in JSON and YAML format or from environment variables. Usage Create config in

null 3 Aug 22, 2022
JSON or YAML configuration wrapper with convenient access methods.

Config Package config provides convenient access methods to configuration stored as JSON or YAML. This is a fork of the original version. This version

Oleg Lebedev 254 Sep 26, 2022
It syncronizes the configuration described in a YAML file against your GitHub Organization

It syncronizes the configuration described in a YAML file against your GitHub Organization. Combined with a CI system, it can be used to implement GitOps for GitHub.

Carlos Tadeu Panato Junior 6 Jul 19, 2021
goconfig uses a struct as input and populates the fields of this struct with parameters from command line, environment variables and configuration file.

goconfig goconfig uses a struct as input and populates the fields of this struct with parameters from command line, environment variables and configur

Go Sidekick 0 May 30, 2022
Genv is a library for Go (golang) that makes it easy to read and use environment variables in your projects. It also allows environment variables to be loaded from the .env file.

genv Genv is a library for Go (golang) that makes it easy to read and use environment variables in your projects. It also allows environment variables

Şakir Şensoy 29 Sep 26, 2022
Go-yaml - Yaml parsing Toolkit For Golang

go-yaml 介绍 gopkg.in/yaml.v3 已经是个非常好用的包,但是在实际开发中总有类型转换带来的麻烦,go-yaml只是在它的基础上,简单的一层

xiaowu 4 Jan 13, 2022
Viper wrapper with config inheritance and key generation

piper - Simple Wrapper For Viper Single Source of Truth Generated Key Structs, No Typo Config Inheritance Multiple Config Strategies Support Cache For

null 7 Sep 26, 2022
formicidate is a small tool for Go application can update the value of environment variables in a .env file with code

formicidae Update .env files in Go with code. What is fomicidae? formicidate is a small tool for Go application. You can update the value of environme

akuma 0 Jan 23, 2022
YML2FSTAB - Convert from yml data to /etc/fstab configuration

YML2FSTAB - Convert from yml data to /etc/fstab configuration

null 0 Nov 1, 2021
Go library for the TOML language

go-toml Go library for the TOML format. This library supports TOML version v1.0.0-rc.3 Features Go-toml provides the following features for using data

Thomas Pelletier 1.3k Sep 23, 2022
TOML parser and encoder library for Golang

TOML parser and encoder library for Golang TOML parser and encoder library for Golang. This library is compatible with TOML version v0.4.0. Installati

Naoya Inada 289 Aug 24, 2022
shops is a simple command-line tool written in Go that helps you simplify the way you manage configuration across a set of machines.

shops is a simple command-line tool written in Go that helps you simplify the way you manage configuration across a set of machines. shops is your configuration management tool of choice when Chef, Puppet, Ansible are all too complicated and all you really want to do is run a bunch of regular shell against a set of hosts.

James Mills 16 Jul 5, 2021