Load configuration in cascade from multiple backends into a struct

Overview

Build Status GoDoc Go Report Card

Confita is a library that loads configuration from multiple backends and stores it in a struct.

Supported backends

Install

go get -u github.com/heetch/confita

Usage

Confita scans a struct for config tags and calls all the backends one after another until the key is found. The value is then converted into the type of the field.

Struct layout

Go primitives are supported:

type Config struct {
  Host        string        `config:"host"`
  Port        uint32        `config:"port"`
  Timeout     time.Duration `config:"timeout"`
}

By default, all fields are optional. With the required option, if a key is not found then Confita will return an error.

type Config struct {
  Addr        string        `config:"addr,required"`
  Timeout     time.Duration `config:"timeout"`
}

Nested structs are supported too:

type Config struct {
  Host        string        `config:"host"`
  Port        uint32        `config:"port"`
  Timeout time.Duration     `config:"timeout"`
  Database struct {
    URI string              `config:"database-uri,required"`
  }
}

If a field is a slice, Confita will automatically split the config value by commas and fill the slice with each sub value.

type Config struct {
  Endpoints []string `config:"endpoints"`
}

As a special case, if the field tag is "-", the field is always omitted. This is useful if you want to populate this field on your own.

type Config struct {
  // Field is ignored by this package.
  Field float64 `config:"-"`

  // Confita scans any structure recursively, the "-" value prevents that.
  Client http.Client `config:"-"`
}

Loading configuration

Creating a loader:

loader := confita.NewLoader()

By default, a Confita loader loads all the keys from the environment. A loader can take other configured backends as parameters.

loader := confita.NewLoader(
  env.NewBackend(),
  file.NewBackend("/path/to/config.json"),
  file.NewBackend("/path/to/config.yaml"),
  flags.NewBackend(),
  etcd.NewBackend(etcdClientv3),
  consul.NewBackend(consulClient),
  vault.NewBackend(vaultClient),
)

Loading configuration:

err := loader.Load(context.Background(), &cfg)

Since loading configuration can take time when used with multiple remote backends, context can be used for timeout and cancelation:

ctx, cancel := context.WithTimeout(context.Background(), 5 * time.Second)
defer cancel()
err := loader.Load(ctx, &cfg)

Default values

If a key is not found, Confita won't change the respective struct field. With that in mind, default values can simply be implemented by filling the structure before passing it to Confita.

type Config struct {
  Host        string        `config:"host"`
  Port        uint32        `config:"port"`
  Timeout     time.Duration `config:"timeout"`
  Password    string        `config:"password,required"`
}

// default values
cfg := Config{
  Host: "127.0.0.1",
  Port: "5656",
  Timeout: 5 * time.Second,
}

err := confita.NewLoader().Load(context.Background(), &cfg)

Backend option

By default, Confita queries each backend one after another until a key is found. However, in order to avoid some useless processing the backend option can be specified to describe in which backend this key is expected to be found. This is especially useful when the location of the key is known beforehand.

type Config struct {
  Host        string        `config:"host,backend=env"`
  Port        uint32        `config:"port,required,backend=etcd"`
  Timeout     time.Duration `config:"timeout"`
}

Command line flags

The flags backend allows to load individual configuration keys from the command line. The default values are extracted from the struct fields values.

A short option is also supported.

To update usage message on the command line, provide a description to the given field.

type Config struct {
  Host        string        `config:"host,short=h"`
  Port        uint32        `config:"port,short=p"`
  Timeout     time.Duration `config:"timeout,description=timeout (in seconds) for failure"`
}
./bin -h

Usage of ./bin:
  -host string
       (default "127.0.0.1")
  -h string
       (default "127.0.0.1")
  -port int
       (default 5656)
  -p int
       (default 5656)
  -timeout duration
       timeout (in seconds) for failure (default 10s)

License

The library is released under the MIT license. See LICENSE file.

Comments
  • struct backend

    struct backend

    this is my attempt at a fix for issue #24

    The work isn't finished yet (except for int, string and bool types) but I wanted to open the PR earlier to get some feedback on whether there is a quicker way of doing this, as I feel like I'm doing a lot of manual work.

    opened by aecepoglu 5
  • Map backend #12 - added a map to store default in backend

    Map backend #12 - added a map to store default in backend

    In this pull request, we are adding a map of defaults for usage by the different backends. Based on the ticket, I believe this is what is required, however I do apologize if I misunderstood it

    I also wasn't sure if a the type map[string][string] is the best solution. I was thinking of using a map[string]interface{}, but wasn't sure if the user would want to define an object as a value

    Issue # 12

    opened by zelahi 3
  • adds ssm backend

    adds ssm backend

    Adds Amazon SSM's Parameters as a backend.

    There are many ways the calls could be made to Amazon, but, just starting with the simplest now. User just gives the path for their parameter store. Can be easily changed in the future if people want to use it in a different way. Very open to ideas here.

    opened by steamrolla 2
  • When value defined in multiple backend, last one wins

    When value defined in multiple backend, last one wins

    This was previously discussed in https://github.com/heetch/confita/issues/7, but my tests are giving different results, If I define a variable in a file as an env var, the last backend on the list wins

    Examples: l := confita.NewLoader(file.NewBackend(path),env.NewBackend()) in this case the env backend overwrites the values in the file files

    n := confita.NewLoader(env.NewBackend(),file.NewBackend(path)) same issue here, the file will overwrite any value previously set as env var

    The docs say: "Confita scans a struct for config tags and calls all the backends one after another until the key is found" from my understanding once found it stops and continue looking for the next config key

    There is a working example of the issue:

    https://play.golang.org/p/c8l9NbRnJOT

    Thanks to @rogpeppe for your go playground example on issue 61

    opened by dmaciel 2
  • Support short/long flags

    Support short/long flags

    Most command-line tools support a short and a long form for flags, e.g. "-p" or "--port". Currently confita only supports a single flag per member.

    Suggestion: add a "short" sub-tag and handle it as a synonym of the default name. Also update help usage generation to reflect this.

    type Configuration struct {
    	Port          int    `config:"port,short=p"`
    }
    
    
    enhancement 
    opened by ogerardin 2
  • parse yaml file failled, I have that key but confita said not found

    parse yaml file failled, I have that key but confita said not found

    hi, guys. I met some trouble with use confita. Anybody can help me?Please. I have a nest struct:

    type Database struct {
    	Which 	string  `config:"which"`
    	DbDir 	string	`config:"db_dir"`
    }
    
    type Config struct {
    	Database Database 	`config:"database,required"`
    	AppName string 		`config:"app_name,required"`
    }
    

    and then,when i use goconvey to test, it failed. here is my code:

    func createTempFile(name, content string) (string, func()) {
    	dir, err := ioutil.TempDir("", "testDir")
    	So(err, ShouldBeNil)
    
    	path := filepath.Join(dir, name)
    	f, err := os.Create(path)
    	So(err, ShouldBeNil)
    
    	_, err = fmt.Fprint(f, content)
    	So(err, ShouldBeNil)
    	So(f.Close(), ShouldBeNil)
    
    	return path, func() {
    		So(os.RemoveAll(dir), ShouldBeNil)
    	}
    }
    
    func TestLoadWithTempFile(t *testing.T) {
    	Convey("test Load with temp file", t, func() {
    		var cfg Config
    		path, cleanUp := createTempFile("app.yaml", `
    app_name: blog
    database:
      which: sqlite3
      db_dir: /tmp/test.db
    `)
    		defer cleanUp()
    
    		loader := confita.NewLoader(file.NewBackend(path))
    		err := loader.Load(context.Background(), &cfg)
    
    		So(err, ShouldBeNil)
    		So(cfg.Database, ShouldNotBeNil)
    		So(cfg.AppName, ShouldEqual, "blog")
    		So(cfg.Database.Which, ShouldEqual, "sqlite3")
    		So(cfg.Database.DbDir, ShouldEqual, "/tmp/test.db")
    	})
    }
    

    and the test result:

    ....x.
    Failures:
    
      * /home/lks/go/src/github.com/crazypandas/goBlog/config/config_test.go 
      Line 46:
      Expected: nil
      Actual:   'required key 'app_name' for field 'AppName' not found'
    
    
    6 total assertions
    

    Is there anything wrong?

    opened by CrazyPandas 2
  • Issue when trying to load a list of struct from a yaml file.

    Issue when trying to load a list of struct from a yaml file.

    Given this type:

    type migrationSettings struct {
    	Source      string `config:"source,required"`
    	Destination string `config:"destination,required"`
    	Partitions  int    `config:"partitions,required"`
    }
    

    The following yaml:

    - source: topic1
      destination: topic2
      partitions: 3
    
    - source: topic1
      destination: topic2
      partitions: 3
    
    - source: topic1
      destination: topic2
      partitions: 3
    

    The snippet:

    var s []migrationSettings
    
    l := confita.NewLoader(file.NewBackend("config.yaml"))
    l.Load(context.Background(), &s)
    
    fmt.Println(s)
    

    The output:

    []
    
    bug 
    opened by yaziine 2
  • Go modules

    Go modules

    Fix #40

    Hi @YassineE :-) Hope your are doing well. Recently I am trying to get more and more familiar with the modules in the Go 1.11. Hence this PR is a great opportunity to apply this approach to the real project.

    Few notes about the PR:

    1. My source of knowledge how to introduce go modules and configure travis build are following articles:
    1. Correct me if I am wrong, but since repository is an open source library you want to still have support for different go versions. Therefore we still have to keep go dep for go 1.9.x & 1.10.x + I've introduced go mod for go 1.11.x & tip. I've managed to configure a build matrix to satisfy these requirements.
    2. I've marked tip go version in the matrix as the one which can fail. It's ok if the code fails on unstable development versions of Go, however it should not break the whole build if other version are green.
    3. I've tried my best to avoid duplication in .travis.yml, unfortunately not everywhere it was possible. It is because we have 4 go versions and 2 different dependencies managment strategies. Let me know if you have any idea how to improve it.

    Waiting for the feedback!

    opened by wojteninho 2
  • Specify the backend

    Specify the backend

    In order to avoid some useless processing we can specify for a field in which backend we can find its value.

    e.g.

    type Config struct {
      Foo        string        `config:"foo",bcknd:"etcd"`
      Bar        string        `config:"bar",bcknd:"consul"`
    }
    
    enhancement 
    opened by yaziine 2
  • Environment variable specification

    Environment variable specification

    The XCU specification says that the environment variable names consist solely of upper-case letters, digits and the "_" (underscore).

    Maybe, we should respect this spec and not handle the case if names contains "-" (dash).

    enhancement 
    opened by yaziine 2
  • Improve documentation

    Improve documentation

    IMO we should specify in the doc that when a key is found in a backend, we won't continue to look for it in the remaining backends.

    So there is an importance on how the backends are sorted inside the slice.

    What do you think ?

    enhancement 
    opened by yaziine 2
  • Transition from dgrijalva/jwt-go pkg to golang-jwt/jwt

    Transition from dgrijalva/jwt-go pkg to golang-jwt/jwt

    jwt-go has a security vulnerability (CVE-2020-26160). There is no patch available and users of jwt-go are advised to migrate to golang-jwt at version 3.2.1. 👀

    opened by axiomista 1
  • Reusing nested config structs

    Reusing nested config structs

    Hi! I am using confita configs as API for internal libraries in my projects. I've tried to use config structs multiple times in single project setup. And I found a problem for my usage. Field tags can't repeat in one config:

    type HTTPConfig struct {
    	Addr string `config:"addr"`
    	Port int `config:"port"`
    }
    
    type AppConfig struct {
    	MetricsHTTPServer HTTPConfig
    	LoggingHTTPServer HTTPConfig
    }
    

    There are not possibilities for defining different options for different servers. Also if we'll try to use flags as backend, we'll get panic (below):

    panic: ./cmd/example/example flag redefined: addr
    
    goroutine 1 [running]:
    flag.(*FlagSet).Var(0xc0001371a0, 0x551e910, 0xc000614440, 0x4f288db, 0xc, 0x0, 0x0)
    	/-S/contrib/go/_std/src/flag/flag.go:871 +0x485
    flag.(*FlagSet).StringVar(...)
    	/-S/contrib/go/_std/src/flag/flag.go:760
    flag.(*FlagSet).String(0xc0001371a0, 0x4f288db, 0xc, 0x0, 0x0, 0x0, 0x0, 0xc000614430)
    	/-S/contrib/go/_std/src/flag/flag.go:773 +0xa5
    flag.String(...)
    	/-S/contrib/go/_std/src/flag/flag.go:780
    github.com/heetch/confita/backend/flags.(*Backend).LoadStruct(0x5d76df8, 0x552dea8, 0xc0005d2340, 0xc00025f080, 0x0, 0x0)
    	/-S/vendor/github.com/heetch/confita/backend/flags/flags.go:73 +0x548
    github.com/heetch/confita.(*Loader).resolve(0xc00025f050, 0x552dea8, 0xc0005d2340, 0xc00025f080, 0x5d30240, 0x199)
    	/-S/vendor/github.com/heetch/confita/config.go:171 +0x788
    github.com/heetch/confita.(*Loader).Load(0xc00025f050, 0x552dea8, 0xc0005d2340, 0x4f63ac0, 0x5d30240, 0x49, 0xd)
    	/-S/vendor/github.com/heetch/confita/config.go:58 +0x20a
    

    It may be solved using something like prefixes. Example usage is following:

    type HTTPConfig struct {
    	Addr string `config:"addr"`
    	Port int `config:"port"`
    }
    
    type AppConfig struct {
    	MetricsHTTPServer HTTPConfig `config:"prefix=metrics"`
    	LoggingHTTPServer HTTPConfig `config:"prefix=logging"`
    }
    

    Or more idiomatic if it needs. I think this feature will be useful not only me))

    opened by zefirior 0
  • Can't load config into struct with json tags

    Can't load config into struct with json tags

    I noticed a regression in behaviour. Here is code for reproducing the issue.

    package confitajson
    
    import (
    	"context"
    	"log"
    	"testing"
    
    	"github.com/heetch/confita"
    	"github.com/heetch/confita/backend/file"
    )
    
    func TestLoad(t *testing.T) {
    	b := file.NewBackend("config.json")
    
    	var c struct {
    		Field string `json:"field"`
    	}
    
    	if err := confita.NewLoader(b).Load(context.Background(), &c); err != nil {
    		log.Fatalf("failed to load config: %v", err)
    	}
    
    	if c.Field != "value" {
    		log.Fatal("field value is not updated")
    	}
    }
    

    Content of config.json file is following:

    {
        "field": "value"
    }
    

    When running this test on the latest confita release, struct field is not updated. But on version v0.7.0 code works fine.

    [email protected] ~/C/confitajson> go get github.com/heetch/[email protected]
    go: downloading github.com/heetch/confita v0.7.0
    [email protected] ~/C/confitajson> go test .
    ok  	github.com/slon/confitajson	0.002s
    [email protected] ~/C/confitajson> go get github.com/heetch/confita
    go: github.com/heetch/confita upgrade => v0.9.1
    [email protected] ~/C/confitajson> go test .
    2020/05/27 20:02:34 field value is not updated
    FAIL	github.com/slon/confitajson	0.002s
    FAIL
    

    I also noticed, that removing these 3 lines, seems to fix this issue for me. https://github.com/heetch/confita/blob/master/config.go#L183-L185

    Is that change in behaviour intentional, or is it indeed a regression?

    opened by slon 1
  • Structured configuration support (path/dots)

    Structured configuration support (path/dots)

    Does confita support structured configuration? It would be useful to make configurations more clean/organized/structure, and for configurations of growing/large projects.

    I have read the README.md, but there's no example for structure config,... And, I didn't find any TOML/YAML/JSON examples of structured configurations.

    I would like to populate the following structure:

    
    type Config struct {
    	// notifications configuration
    	Notifications struct {
    		// turns on user notifications for changes/updates/creations
    		Enabled bool
    		// from email address of emails
    		From string
    		// BCC addresses of every notification sent
    		BCC []string
    	}
    
    	// SMTP server/mail-sending configuration
    	SMTP struct {
    		// format server:port
    		ServerAddr string
    
    		// plain auth credentials
    		Username, Password string
    
    		// default email address to sent from
    		DefaultFrom string
    	}
    }
    

    For example, with TOML:

    notifications.enabled = true
    notifications.from = "[email protected]"
    notifications.bcc = "[email protected]"
    
    smtp.serveraddr = "smtp.gmail.com:587"
    smtp.username = "[email protected]"
    smtp.password = "secret"
    smtp.defaultfrom = "[email protected]"
    

    Or, with YAML:

    notifications:
      - enabled: true
      - from: "[email protected]"
      - bcc:  "[email protected]"
    
    smtp:
      - serveraddr: "smtp.gmail.com:587"
      - username: "[email protected]"
      - password: "secret"
      - defaultfrom: "[email protected]"
    
    opened by kravemir 0
Releases(v0.10.0)
  • v0.10.0(Mar 24, 2021)

    New features

    The Vault backend now supports the Vault KV secrets engine v2. Use the new vault.NewBackendV2 to create a backend that supports it.

    For some background info on Vault see https://www.vaultproject.io/docs/secrets/kv and https://www.vaultproject.io/api-docs/secret/kv.

    Thanks @mopemope for the initial contribution!

    Source code(tar.gz)
    Source code(zip)
  • v0.9.2(Jun 23, 2020)

    Bug fixes

    • #80 Avoiding setting values from flags backend when they aren't specified allowing confita mix environment and flags configuration.
    Source code(tar.gz)
    Source code(zip)
  • v0.9.1(May 6, 2020)

  • v0.9.0(Nov 20, 2019)

    New feature

    • Add Amazon SSM as a backend #71

    Improvements

    • Treat empty environment variables as not found (backend/env) #78
    • Use consul v1.1.0 #72
    • Add description json tags (backend/flags) #56
    • Add support of short and long flags (backend/flags) #57
    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Sep 18, 2019)

    This release adds one new feature (the ability to specify alternative flag names, thanks to @ogerardin), and updates the consul dependency.

    Source code(tar.gz)
    Source code(zip)
  • v0.7.0(Aug 27, 2019)

    New feature

    • Add file.NewOptionalBackend for optional file configuration #55

    Bug fix

    • Now Confita stops looking for a key when it is found #64
    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(Jun 14, 2019)

  • v0.5.1(Jul 16, 2018)

  • v0.5.0(Jun 28, 2018)

  • v0.4.0(Apr 19, 2018)

  • v0.3.0(Mar 29, 2018)

  • v0.2.0(Jan 17, 2018)

  • v0.1.0(Jan 11, 2018)

Owner
Heetch
Enjoy your night out
Heetch
Load configuration from multiple sources in Go

Multiconfig Load configuration from multiple sources. Multiconfig makes loading/parsing from different configuration sources an easy task. The problem

Koding, Inc. 438 Nov 3, 2022
Noel 2 Jul 4, 2022
12 factor configuration as a typesafe struct in as little as two function calls

Config Manage your application config as a typesafe struct in as little as two function calls. type MyConfig struct { DatabaseUrl string `config:"DAT

Jeremy Loy 319 Nov 27, 2022
Tis module used as base fo configuration apps.By default, it expands into the inside of the application.

Tis module used as base fo configuration apps.By default, it expands into the inside of the application. Also, module c reads a dictionary of secrets from the application directory by its AppName and extension json.

LordTor 0 Dec 7, 2021
⚙️ Dead Simple Config Management, load and persist config without having to think about where and how.

Configo Dead Simple Config Management, load and persist config without having to think about where and how. Install go get github.com/UltiRequiem/conf

Eliaz Bobadilla 8 Apr 6, 2022
Quick and easy way to load config files based on a simple set of rules.

config Quick and easy way to load config files based on a simple set of rules. Project inspired by https://github.com/lorenwest/node-config Important

Tarcisio Gruppi 1 Apr 9, 2021
A Go library for parsing struct tags from environment variables.

Envconfig Envconfig populates struct field values based on environment variables or arbitrary lookup functions. It supports pre-setting mutations, whi

Seth Vargo 697 Nov 16, 2022
✨Clean and minimalistic environment configuration reader for Golang

Clean Env Minimalistic configuration reader Overview This is a simple configuration reading tool. It just does the following: reads and parses configu

Ilya Kaznacheev 791 Nov 25, 2022
JSON or YAML configuration wrapper with convenient access methods.

Config Package config provides convenient access methods to configuration stored as JSON or YAML. This is a fork of the original version. This version

Oleg Lebedev 259 Nov 17, 2022
Configure is a Go package that gives you easy configuration of your project through redundancy

Configure Configure is a Go package that gives you easy configuration of your project through redundancy. It has an API inspired by negroni and the fl

Harrison Shoebridge 57 Sep 26, 2022
An opinionated configuration loading framework for Containerized and Cloud-Native applications.

Opinionated configuration loading framework for Containerized and 12-Factor compliant applications. Read configurations from Environment Variables, an

Sherif Abdel-Naby 82 Oct 30, 2022
Small library to read your configuration from environment variables

envconfig envconfig is a library which allows you to parse your configuration from environment variables and fill an arbitrary struct. See the example

Vincent Rischmann 229 Nov 3, 2022
A minimalist Go configuration library

fig fig is a tiny library for loading an application's config file and its environment into a Go struct. Individual fields can have default values def

null 243 Nov 11, 2022
go-up! A simple configuration library with recursive placeholders resolution and no magic.

go-up! A simple configuration library with placeholders resolution and no magic. go-up provides a simple way to configure an application from multiple

Francesco 38 Sep 26, 2022
Go configuration made easy!

gofigure Go configuration made easy! Just define a struct and call Gofigure Supports strings, ints/uints/floats, slices and nested structs Supports en

Ian Kent 65 Sep 26, 2022
Harvest configuration, watch and notify subscriber

Harvester Harvester is a configuration library which helps setting up and monitoring configuration values in order to dynamically reconfigure your app

Beat Labs 122 Nov 16, 2022
go implementation of lightbend's HOCON configuration library https://github.com/lightbend/config

HOCON (Human-Optimized Config Object Notation) Configuration library for working with the Lightbend's HOCON format. HOCON is a human-friendly JSON sup

Gürkan Kaymak 53 Nov 17, 2022
🛠 A configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP

config A small configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP. Example func main() {

Josh Betz 214 Oct 15, 2022