A Go library for parsing struct tags from environment variables.

Overview

Envconfig

GoDoc GitHub Actions

Envconfig populates struct field values based on environment variables or arbitrary lookup functions. It supports pre-setting mutations, which is useful for things like converting values to uppercase, trimming whitespace, or looking up secrets.

Note: Versions prior to v0.2 used a different import path. This README and examples are for v0.2+.

Usage

Define a struct with fields using the env tag:

type MyConfig struct {
  Port     int    `env:"PORT"`
  Username string `env:"USERNAME"`
}

Set some environment variables:

export PORT=5555
export USERNAME=yoyo

Process it using envconfig:

package main

import (
  "context"
  "log"

  "github.com/sethvargo/go-envconfig"
)

func main() {
  ctx := context.Background()

  var c MyConfig
  if err := envconfig.Process(ctx, &c); err != nil {
    log.Fatal(err)
  }

  // c.Port = 5555
  // c.Username = "yoyo"
}

You can also use nested structs, just remember that any fields you want to process must be public:

type MyConfig struct {
  Database *DatabaseConfig
}

type DatabaseConfig struct {
  Port     int    `env:"PORT"`
  Username string `env:"USERNAME"`
}

Configuration

Use the env struct tag to define configuration.

Required

If a field is required, processing will error if the environment variable is unset.

type MyStruct struct {
  Port int `env:"PORT,required"`
}

It is invalid to have a field as both required and default.

Default

If an environment variable is not set, the field will be set to the default value. Note that the environment variable must not be set (e.g. unset PORT). If the environment variable is the empty string, that counts as a "value" and the default will not be used.

type MyStruct struct {
  Port int `env:"PORT,default=5555"`
}

You can also set the default value to another field or value from the environment, for example:

type MyStruct struct {
  DefaultPort int `env:"DEFAULT_PORT,default=5555"`
  Port        int `env:"OVERRIDE_PORT,default=$DEFAULT_PORT"`
}

The value for Port will default to the value of DEFAULT_PORT.

It is invalid to have a field as both required and default.

Prefix

For shared, embedded structs, you can define a prefix to use when processing struct values for that embed.

type SharedConfig struct {
  Port int `env:"PORT,default=5555"`
}

type Server1 struct {
  // This processes Port from $FOO_PORT.
  *SharedConfig `env:",prefix=FOO_"`
}

type Server2 struct {
  // This processes Port from $BAR_PORT.
  *SharedConfig `env:",prefix=BAR_"`
}

It is invalid to specify a prefix on non-struct fields.

Complex Types

Durations

In the environment, time.Duration values are specified as a parsable Go duration:

type MyStruct struct {
  MyVar time.Duration `env:"MYVAR"`
}
export MYVAR="10m" # 10 * time.Minute

TextUnmarshaler / BinaryUnmarshaler

Types that implement TextUnmarshaler or BinaryUnmarshaler are processed as such.

json.Unmarshaler

Types that implement json.Unmarshaler are processed as such.

gob.Decoder

Types that implement gob.Decoder are processed as such.

Slices

Slices are specified as comma-separated values:

type MyStruct struct {
  MyVar []string `env:"MYVAR"`
}
export MYVAR="a,b,c,d" # []string{"a", "b", "c", "d"}

Note that byte slices are special cased and interpreted as strings from the environment.

Maps

Maps are specified as comma-separated key:value pairs:

type MyStruct struct {
  MyVar map[string]string `env:"MYVAR"`
}
export MYVAR="a:b,c:d" # map[string]string{"a":"b", "c":"d"}

Structs

Envconfig walks the entire struct, so deeply-nested fields are also supported. You can also define your own decoder (see below).

Prefixing

You can define a custom prefix using the PrefixLookuper. This will lookup values in the environment by prefixing the keys with the provided value:

type MyStruct struct {
  MyVar string `env:"MYVAR"`
}
// Process variables, but look for the "APP_" prefix.
l := envconfig.PrefixLookuper("APP_", envconfig.OsLookuper())
if err := envconfig.ProcessWith(ctx, &c, l); err != nil {
  panic(err)
}
export APP_MYVAR="foo"

Extension

All built-in types are supported except Func and Chan. If you need to define a custom decoder, implement Decoder:

type MyStruct struct {
  field string
}

func (v *MyStruct) EnvDecode(val string) error {
  v.field = fmt.Sprintf("PREFIX-%s", val)
  return nil
}

If you need to modify environment variable values before processing, you can specify a custom Mutator:

type Config struct {
  Password `env:"PASSWORD"`
}

func resolveSecretFunc(ctx context.Context, key, value string) (string, error) {
  if strings.HasPrefix(key, "secret://") {
    return secretmanager.Resolve(ctx, value) // example
  }
  return value, nil
}

var config Config
envconfig.ProcessWith(ctx, &config, envconfig.OsLookuper(), resolveSecretFunc)

Testing

Relying on the environment in tests can be troublesome because environment variables are global, which makes it difficult to parallelize the tests. Envconfig supports extracting data from anything that returns a value:

lookuper := envconfig.MapLookuper(map[string]string{
  "FOO": "bar",
  "ZIP": "zap",
})

var config Config
envconfig.ProcessWith(ctx, &config, lookuper)

Now you can parallelize all your tests by providing a map for the lookup function. In fact, that's how the tests in this repo work, so check there for an example.

You can also combine multiple lookupers with MultiLookuper. See the GoDoc for more information and examples.

Inspiration

This library is conceptually similar to kelseyhightower/envconfig, with the following major behavioral differences:

  • Adds support for specifying a custom lookup function (such as a map), which is useful for testing.

  • Only populates fields if they contain zero or nil values. This means you can pre-initialize a struct and any pre-populated fields will not be overwritten during processing.

  • Support for interpolation. The default value for a field can be the value of another field.

  • Support for arbitrary mutators that change/resolve data before type conversion.

Comments
  • Add notempty struct tag

    Add notempty struct tag

    Currently there is no easy way of knowing if a environment variable is set with an empty value except for checking the value yourself.

    In certain workflows like for example when using docker-compose with variable substituion this can be a problem. In this case a missing system variable is replaced by an empty string. The resulting container sees a set but empty variable. Therefore the required keyword does not result in an error and is useless.

    notempty uses required and errors on a empty variable with message "empty value"

    opened by HalloTschuess 10
  • Function to determine if a set is performed or not

    Function to determine if a set is performed or not

    Hi, and thanks for a great library!

    I've got a use-case where I load stuff from a JSON file, AWS parameter store & AWS secrets manager and then applies environment variables.

    This is a override chain where the JSON file is static default configuration and secrets and other IAM handled parameters are from AWS services. Lastly sometimes the DevOps need to do a parameter override directly at the lambda level on a environment variable.

    Since current implementation only overrides when ef.IsZero() I needed to have the ability to choose if this is the case or always try to override.

    Therefore I did this PR that allows a custom Decider function to be set.

    	err := ProcessWith(
    		context.Background(),
    		&q,
    		OsLookuper(),
    		func(ctx context.Context, value reflect.Value) bool {
    			return true // always set
    		},
    	)
    

    I've added a DefaultDeciderFunction to be used internally on Process and if just the standard behaviour in ProcessWith.

    // This is the default behaviour
    var DefaultDeciderFunc = func(ctx context.Context, value reflect.Value) bool {
    	return value.IsZero()
    }
    
    // Process uses the DefaultDeciderFunc
    func Process(ctx context.Context, i interface{}) error {
    	return ProcessWith(ctx, i, OsLookuper(), DefaultDeciderFunc)
    }
    
    // In unit test I've replaced with DefaultDeciderFunc
    if err := ProcessWith(ctx, tc.input, tc.lookuper, DefaultDeciderFunc, tc.mutators...); err != nil {
      // ...
    }
    

    Cheers, Mario

    opened by mariotoffia 9
  • Why do you use regex to validate env var name?

    Why do you use regex to validate env var name?

    I saw this line 98 but I believe that be unnecessary to use regex it is 3 times slower compared to \w+ is the same result. Case I'm wrong, please fix me. Maybe you can use Unicode to validate strings and gain performance to reduce processing.

    https://github.com/sethvargo/go-envconfig/blob/63e4089fe1a599af605c681f25b5c34b3252514c/envconfig.go#L98

    opened by renanbastos93 5
  • Support

    Support "-" to skip some exported fields.

    HI there 👋🏼

    It would be great if we can have exported fields that are "skipped". This is because this library seems to instantiate deeply nested structs which may not be desirable.

    Something like

    type Config struct {
      Val string `env:"-"`
    }
    

    is common practice to skip a field.

    Happy to submit a PR if that's something that is worth doing. Thanks!

    opened by marwan-at-work 5
  • Import path seems to be ambiguous

    Import path seems to be ambiguous

    Hi!

    import "github.com/sethvargo/go-envconfig/pkg/envconfig"

    Such import contains almost duplicated parts - go-envconfig and envconfig at the end. Have you thought about moving go code to the project root?

    We will have more elegant and meaningful import path then.

    opened by mwf 5
  • Always call decoders

    Always call decoders

    This reverts part of #62 to always process decoder interfaces for struct fields. This removes a breaking change where certain built-in types like net/url.URL have a custom unmarshaling function for the empty string.

    The other rules for "overwrite" still apply.

    Part of https://github.com/sethvargo/go-envconfig/issues/64

    opened by sethvargo 4
  • Initialization of marshaled types

    Initialization of marshaled types

    When processing structs that implement a marshaller, this library has typically left it up to the unmarshaling function to populate the object. For example, given this config.

    type Config struct {
    	URL *url.URL `env:"URL"`
    }
    

    The result of processing this configuration through env-config would be equivalent to marshaling and unmarshaling a url.URL struct. For example:

    func MarshalTest(t *testing.T) {
    	var cfg Config
    	if err := envconfig.Process(context.Background(), &cfg); err != nil {
    		t.Fatal(err)
    	}
    
    	u := &url.URL{}
    	b, _ := u.MarshalBinary()
    	_ = u.UnmarshalBinary(b)
    	want := Config{
    		URL: u,
    	}
    
    	if diff := cmp.Diff(want, cfg, cmp.AllowUnexported(Config{})); diff != "" {
    		t.Errorf("Configuration() mismatch (-want +got):\n%s", diff)
    	}
    }
    

    Since v0.8.0 this test would fail. It seems that the library is skipping the decoder of a struct field if there are no values set. This generally seems ok for fields that can be controlled with noinit, default tags but in this example, the initialization of fields internal to url.URL cannot be controlled with env tags.

    I think introducing the concept of managed vs unmanaged fields in the configuration will help with this. If a configuration field has an env tag it should follow the rules laid out by the env-config library, which will only use the decoder if a conf value is set (default or otherwise). Any struct fields without the env tag should default to using their underlying decoder no matter what. This will prevent potentially initializing fields that were not intended to be initialized.

    I've included a draft PR to demonstrate this.

    opened by williamgcampbell 4
  • Is there a possibility to use it with map[string][]string

    Is there a possibility to use it with map[string][]string

    I have a quick question. Is there a possibility to use it with map[string][]string type? I didn't find it in the description or the tests but maybe there is possible

    opened by franpog859 4
  • Swap order of encoding.TextUnmarshaler and encoding.BinaryUnmarshaler

    Swap order of encoding.TextUnmarshaler and encoding.BinaryUnmarshaler

    We're transitioning to using the Go 1.18 netip.AddrPort with go-envconfig parsing it from a string. We noticed a small corner case though, that some addresses don't fail on the binary unmarshaling https://github.com/sethvargo/go-envconfig/blob/befaf9a086c5e7db885e1ca3d36aa4e8b9d3a57d/envconfig.go#L521 and therefore we end up with a really weird IPv6 address.

    So e.g. "239.255.76.67:7667" would succeed (not error) when unmarshaled as binary (resulting in [3233:392e:3235:352e:3736:2e36:373a:3736%6]:13623 but e.g. 127.0.0.1:11311 would fail and fall back to using the TextUnmarshaler, resulting in a properly decoded IPv4 address.

    Could we swap the order so the unmarshaling as text happens before the binary one?

    opened by gust1n 4
  • Support for default array values

    Support for default array values

    Love the library, unfortunately there's one important missed edge case, namely passing default values with commas in them. This PR fixes it using the simplest possible approach. We're using it in production without any issues.

    I won't bother with adding tests before getting an OK on the approach from you.

    kind/stale 
    opened by pzduniak 4
  • Magic characters when decoding URLs

    Magic characters when decoding URLs

    Not sure why, but URLs are not being decoded properly.

    The url.URL type implements the BinaryUnmarshaler interface (https://golang.org/pkg/net/url/#URL.UnmarshalBinary)

    However, when used in a config struct, the unmarshaling adds an @ see this (https://play.golang.org/p/VrZDD85bR6j)

    opened by stephenafamo 4
  • Do not call decoders on unset envvars

    Do not call decoders on unset envvars

    Originally part of https://github.com/sethvargo/go-envconfig/pull/64, but then reverted in https://github.com/sethvargo/go-envconfig/pull/68 because some users were depending on the behavior. In 1.0, we should change the API contract so that decoders are never called on unset values.

    opened by sethvargo 0
Releases(v0.8.2)
  • v0.8.2(Aug 8, 2022)

    What's Changed

    • Track pointer initialization for zero values by @sethvargo in https://github.com/sethvargo/go-envconfig/pull/73
    • Drop regexp for performance by @sethvargo in https://github.com/sethvargo/go-envconfig/pull/74

    Full Changelog: https://github.com/sethvargo/go-envconfig/compare/v0.8.1...v0.8.2

    Source code(tar.gz)
    Source code(zip)
  • v0.8.1(Jul 27, 2022)

    What's Changed

    • Propagate noinit from parent fields by @sethvargo in https://github.com/sethvargo/go-envconfig/pull/67
    • Always call decoders by @sethvargo in https://github.com/sethvargo/go-envconfig/pull/68

    Full Changelog: https://github.com/sethvargo/go-envconfig/compare/v0.8.0...v0.8.1

    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Jul 14, 2022)

    What's Changed

    The previous implementation of overwrite would always overwrite values in the given struct, even if values existed. While this is the definition of overwrite, it unintentionally extended to default values as well. So even if a value was explicitly set on a struct, it would be overwritten with the "default" value set in envconfig. This was an unexpected behavior, since defaults should take the lowest precedence.

    The new implementation has the following behavior with overwrite:

    • If the struct field has the zero value and a default is set:

      • If no environment variable is specified, the struct field will be populated with the default value.

      • If an environment variable is specified, the struct field will be populate with the environment variable value.

    • If the struct field has a non-zero value and a default is set:

      • If no environment variable is specified, the struct field's existing value will be used (the default is ignored).

      • If an environment variable is specified, the struct field's existing value will be overwritten with the environment variable value.

    As part of this change, decoder interfaces are only processed when an environment (or a default) is present.

    Full Changelog: https://github.com/sethvargo/go-envconfig/compare/v0.7.0...v0.8.0

    Source code(tar.gz)
    Source code(zip)
  • v0.7.0(Jun 9, 2022)

    What's Changed

    • Change custom unmarshaling order by @gust1n in https://github.com/sethvargo/go-envconfig/pull/59. This changes the order of resolution for unmarshalling to:

      1. envconfig.Decoder
      2. encoding.TextUnmarshaler
      3. json.Unmarshaler
      4. encoding.BinaryUnmarshaler
      5. gob.GobDecoder

    New Contributors

    • @gust1n made their first contribution in https://github.com/sethvargo/go-envconfig/pull/59

    Full Changelog: https://github.com/sethvargo/go-envconfig/compare/v0.6.2...v0.7.0

    Source code(tar.gz)
    Source code(zip)
  • v0.6.2(May 31, 2022)

    What's Changed

    • Fixing struct pointer initialization with no values set by @williamgcampbell in https://github.com/sethvargo/go-envconfig/pull/57

    New Contributors

    • @williamgcampbell made their first contribution in https://github.com/sethvargo/go-envconfig/pull/57

    Full Changelog: https://github.com/sethvargo/go-envconfig/compare/v0.6.1...v0.6.2

    Source code(tar.gz)
    Source code(zip)
  • v0.6.1(May 31, 2022)

    What's Changed

    • Sort struct fields for reduce memory by @zchee in https://github.com/sethvargo/go-envconfig/pull/51
    • Fix GitHub Actions badge url and query by @zchee in https://github.com/sethvargo/go-envconfig/pull/52
    • Fix typo in README.md by @ucpr in https://github.com/sethvargo/go-envconfig/pull/53
    • Extend noinit to all pointer fields by @sethvargo in https://github.com/sethvargo/go-envconfig/pull/55

    New Contributors

    • @zchee made their first contribution in https://github.com/sethvargo/go-envconfig/pull/51
    • @ucpr made their first contribution in https://github.com/sethvargo/go-envconfig/pull/53

    Full Changelog: https://github.com/sethvargo/go-envconfig/compare/v0.6.0...v0.6.1

    Source code(tar.gz)
    Source code(zip)
  • v0.6.0(Apr 6, 2022)

Owner
Seth Vargo
Engineer @google
Seth Vargo
goconfig uses a struct as input and populates the fields of this struct with parameters from command line, environment variables and configuration file.

goconfig goconfig uses a struct as input and populates the fields of this struct with parameters from command line, environment variables and configur

Go Sidekick 1 Sep 26, 2022
Small library to read your configuration from environment variables

envconfig envconfig is a library which allows you to parse your configuration from environment variables and fill an arbitrary struct. See the example

Vincent Rischmann 229 Nov 3, 2022
A Go port of Ruby's dotenv library (Loads environment variables from `.env`.)

GoDotEnv A Go (golang) port of the Ruby dotenv project (which loads env vars from a .env file) From the original Library: Storing configuration in the

John Barton 5.6k Dec 5, 2022
🛠 A configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP

config A small configuration library for Go that parses environment variables, JSON files, and reloads automatically on SIGHUP. Example func main() {

Josh Betz 214 Oct 15, 2022
Golang library for managing configuration data from environment variables

envconfig import "github.com/kelseyhightower/envconfig" Documentation See godoc Usage Set some environment variables: export MYAPP_DEBUG=false export

Kelsey Hightower 4.3k Dec 3, 2022
Golang library for reading properties from configuration files in JSON and YAML format or from environment variables.

go-config Golang library for reading properties from configuration files in JSON and YAML format or from environment variables. Usage Create config in

null 3 Aug 22, 2022
Simple lib to parse environment variables to structs

env Simple lib to parse envs to structs in Go. Example A very basic example: package main import ( "fmt" "time" // if using go modules "github.c

Carlos Alexandro Becker 2.9k Dec 4, 2022
Un-marshaling environment variables to Go structs

envcfg Un-marshaling environment variables to Go structs Getting Started Let's set a bunch of environment variables and then run your go app #!/usr/bi

Tomaž Kovačič 99 Sep 26, 2022
Go helpers to manage environment variables

Envh This library is made up of two parts : Env object : it wraps your environments variables in an object and provides convenient helpers. Env tree o

Anthony HAMON 96 Sep 26, 2022
Environment variables substitution for Go

envsubst Environment variables substitution for Go. see docs below Installation: From binaries Latest stable envsubst prebuilt binaries for 64-bit Lin

Ariel Mashraki 555 Nov 23, 2022
Quickly read variables from environment files

go-quick-env Quickly read variables from environment files The best way to import environment variables to your code, is by using .env files. This lib

Panos Petropoulos 3 May 11, 2021
Read files into environment variables and execute command

read-file-to-env -- Read files into environment variables and execute command Example use: read-file-to-env -one-line=HOST=/etc/hostname sh -c 'echo h

Tv 2 Nov 12, 2021
A mapper of ENVironment variables to Structure for Go

envs a mapper of ENVironment variables to a Structure for Go. This library maps the environment variables to the struct according to the fields' types

moznion 3 Dec 3, 2021
Environment variables configuration package for Go microservices.

gocfg Environment variables configuration package for Go microservices. It helps validate environment variable values and set default values if needed

Sergey Prokhorov 0 Dec 30, 2021
formicidate is a small tool for Go application can update the value of environment variables in a .env file with code

formicidae Update .env files in Go with code. What is fomicidae? formicidate is a small tool for Go application. You can update the value of environme

akuma 0 Jan 23, 2022
Lightweight package that makes easier and safer to deal with environment variables.

Envisage A lightweight package that makes easier and safer to deal with environment variables. Example Try it on On GoPlay https://goplay.tools/snippe

GOLang Sugar 4 Apr 11, 2022
Tmpl - A tool to apply variables from cli, env, JSON/TOML/YAML files to templates

tmpl allows to apply variables from JSON/TOML/YAML files, environment variables or CLI arguments to template files using Golang text/template and functions from the Sprig project.

krako 2 Nov 14, 2022
A golang package for parsing ini-style configuration files

Mini Mini is a simple ini configuration file parser. The ini syntax supported includes: The standard name=value Comments on new lines starting with #

Stephen Asbury 32 Sep 26, 2022
Go-yaml - Yaml parsing Toolkit For Golang

go-yaml 介绍 gopkg.in/yaml.v3 已经是个非常好用的包,但是在实际开发中总有类型转换带来的麻烦,go-yaml只是在它的基础上,简单的一层

xiaowu 4 Jan 13, 2022