A flexible and composable configuration library for Go that doesn't suck

Overview

croconf

A flexible and composable configuration library for Go

Why?

We know that there are plenty of other Go configuration and CLI libraries out there already - insert obligatory xkcd... πŸ˜… Unfortunately, most (all?) of them suffer from at least one of these serious issues and limitations:

  1. Difficult to test:
    • e.g. they rely directly on os.Args() or os.Environ() or some other shared global state
    • can't check what results various inputs will produce without a lot of effort for managing that state
  2. Difficult or impossible to extend - some variation of:
    • limited value sources, e.g. they might support CLI flags and env vars, but not JSON or YAML
    • you can't easily write your own custom first-class option types or value sources
    • the value sources are not layered, values from different sources may be difficult or impossible to merge automatically
  3. Untyped and reflection-heavy:
    • they fail at run-time instead of compile-time
    • e.g. your app panics because the type for some infrequently used and not very well tested option doesn't implement encoding.TextUnmarshaler
    • struct tags are used for everything 😱
    • alternatively, you may have to do a ton of type assertions deep in your codebase
  4. Un-queriable:
    • there is no metadata about the final consolidated config values
    • you cannot know if a certain option was set by the user or if its default value was used
    • you may have to rely on null-able or other custom wrapper types for such information
  5. Too string-y:
    • you have to specify string IDs (e.g. CLI flag names, environment variable names, etc.) multiple times
    • a typo in only some of these these strings might go unnoticed for a long while or cause a panic
  6. Terrible error messages:
    • users of a Go application don’t need to know Go implementation details like strconv.ParseInt or

The impetus for croconf was k6's very complicated configuration. We have a lot of options and most options have at least 5 hierarchical value sources: their default values, JSON config, exported options in the JS scripts, environment variables, and CLI flag values. Some options have more... 😭

We currently use several Go config libraries and a lot of glue code to manage this, and it's still a frequent source of bugs and heavy technical debt. As far as we know, no single other existing Go configuration library is sufficient to cover all of our use cases well. And, from what we can see, these issues are only partially explained by Go's weak type system...

So when we tried to find a Go config library that avoids all of these problems and couldn't, croconf was born! πŸŽ‰

Architecture

⚠️ croconf is still in the "proof of concept" stage

The library is not yet ready for production use. It has bugs, not all features are finished, comments and tests are spotty, and the module structure and type names are expected to change a lot in the coming weeks.

In short, croconf shouldn't suffer from any of the issues ⬆️ , hopefully without introducing any new ones! 🀞 It should be suitable for any size of a Go project - from the simplest toy project, to the most complicated CLI application and everything in-between!

Some details about croconf's API design

  • it uses type safe, plain old Go values for the config values
  • works for standalone values as well as struct properties
  • everything about a config field is defined in a single place, no string identifier has to ever be written more than once
  • after consolidating the config values, you can query which config source was responsible for setting a specific value (or if the default value was set)
  • batteries included, while at the same time completely extensible:
    • built-in frontends for all native Go types, incl. encoding.TextUnmarshaler and slices
    • support for CLI flags, environment variables and JSON options (and others in the future) out of the box, with zero dependencies
    • none of the built-in types are special, you can easily add custom value types and config sources by implementing a few of the small well-defined interfaces in types.go
  • no unsafe and no magic ✨
  • no reflect and no type assertions needed for user-facing code (both are used very sparingly internally in the library)

These nice features and guarantees are achieved because of the type-safe lazy bindings between value destinations and source paths that croconf uses. The configuration definition just defines the source bindings for every value, the actual resolving is done as a subsequent step.

Example

// SimpleConfig is a normal Go struct with plain Go property types.
type SimpleConfig struct {
	RPPs int64
	DNS  struct {
		Server net.IP // type that implements encoding.TextUnmarshaler
		// ... more nested fields
	}
	// ... more config fields...
}

// NewScriptConfig defines the sources and metadata for every config field.
func NewScriptConfig(
	cm *croconf.Manager, cliSource *croconf.SourceCLI,
	envVarsSource *croconf.SourceEnvVars, jsonSource *croconf.SourceJSON,
) *SimpleConfig {
	conf := &SimpleConfig{}

	cm.AddField(
		croconf.NewInt64Field(
			&conf.RPPs,
			jsonSource.From("rps"),
			envVarsSource.From("APP_RPS"),
			cliSource.FromNameAndShorthand("rps", "r"),
			// ... more bindings - every field can have as many or as few as needed
		),
		croconf.WithDescription("number of virtual users"),
		croconf.IsRequired(),
		// ... more field options like validators, meta-information, etc.
	)

	cm.AddField(
		croconf.NewTextBasedField(
			&conf.DNS.Server,
			croconf.DefaultStringValue("8.8.8.8"),
			jsonSource.From("dns").From("server"),
			envVarsSource.From("APP_DNS_SERVER"),
		),
		croconf.WithDescription("server for DNS queries"),
	)

	// ... more fields

	return conf
}

func main() {
	configManager := croconf.NewManager()
	// Manually create config sources - fully testable, no implicit shared globals!
	cliSource := croconf.NewSourceFromCLIFlags(os.Args[1:])
	envVarsSource := croconf.NewSourceFromEnv(os.Environ())
	jsonSource := croconf.NewJSONSource(getJSONConfigContents())

	config := NewScriptConfig(configManager, cliSource, envVarsSource, jsonSource)

	if err := configManager.Consolidate(); err != nil {
		log.Fatalf("error consolidating the config: %s", err)
	}

	jsonResult, err := json.MarshalIndent(config, "", "    ")
	if err != nil {
		log.Fatalf("error marshaling JSON: %s", err)
	}
	fmt.Fprint(os.Stdout, string(jsonResult))
}

This was a relatively simple example taken from here, and it still manages to combine 4 config value sources! For other examples, take a look in the examples folder in this repo.

Origins of name

croconf comes from croco-dile conf-iguration. So, 🐊 not πŸ‡­πŸ‡· πŸ˜„ And in the tradition set by k6, if we don't like it, we might decide to abbreviate it to c6 later... πŸ˜…

Remaining tasks

As mentioned above, this library is still in the proof-of-concept stage. It is usable for toy projects and experiments, but it is very far from production-ready. These are some of the remaining tasks:

  • Refactor module structure and type names
  • More value sources (e.g. TOML, YAML, INI, etc.) and improvements in the current ones
  • Add built-in support for all Go basic and common stdlib types and interfaces
  • Code comments and linter fixes
  • Fix bugs and write a lot more tests
  • Documentation and examples
  • Better (more user-friendly) error messages
  • An equivalent to cobra or kong, a wrapper for CLI application frameworks that is able to handle CLI sub-commands, shell autocompletion, etc.
  • Add drop-in support for marshaling config structs (e.g. to JSON) with the same format they were unmarshaled from.
  • Be able to emit errors on unknown CLI flags, JSON options, etc.
Issues
  • Ned's very WIP proposal for the API

    Ned's very WIP proposal for the API

    This is very, very far from final, but it's an expansion on the ad-hoc proposal I sent in Slack, together with the rough requirements in and a basic TODO proposal the README.

    opened by na-- 0
  • WIP proposal 2, with very basic (and ugly) CLI flag support

    WIP proposal 2, with very basic (and ugly) CLI flag support

    As you can see, pflag requires a ton of boilerplate, especially to distinguish between positional and flag arguments... :disappointed: And we don't use the majority of its features anyway. So we desperately needs a better library or to write our own CLI parser from scratch, but this serves as a useful demo for now...

    opened by na-- 0
  • Custom Types

    Custom Types

    What I don't like from the latest API's iteration is the requirement to re-declare the type into the To method. It's repetition and it opens to have a mismatch between types.

    I tried to implement a solution to remove it and re-use the already passed type in the first argument. I know, it requires using interface{} and type checking that is not optimal.

    opened by codebien 0
  • `TODO`

    `TODO`

    We can do: - base types besides int64 and string: float{,32,64}, int, uint, etc. - tests for the current behavior and :arrow_up: - parsing CLI flags

    Items with major unknowns: - manager options - default values (for --help) - slices (e.g. a value []int64) - nested/deep values

    opened by na-- 0
  • Basic types extension

    Basic types extension

    status for int(), uint(), float(*):

    • env vars

      • [x] code
      • [x] tests
    • [x?] json

      • [x] code
      • [x?] tests (but fails for still unknown reasons)
    • [ ] flags:

      • [x] code
      • [ ] tests
    opened by artych 0
  • Add a PoC for lazily parsing nested JSON options

    Add a PoC for lazily parsing nested JSON options

    This will probably require more refactoring in the future, and some caching, but this should be ok as a PoC :tada:

    opened by na-- 0
  • Add unified number parsing functions in sources

    Add unified number parsing functions in sources

    This preserves the completely type-safe "frontends" (i.e. Field constructors like NewInt64Field(dest *int64, ...)), but makes the job of implementing a Source much simpler. By using the biggest types, we don't need to implement a method for every single base Go number type (int, int8, int16, int32, int64, uint, uint8, uint16, uint32, uint64, byte, float32, float64), instead just 3 (int64, uint64, float64) and some extra validation.

    opened by na-- 0
  • Flags parser

    Flags parser

    opened by codebien 0
  • Add a PoC for slice fields

    Add a PoC for slice fields

    I am not super satisfied with this, but it will probably do for now, so yey! :tada: :sweat_smile:

    I've only implemented slice support in the env vars and JSON sources so far, and I am not sure how to allow the configuration of different delimiters for the env vars, so please share if you have any ideas.

    opened by na-- 0
Owner
Grafana Labs
Grafana Labs is behind leading open source projects Grafana and Loki, and the creator of the first open & composable observability platform.
Grafana Labs
A flexible and composable configuration library for Go that doesn't suck

croconf A flexible and composable configuration library for Go Why? We know that there are plenty of other Go configuration and CLI libraries out ther

Grafana Labs 13 Jul 15, 2021
Go configuration with fangs

Viper v2 feedback Viper is heading towards v2 and we would love to hear what you would like to see in it. Share your thoughts here: https://forms.gle/

Steve Francia 16.3k Jul 25, 2021
Composable, observable and performant config handling for Go for the distributed processing era

Konfig Composable, observable and performant config handling for Go. Written for larger distributed systems where you may have plenty of configuration

Lalamove 612 Jul 26, 2021
Library providing routines to merge and validate JSON, YAML and/or TOML files

CONFLATE Library providing routines to merge and validate JSON, YAML, TOML files and/or structs (godoc) Typical use case: Make your application config

Andy 19 May 20, 2021
Butler CMS (Configuration Management System)

Butler CMS Butler CMS (Configuration Management System) Overview The Butler CMS (butler) tool is designed to grab any configuration files, defined in

Adobe, Inc. 13 Apr 26, 2021
An opinionated configuration loading framework for Containerized and Cloud-Native applications.

Opinionated configuration loading framework for Containerized and 12-Factor compliant applications. Read configurations from Environment Variables, an

Sherif Abdel-Naby 73 Jul 4, 2021
Open-metrics endpoint collector for ONTAP

NetApp Harvest 2.0 The swiss-army knife for monitoring datacenters. The default package collects performance, capacity and hardware metrics from ONTAP

NetApp 39 Jul 23, 2021
✨Clean and minimalistic environment configuration reader for Golang

Clean Env Minimalistic configuration reader Overview This is a simple configuration reading tool. It just does the following: reads and parses configu

Ilya Kaznacheev 314 Jul 22, 2021
HCL is the HashiCorp configuration language.

HCL HCL is a toolkit for creating structured configuration languages that are both human- and machine-friendly, for use with command-line tools. Altho

HashiCorp 3.6k Jul 25, 2021
Small library to read your configuration from environment variables

envconfig envconfig is a library which allows you to parse your configuration from environment variables and fill an arbitrary struct. See the example

Vincent Rischmann 205 Jul 15, 2021
Library for setting values to structs' fields from env, flags, files or default tag

Configuration is a library for injecting values recursively into structs - a convenient way of setting up a configuration object. Available features:

Bogdan Daragan 39 Jun 3, 2021
Light weight, extensible configuration management library for Go. Built in support for JSON, TOML, YAML, env, command line, file, S3 etc. Alternative to viper.

koanf (pronounced conf; a play on the Japanese Koan) is a library for reading configuration from different sources in different formats in Go applicat

Kailash Nadh 541 Jul 23, 2021
Golang Configuration tool that support YAML, JSON, TOML, Shell Environment

Configor Golang Configuration tool that support YAML, JSON, TOML, Shell Environment (Supports Go 1.10+) Usage package main import ( "fmt" "github.c

Jinzhu 1.2k Jul 22, 2021
Load configuration in cascade from multiple backends into a struct

Confita is a library that loads configuration from multiple backends and stores it in a struct. Supported backends Environment variables JSON files Ya

Heetch 399 Jul 2, 2021