Java properties scanner for Go

Overview

Travis CI Status License GoDoc

Overview

Please run git pull --tags to update the tags. See below why.

properties is a Go library for reading and writing properties files.

It supports reading from multiple files or URLs and Spring style recursive property expansion of expressions like ${key} to their corresponding value. Value expressions can refer to other keys like in ${key} or to environment variables like in ${USER}. Filenames can also contain environment variables like in /home/${USER}/myapp.properties.

Properties can be decoded into structs, maps, arrays and values through struct tags.

Comments and the order of keys are preserved. Comments can be modified and can be written to the output.

The properties library supports both ISO-8859-1 and UTF-8 encoded data.

Starting from version 1.3.0 the behavior of the MustXXX() functions is configurable by providing a custom ErrorHandler function. The default has changed from panic to log.Fatal but this is configurable and custom error handling functions can be provided. See the package documentation for details.

Read the full documentation on GoDoc

Getting Started

import (
	"flag"
	"github.com/magiconair/properties"
)

func main() {
	// init from a file
	p := properties.MustLoadFile("${HOME}/config.properties", properties.UTF8)

	// or multiple files
	p = properties.MustLoadFiles([]string{
			"${HOME}/config.properties",
			"${HOME}/config-${USER}.properties",
		}, properties.UTF8, true)

	// or from a map
	p = properties.LoadMap(map[string]string{"key": "value", "abc": "def"})

	// or from a string
	p = properties.MustLoadString("key=value\nabc=def")

	// or from a URL
	p = properties.MustLoadURL("http://host/path")

	// or from multiple URLs
	p = properties.MustLoadURL([]string{
			"http://host/config",
			"http://host/config-${USER}",
		}, true)

	// or from flags
	p.MustFlag(flag.CommandLine)

	// get values through getters
	host := p.MustGetString("host")
	port := p.GetInt("port", 8080)

	// or through Decode
	type Config struct {
		Host    string        `properties:"host"`
		Port    int           `properties:"port,default=9000"`
		Accept  []string      `properties:"accept,default=image/png;image;gif"`
		Timeout time.Duration `properties:"timeout,default=5s"`
	}
	var cfg Config
	if err := p.Decode(&cfg); err != nil {
		log.Fatal(err)
	}
}

Installation and Upgrade

$ go get -u github.com/magiconair/properties

License

2 clause BSD license. See LICENSE file for details.

ToDo

  • Dump contents with passwords and secrets obscured

Updated Git tags

13 Feb 2018

I realized that all of the git tags I had pushed before v1.7.5 were lightweight tags and I've only recently learned that this doesn't play well with git describe 😞

I have replaced all lightweight tags with signed tags using this script which should retain the commit date, name and email address. Please run git pull --tags to update them.

Worst case you have to reclone the repo.

#!/bin/bash
tag=$1
echo "Updating $tag"
date=$(git show ${tag}^0 --format=%aD | head -1)
email=$(git show ${tag}^0 --format=%aE | head -1)
name=$(git show ${tag}^0 --format=%aN | head -1)
GIT_COMMITTER_DATE="$date" GIT_COMMITTER_NAME="$name" GIT_COMMITTER_EMAIL="$email" git tag -s -f ${tag} ${tag}^0 -m ${tag}

I apologize for the inconvenience.

Frank

Comments
  • FTBFS: test failure:

    FTBFS: test failure: "circular reference"

    As reported in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=830727 magiconair/properties fails test on x86_64 as follows:

      === RUN   Test
      2016/07/10 22:26:08 circular reference
      exit status 1
      FAIL  github.com/magiconair/properties    0.004s
    
    opened by onlyjob 12
  • backslashes must be escaped on write

    backslashes must be escaped on write

    according to https://docs.oracle.com/cd/E23095_01/Platform.93/ATGProgGuide/html/s0204propertiesfileformat01.html The backslash character must be escaped as a double backslash

    opened by apesternikov 9
  • Make TestMustGetParsedDuration backwards compatible

    Make TestMustGetParsedDuration backwards compatible

    Fails:

    % go version go version go1.18beta2 linux/amd64

    As per:

    https://go.dev/doc/devel/release#policy
    

    Supported versions are today 1.16 and 1.17, so we can drop this brittle check.

    opened by maage 6
  • enhance error message for circular references

    enhance error message for circular references

    Note: This is an improvement over #49 for circular reference error message.

    Context

    When there are properties with circular references, the error message thrown contains one of the key in the cycle. This can be a bit annoying to fix, since it does not give a trace. Given this library supports loading properties from multiple sources, grep etc can fail.

    Input:

    foo = depends on ${bar}
    
    bar = which depends on ${baz}
    
    baz = ${foo} completes a cyclic reference
    
    

    The error message thrown when properties.MustLoadFile is called with the above file:

    circular reference in "foo = ${foo}"
    

    This pull request is an attempt to make it look better, my take:

    circular reference in:
    foo=depends on ${bar}
    bar=which depends on ${baz}
    baz=${foo} completes a cyclic reference
    

    This is not perfect, it still does not highlight the exact source of these properties, but I felt that the overhead of carrying that context to properties.expand may be a bit too much, since the method is currently stateless.

    Thanks for the library!

    opened by sriv 6
  • have symbol equal in the value property

    have symbol equal in the value property

    Hi I ran into this issue from eBay fabio, i want to set property 'registry.consul.register.tags'. i.e. "key1=val1" as one of the tag, but it has this kind of error: [FATAL] 1.1.5. circular reference

    is that a way to escape equal symbol. thanks

    invalid 
    opened by wsf1991 6
  • Support typed setter functions

    Support typed setter functions

    I'm not sure if I'm missing something, but it seems you can only set properties to strings. I know you can read other types, but what if I was to SetInt()?

    enhancement 
    opened by theherk 5
  • Bug(writeComments): Changes comments if \ is included

    Bug(writeComments): Changes comments if \ is included

    When reading comments \ are loaded correctly, but when writing they are then replaced by \\. This leads to wrong comments when writing and reading multiple times.

    I assume that it is not necessary to encode the comments.

    opened by doxsch 4
  • Support duplicate expansions in the same value

    Support duplicate expansions in the same value

    This enables properties like this:

    a = b
    c = ${a} ${a}
    

    Previously, this would incorrectly throw a circular reference error. Now instead this will correctly evaluate the above as "c = b b".

    opened by yobert 4
  • Make expansion of expressions optional per load

    Make expansion of expressions optional per load

    As part of our configuration service, we need to store configurations as is in key-value records without substituing the ${..} expressions. Application that retrieve properties from this service will do the expansions.

    My suggestion is to provide another Load function that takes the extra boolean parameter expandExpressions

    opened by emicklei 4
  • Add preserveFormatting option for comments/whitespace

    Add preserveFormatting option for comments/whitespace

    This pull request addresses a use case for reading and updating properties files that have:

    • commented out sample values for key/value entries that need to be preserved. eg:
    # Uncomment the following to override the default value
    # serverHostname = host.com
    
    • sectional comments with whitespace formatting. eg:
    # 
    # Section 1 
    #
    
    # comment1
    key = value
    

    Changes:

    • Add a preserveFormatting bool option to the loader to allow scanning and retaining whitespace as part of comments.
    • Add a preserveFormatting bool option when writing comments to pass through the whitespace formatting to the output.
    • Add a virtual key/value for the end of the input to allow trailing comments to be retained and emitted.
    • Update lexer to not discard whitespace if preserveFormatting is set.
    • Comments are now structs containing 2 elements: the original prefixes from the input, and the comment value itself.

    Notes:

    • I chose not to add a preserveFormatting option to each of the Load* methods. I felt doing this would explode the number of combinations of potential parameters.
      • Instead, the preserveFormatting option for loading is only available through the GetLoader method followed by explicitly setting the loader options before invoking the underlying Load methods.
    opened by arvindth 3
  • Do not fail on blank lines with whitespace only

    Do not fail on blank lines with whitespace only

    It's pretty easy to leave a single tab on a line after editing a large properties file. I think the parser should simply ignore such a line instead of failing.

    opened by homiak 3
  • ⚡ Speedup merge

    ⚡ Speedup merge

    The Merge function currently checks if a key in other exists in p by iterating over all keys in p which is redundant as we already have a faster way to look that up in p.m.

    This reduces the asymptotic complexity of Merge from quadratic O(N2) down to linear O(N) where N is the number of properties.

    Also wrote a benchmark to measure the exact speedup and here are the results:

    Merging hundred properties - 6x faster Merging thousand properties - 37x faster Merging 10K properties. - 300x faster Merging 100K properties - 2600x faster

    Before

    $ GOMAXPROCS=1 go test -run=^$ -bench ^BenchmarkMerge$ github.com/magiconair/properties
    goos: linux
    goarch: amd64
    pkg: github.com/magiconair/properties
    cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
    BenchmarkMerge/num_properties_100                  58174             19691 ns/op
    BenchmarkMerge/num_properties_1000                   687           1637600 ns/op
    BenchmarkMerge/num_properties_10000                    7         156907466 ns/op
    BenchmarkMerge/num_properties_100000                   1        16630149940 ns/op
    PASS
    

    After

    $ GOMAXPROCS=1 go test -run=^$ -bench ^BenchmarkMerge$ github.com/magiconair/properties
    goos: linux
    goarch: amd64
    pkg: github.com/magiconair/properties
    cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
    BenchmarkMerge/num_properties_100                 342036              3254 ns/op
    BenchmarkMerge/num_properties_1000                 27264             44239 ns/op
    BenchmarkMerge/num_properties_10000                 2245            523613 ns/op
    BenchmarkMerge/num_properties_100000                 190           6237122 ns/op
    PASS
    
    opened by AdityaVallabh 1
  • Skip ignored private fields

    Skip ignored private fields

    This pull request allows adding some private fields on properties structs without having problems, only marking this field as skipped with a specific tag as the test example below.:

    	type S struct {
    		p     string `properties:"-"`
    		X     string `properties:"-"`
    		Undef string `properties:",default=some value"`
    	}
    

    This change will avoid an error cannot set p using the struct above.

    opened by rafaelhl 1
  • Equal sign is incorrectly escaped in key

    Equal sign is incorrectly escaped in key

    I'm doing the following test:

    // Set a key to "a=b" and value "c"
    p := properties.NewProperties()
    p.Set("a=b", "c")
    buf := new(bytes.Buffer)
    p.Write(buf, properties.UTF8)
    println(buf.String())
    

    I'd espect the serialized version to be:

    a\=b = c
    

    Since the \ should be used to escape the first equal sign, but the program prints:

    a=b = c
    

    Which seems wrong. If saved in a file and read again, the key becomes "a" (while, when reading from the correct file, the key is correctly interpreted as "a=b").

    opened by nicolaferraro 0
  • Ability Decode to Multiple structs

    Ability Decode to Multiple structs

    In case the properties file has a logical division of the configurations, I'd like to decode to several dedicated structs.

    For example: Given the following config.properties file

    app.port=3000
    db.host=127.0.0.1
    db.port=8000
    db.username=username123
    db.password=password123
    pglistener.min_reconn=10 * time.Second
    pglistener.max_reconn=1 * time.Minute
    ...
    ...
    

    I'd like to decode to the following structs:

    type Config struct {
    	AppPort string `properties:"app.port,default=3000"`
    }
    
    type DBConfig struct {
    	Host          string  `properties:"db.host,default=127.0.0.1"`
    	Port           int       `properties:"db.port,default=5432"`
    	Username string `properties:"db.username"`
    	Password  string `properties:"db.password"`
    }
    
    
    type PGListener struct {
    	MinReconnectInterval time.Duration `properties:"pglistener.min_reconn"`
    	MaxReconnectInterval time.Duration `properties:"pglistener.max_reconn"`
    }
    
    opened by BoraBak 4
  • Unable to parse an array of values

    Unable to parse an array of values

    Given a properties file consisting of:

    Test = foo,bar,baz
    

    A config struct of

    type Config struct {
        Test []string `properties:"Test"`
    }
    

    And the following decoding

        var cfg Config
        if err := p.Decode(&cfg); err != nil {
            log.Fatal(err)
        }
        fmt.Printf("got test values: %s\n", cfg.Test[1])
    ```
    
    My program crashes as cfg.Test is of length 1.
    opened by ahshah 0
Owner
Frank Schröder
Distributed Systems Engineer, Go Developer, @gopcua and [email protected] maintainer. Working @northvolt building better batteries.
Frank Schröder
Go -> Haxe -> JS Java C# C++ C Python Lua

go2hx Compile: Go -> Haxe -> Js, Lua, C#, C++, Java, C, Python warning: heavily experimental still a ways to go before an alpha. Come give feedback on

go2hx 78 Nov 30, 2022
Aes for go and java; build go fo wasm and use wasm parse java response.

aes_go_wasm_java aes for go and java; build go fo wasm and use wasm parse java response. vscode setting config settings.json { "go.toolsEnvVars":

忆年 0 Dec 14, 2021
Update-java-ca-certificates - Small utility to convert the system trust store to a system Java KeyStore

update-java-ca-certificates This small utility takes care of creating a system-w

Swisscom 6 Nov 11, 2022
Super Java Vulnerability Scanner

XiuScan 不完善,正在开发中 介绍 一个纯Golang编写基于命令行的Java框架漏洞扫描工具 致力于参考xray打造一款高效方便的漏扫神器 计划支持Fastjson、Shiro、Struts2、Spring、WebLogic等框架 PS: 取名为XiuScan因为带我入安全的大哥是修君 特点

4ra1n 116 Dec 30, 2021
Tool to support the estimation for true sales prices for Danish properties.

Hjem Dette værktøj er designet til at støtte huskøbere til at kunne danne sig et overblik historiske købspriser for nærområdet givet man har udset sig

Thomas Kobber Panum 116 Nov 9, 2022
Golang library for reading properties from configuration files in JSON and YAML format or from environment variables.

go-config Golang library for reading properties from configuration files in JSON and YAML format or from environment variables. Usage Create config in

null 3 Aug 22, 2022
A protoc plugin that generates fieldmask paths as static type properties for proto messages

protoc-gen-fieldmask A protoc plugin that generates fieldmask paths as static ty

null 12 Nov 3, 2022
Rental-api - A RESTful-API that allows developers to connect to data about rental properties

Rentals-API is a RESTful-API that allows developers to connect to data about rental properties.

null 0 Jan 24, 2022
Gbu-scanner - Go Blog Updates (Scanner service)

Go Blog Updates - Scanner This service scans go blog (go.dev) and publishes new posts to message broker (rabbitmq). It uses mongodb as a storage for a

null 1 Jan 10, 2022
GONET-Scanner - Golang network scanner with arp discovery and own parser

GO/NET Scanner ScreenShots Install chmod +x install.sh ./install.sh [as root] U

Luis Javier 61 Nov 18, 2022
bluemonday: a fast golang HTML sanitizer (inspired by the OWASP Java HTML Sanitizer) to scrub user generated content of XSS

bluemonday bluemonday is a HTML sanitizer implemented in Go. It is fast and highly configurable. bluemonday takes untrusted user generated content as

Microcosm 2.5k Dec 1, 2022
XSD (XML Schema Definition) parser and Go/C/Java/Rust/TypeScript code generator

xgen Introduction xgen is a library written in pure Go providing a set of functions that allow you to parse XSD (XML schema definition) files. This li

null 207 Dec 3, 2022
Golang->Haxe->CPP/CSharp/Java/JavaScript transpiler

TARDIS Go -> Haxe transpiler Haxe -> C++ / C# / Java / JavaScript Project status: a non-working curiosity, development currently on-ice The advent of

TARDIS Go 423 Nov 27, 2022
Instant messaging platform. Backend in Go. Clients: Swift iOS, Java Android, JS webapp, scriptable command line; chatbots

Tinode Instant Messaging Server Instant messaging server. Backend in pure Go (license GPL 3.0), client-side binding in Java, Javascript, and Swift, as

Tinode 9.6k Dec 4, 2022
Go Stream, like Java 8 Stream.

Go Stream, like Java 8 Stream.

Youth.霖 70 Dec 1, 2022
minectl 🗺 is a cli for creating Minecraft (java or bedrock) server on different cloud provider.

minectl ?? minectl️️ is a cli for creating Minecraft (java or bedrock) server on different cloud provider. It is a private side project of me, to lear

Engin Diri 102 Nov 21, 2022
jacobin - A more than minimal JVM written in Go and capable of running Java 11 bytecode.

This overview gives the background on this project, including its aspirations and the features that it supports. The remaining pages discuss the basics of JVM operation and, where applicable, how Jacobin implements the various steps, noting any items that would be of particular interest to JVM cognoscenti.

Andrew Binstock 56 Nov 22, 2022
Cake is a lightweight HTTP client library for GO, inspired by Java Open-Feign.

Cake is a lightweight HTTP client library for GO, inspired by Java Open-Feign. Installation # With Go Modules, recommanded with go version > 1.16

snown 8 Oct 6, 2022
convert curl commands to Python, JavaScript, Go, PHP, R, Dart, Java, MATLAB, Rust, Elixir and more

curlconverter curlconverter transpiles curl commands into programs in other programming languages. $ curlconverter --data "Hello, world!" example.com

null 6k Nov 30, 2022
A Cloud Native Buildpack that contributes SDKMAN and uses it to install dependencies like the Java Virtual Machine

gcr.io/paketo-buildpacks/sdkman A Cloud Native Buildpack that contributes SDKMAN and uses it to install dependencies like the Java Virtual Machine. Be

Daniel Mikusa 1 Jan 8, 2022