Loads Consul service data into /_consul/service namespace. (https://github.com/kelseyhightower/confd/issues/100)

Format for keys is:

/_consul/service/<serviceName>[/<tagName>]/<nodeIndex>

Consul datacenters are not supported yet. tagName is optional. Both sets of keys are generated. Consul data is loaded as a JSON marshalled String as the Value in the keystore. Using data in template requires unmarshalling the data using the new json or jsonArray functions bounds to the template.

Additionally, added some convenience methods parent and sibling to the template for easier navigation around the KV store.

Note: this is my 1st attempt at Go, so feel free to criticize style/naming/etc. I tried to follow standard where it was apparent.

Supporting change at: https://github.com/kelseyhightower/memkv/pull/1

Firstly, -srv-domain doesn't support _etcd-client-ssl._tcp.example.com which is etcd dns srv with ssl. Even if I use -scheme https

Secondly, when I use -srv-record _etcd-client-ssl._tcp.example.com -scheme https to replace, it shows

2017-08-30T15:27:13+08:00 linux confd[9653]: INFO SRV record set to _etcd-client-ssl._tcp.example.com
2017-08-30T15:27:13+08:00 linux confd[9653]: INFO Backend set to etcd
2017-08-30T15:27:13+08:00 linux confd[9653]: INFO Starting confd
2017-08-30T15:27:13+08:00 linux confd[9653]: INFO Backend nodes set to 02.example.com:4758, 01.example.com:4758, 03.example.com:4758
2017-08-30T15:27:13+08:00 linux confd[9653]: ERROR client: etcd cluster is unavailable or misconfigured; error #0: unsupported protocol scheme "01.example.com"
; error #1: unsupported protocol scheme "03.example.com"
; error #2: unsupported protocol scheme "02.example.com"

2017-08-30T15:27:13+08:00 linux confd[9653]: FATAL client: etcd cluster is unavailable or misconfigured; error #0: unsupported protocol scheme "01.example.com"
; error #1: unsupported protocol scheme "03.example.com"
; error #2: unsupported protocol scheme "02.example.com"

It seems that -scheme doesn't work.

Finally, I use -node and it connects successfully. But it tells me that key doesn't exist, because confd prepends / to my key automatically, even if I set prefix = "" in toml.

I am running confd in a docker container, with the docker host running etcd. I am able to get values from etcdctl as well as curl, but confd gives the 501: All the given peers are not reachable. Bash script (for testing) running inside the container:

#!/bin/bash
/usr/global/apps/etcdctl --no-sync -peers http://172.17.42.1:4001 ls system/zookeeper
echo "curl test: " $(curl -X GET http://172.17.42.1:4001/v2/keys/system/zookeeper?consistent=true&recursive=false&sorted=false)
/usr/global/apps/confd -onetime=true -verbose=true -debug=true -node="172.17.42.1:4001" -confdir=/usr/global/config/shared/zookeeper/confd/

And here is the output I am getting:

/system/zookeeper/zookeeper-1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   294    0   294    0     0   169k      0 --:--:-- --:--:-- --:--:--  287k
test:  {"action":"get","node":{"key":"/system/zookeeper","dir":true,"nodes":[{"key":"/system/zookeeper/zookeeper-1","value":"{\"id\":\"1\",\"ip\":\"10.1.51.252\",\"cport\":\"49153\",\"pport\":\"49154\",\"lport\":\"49155\"}","modifiedIndex":30,"createdIndex":30}],"modifiedIndex":29,"createdIndex":29}}
2014-10-15T12:54:51-04:00 78d09c06c9fc /usr/global/apps/confd[16]: WARNING Skipping confd config file.
2014-10-15T12:54:51-04:00 78d09c06c9fc /usr/global/apps/confd[16]: NOTICE Backend set to etcd
2014-10-15T12:54:51-04:00 78d09c06c9fc /usr/global/apps/confd[16]: NOTICE Starting confd
2014-10-15T12:54:51-04:00 78d09c06c9fc /usr/global/apps/confd[16]: NOTICE Backend nodes set to 172.17.42.1:4001
2014-10-15T12:54:51-04:00 78d09c06c9fc /usr/global/apps/confd[16]: DEBUG Loading template resources from confdir /usr/global/config/shared/zookeeper/confd/
2014-10-15T12:54:51-04:00 78d09c06c9fc /usr/global/apps/confd[16]: DEBUG Processing template resource /usr/global/config/shared/zookeeper/confd/conf.d/zookeeperconf.toml
2014-10-15T12:54:51-04:00 78d09c06c9fc /usr/global/apps/confd[16]: DEBUG Loading template resource from /usr/global/config/shared/zookeeper/confd/conf.d/zookeeperconf.toml
2014-10-15T12:54:51-04:00 78d09c06c9fc /usr/global/apps/confd[16]: ERROR 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
2014-10-15T12:54:51-04:00 78d09c06c9fc /usr/global/apps/confd[16]: DEBUG Retrieving keys from store
2014-10-15T12:54:51-04:00 78d09c06c9fc /usr/global/apps/confd[16]: DEBUG Key prefix set to /

The top two lines show the correct ip and port are being used (unlike https://github.com/kelseyhightower/confd/issues/130) versions: etcd version 0.4.5 etcdctl version 0.4.5 confd 0.6.3

Hi,

I'm trying to use this to configure haproxy on a 1 node etcd cluster for testing I can add keys to Etcd and Confd builds my config and reloads haproxy fine, but if I remove keys, the config file it makes matches the old one. That is until I restart Confd then it builds the file properly.

I've noticed this using -watch and -interval watch: http://pastebin.com/F0uBNbEi interval: http://pastebin.com/HcaJRmbt

Cheers

We have paths in etcd as follows:

$ etcdctl ls /deis/services/inward-overalls
/deis/services/inward-overalls/inward-overalls_v2.web.1
/deis/services/inward-overalls/inward-overalls_v2.web.3
/deis/services/inward-overalls/inward-overalls_v2.web.4

I'm trying to template this with confd. It seems like I can match these keys as follows:

{{ gets "/deis/services/*/*" }} 

However, I need to use the elements of the path in my template logic. In this case, I need to write out inward-overalls in my template. I wish I could do something like this:

{{ range gets "/deis/services/*/*" }} 
foo {{ Path.split(.Key)[2] }}
{{ end }} 

...but I'm not sure how arbitrary function calls work in golang templates. This just looks wrong.

Could we define additional helper methods to use for this? It seems that something like that would be useful - paths are important. I would love to have the ability to match/enumerate paths, not just keys. For example:

{{ gets "/deis/services/*" }}

Then I could reference the children of this path that aren't necessarily keys. I think this was an advantage of having Nodes in the old confd.

What is the recommended way to do this now?

Though the documentation says just run go build, the following packages are still telling me I need to use go get prior to the build. Am I doing something wrong here or is this in fact the necessary steps?

[/repositories/git/opensource/confd]$ go build
config.go:14:2: cannot find package "github.com/BurntSushi/toml" in any of:
    /usr/local/go/src/github.com/BurntSushi/toml (from $GOROOT)
    /repositories/git/opensource/confd/src/github.com/BurntSushi/toml (from $GOPATH)
confd.go:10:2: cannot find package "github.com/kelseyhightower/confd/backends" in any of:
    /usr/local/go/src/github.com/kelseyhightower/confd/backends (from $GOROOT)
    /repositories/git/opensource/confd/src/github.com/kelseyhightower/confd/backends (from $GOPATH)
confd.go:11:2: cannot find package "github.com/kelseyhightower/confd/log" in any of:
    /usr/local/go/src/github.com/kelseyhightower/confd/log (from $GOROOT)
    /repositories/git/opensource/confd/src/github.com/kelseyhightower/confd/log (from $GOPATH)
confd.go:12:2: cannot find package "github.com/kelseyhightower/confd/resource/template" in any of:
    /usr/local/go/src/github.com/kelseyhightower/confd/resource/template (from $GOROOT)
    /repositories/git/opensource/confd/src/github.com/kelseyhightower/confd/resource/template (from $GOPATH)

[/repositories/git/opensource/confd]$ ./build
Building confd...
config.go:14:2: cannot find package "github.com/BurntSushi/toml" in any of:
    /usr/local/go/src/github.com/BurntSushi/toml (from $GOROOT)
    /repositories/git/opensource/confd/src/github.com/BurntSushi/toml (from $GOPATH)
confd.go:10:2: cannot find package "github.com/kelseyhightower/confd/backends" in any of:
    /usr/local/go/src/github.com/kelseyhightower/confd/backends (from $GOROOT)
    /repositories/git/opensource/confd/src/github.com/kelseyhightower/confd/backends (from $GOPATH)
confd.go:11:2: cannot find package "github.com/kelseyhightower/confd/log" in any of:
    /usr/local/go/src/github.com/kelseyhightower/confd/log (from $GOROOT)
    /repositories/git/opensource/confd/src/github.com/kelseyhightower/confd/log (from $GOPATH)
confd.go:12:2: cannot find package "github.com/kelseyhightower/confd/resource/template" in any of:
    /usr/local/go/src/github.com/kelseyhightower/confd/resource/template (from $GOROOT)
    /repositories/git/opensource/confd/src/github.com/kelseyhightower/confd/resource/template (from $GOPATH)

If this is in fact the correct approach, could we add it to the documentation, the ./build script and/or perhaps the new Vendor subdirectory? I know its pretty obvious on how to get the packages updated, but not having it in the build from source documentation lead me to think I was doing something wrong at first....might help others avoid the same suspicion (unless I am doing something wrong!).

When a key is missing or not found, the entire template stops processing. I saw there was an old issue about this and it was decided to continue generating the template - was that abandoned?

I am doing:

{{range $server := gets "/servers/*"}}

If the /servers/ directory is empty, I get:

executing "haproxy.cfg.tmpl" at <gets "/servers/...>: error calling gets: no keys match

confd then stops processing.

I would expect this to continue executing and throw the error or have a way for me to check that keys exists in there before running the gets. exists does not work because I will not know what would or would not exist in that servers dir.

Im running confd to monitor etcd for changes to update my HAProxy config, and I noticed that when I try using {{if exists /some/dir}}} it seems to fail even though the directory exists.

Here's a fuller example:

values in etcd

/apps/webapp/node-1
/apps/webapp/node-2

/etc/confd/conf.d/haproxy.toml

keys = [
    "/apps"
]

/etcd/confd/templates/haproxy.cfg.toml

{{if exists "/apps/webapp"}}
    {{range getvs "/apps/webapp/*"}}
        {{.}}
    {{end}}
{{end}}

So what Im wondering is, should if exists work on etcd directories, or is it limited to keys only? If it doesn't' currently work for directories, do you think it would make sense to add support for it? With this given use case, apps could register themselves under the /apps directory, making it so you don't have to update the list of keys. It also makes confd less prone to crashing when a specified key doesnt exist since everything will be nested under /app, rather than explicitly specifying a key/path for each app.

I took a quick stab at integrating etcd and consul's watch features per issue #56. This update basically wraps the go-etcd Watch command and consul's WatchList so they both have the same interface (go-etcd uses goroutines/channels while consul is a long-poll).

It adds a -watch flag that takes precedence over any intervals: confd -watch -node 'http://127.0.0.1:4001' -confdir ~/confd

My templates often end up being far too verbose because of things like this:

option = {{if exists "/example/app/option"}}{{getv "/example/app/option"}}{{else}}default_value{{end}}

That makes it quite hard to visually parse the config files. Much easier would be something like:

option = {{getv "/example/app/option" "default_value"}}

I'm running confd v0.5.0 on an Ubuntu 14.04 LTS host and am having trouble with altering the owner and group for a template...no matter what I do, it changes things to be owned by the root user and root group, which is not desirable.

/etc/confd/conf.d/example-json.toml

[template]
src = "example.json.tmpl"
dest = "/etc/example.json"
owner = "elasticdog"
group = "elasticdog"
mode = "0644"
keys = [
  "/jdbc_url",
]
reload_cmd = "/sbin/reload example-server"

...and it's creating the file with the desired contents, but the permissions are always set back to 0:

$ ls -l /etc/example.json
-rw-r--r-- 1 root root 1011 Aug  7 04:54 /etc/example.json
$ id elasticdog
uid=999(elasticdog) gid=999(elasticdog) groups=999(elasticdog)
$ sudo chmod elasticdog:elasticdog /etc/example.json
$ sudo reload confd

/var/log/upstart/confd.log

2014-08-07T04:54:21Z ip-10-193-17-174 /usr/local/bin/confd[2404]: INFO /etc/example.json has UID 999 should be 0
2014-08-07T04:54:21Z ip-10-193-17-174 /usr/local/bin/confd[2404]: INFO /etc/example.json has GID 999 should be 0

Hi,

Needing advanced functions to test the fact a string is numeric (key having a path starting by /0/), I miss sprig functions in confd, the one available with helm templating. This PR bring us sprig templating functions in confd, to avoid any collision between available in confd functions and the one brought by sprig, all the funcs are imported prefixed by sprig...

Example: sprigRegexMatch matches regexMatch in sprig

Thx for the consideration you will have for this PR.

I has been four years since last release, even though the homebrew formula is still buildable, but it does not include the latest features.

relates to #861

go install github.com/kelseyhightower/conf resulted in go: finding module for package code.google.com/p/go.crypto/openpgp ......\go\pkg\mod\github.com\xordataexchange\[email protected]\encoding\secconf\secconf.go:15:2: cannot find module providing package code.google.com/p/go.crypto/openpgp: unrec ognized import path "code.google.com/p/go.crypto/openpgp": parse https://code.google.com/p/go.crypto/openpgp?go-get=1: no go-import meta tags (meta tag github.com/golang/go did not match import path code.google.com/p/go.crypto/openpgp)

https://github.com/xordataexchange/crypt/issues/34 says github.com\xordataexchange\crypt is not active, use https://github.com/bketelsen/crypt.

With the following tmpl codes, I intended to have the {{$test}} variable is "abcabc" at the line 6. While the result is "abc" which means the changes in {{$if}} block doesn't take effect. How could I achieve this target? What's more, the value of $test is abcabc at line 4.

Thanks

1. {{$test := "abc" -}}
2. {{if eq $test "abc" -}}
3. {{$test := printf "%s%s" $test $test}}
4. {{$test}}
5. {{end}}
6. {{$test}}

It appears that confd does not re-resolve etcd SRV records after startup. This makes the SRV support actually quite dangerous to use if you ever intend to change the contents of the record...

We had many confds across our fleet 'stuck' attempting to look up the previous, decommissioned members of our etcd cluster: Aug 03 13:43:57 deploy1002 confd[32274]: 2022-08-03T13:43:57Z deploy1002 /usr/bin/confd[32274]: ERROR client: etcd cluster is unavailable or misconfigured; error #0: dial tcp: lookup conf1004.eqiad.wmnet on 10.3.0.1:53: no such host Aug 03 13:43:57 deploy1002 confd[32274]: ; error #1: dial tcp: lookup conf1006.eqiad.wmnet on 10.3.0.1:53: no such host Aug 03 13:43:57 deploy1002 confd[32274]: ; error #2: dial tcp: lookup conf1005.eqiad.wmnet on 10.3.0.1:53: no such host

Meanwhile the SRV record had looked like this in our DNS for at least a week's time: _etcd._tcp.eqiad.wmnet has SRV record 0 1 4001 conf1008.eqiad.wmnet. _etcd._tcp.eqiad.wmnet has SRV record 0 1 4001 conf1009.eqiad.wmnet. _etcd._tcp.eqiad.wmnet has SRV record 0 1 4001 conf1007.eqiad.wmnet.

This is with version: confd 0.16.0 (Git SHA: , Go Version: go1.11.6)