Does the "keys" really work?

[ozbillwang]

Read the quick start guide to create a template resource config file as below.

$ cat /etc/confd/conf.d/myconfig.toml
...
keys = [
    "/myapp/database/url",
    "/myapp/database/user",
]
...
$ cat /etc/confd/templates/myconfig.conf.tmpl

[myconfig]
database_url = {{getv "/myapp/database/url"}}
database_user = {{getv "/myapp/database/user"}}

As my understanding is, confd will replace with real secrets for all keys list in /etc/confd/conf.d/myconfig.toml .

But in fact, it still overrides all keys. If I remove one key from /etc/confd/conf.d/myconfig.toml,

$ cat /etc/confd/conf.d/myconfig.toml
...
keys = [
    "/myapp/database/user",
]
...

the output file should only be replaced by one key /myapp/database/user. But I found all confd key {{ getv ....}} are replaced with secrets.

[hubo1016]

Which backend is used?

[ozbillwang]

Good question. @hubo1016

I used the un-merged feature (https://github.com/kelseyhightower/confd/pull/697) with new backend secretsmanager and saw this issue.

Will test with other backend later.

[jrhoward]

I think it may be specific to SecretsManager.

I am looking at a fix now

[jrhoward]

I believe this will happen if you remove the key an a reference to it still exists in the template.

ERROR template: myconfig.conf.tmpl:2:17: executing "myconfig.conf.tmpl" at <getv "/myapp/databas...>: error calling getv: key does not exist: /myapp/database/url

At the moment Secrets Manager does not do that however.

[ozbillwang]

@hubo1016

Get some helps from @jrhoward , but seems this is related wth the usage in confd . I update my request in another way to explain what I need:

My purpose is, I need convert secrets for different environments.

$ cat /etc/confd/conf.d/myconfig-dev.toml
[template]
src = "myconfig.conf.tmpl"
dest = "/tmp/myconfig.conf"
keys = [
    "/myapp/database/dev/user",
]

$ cat /etc/confd/templates/myconfig.conf.tmpl
[dev]
database_user = {{getv "/myapp/database/dev/user"}}
[staging]
database_user = {{getv "/myapp/database/staging/user"}}

So each time, I only need convert several secrets for nominated environment which defined in /etc/confd/conf.d/myconfig-dev.toml, whatever all keys are marked by {{ getv ...}}.

Because in aws dev account, there is not key of /myapp/database/staging/user If confd try to convert all keys, the command will be failed.

Not sure how to do that.

[jrhoward]

how about this?

{{if exists "/myapp/database/dev/user" }}
[dev]
database_user = {{getv "/myapp/database/dev/user"}}
{{else if exists "/myapp/database/staging/user" }}
[staging]
database_user = {{getv "/myapp/database/staging/user"}}
{{end}}

[ozbillwang]

Thanks, @jrhoward

I have more than 20 secrets for each environment, this will be big job for me .

[okushchenko]

@ozbillwang I suggest you to use a different prefix for different environments, then use the -prefix option of confd to get only the keys that are relevant for the given environment. Environment value can be injected from the environment variables like that -prefix /myapp/database/$ENVIRONMENT. You can then reference the keys in confd without the prefix part, which is very convenient. Also, I suggest you to prefix all of the keys with the /$SERVICE/$ENVIRONMENT string (SERVICE=myapp ENVIRONMENT=staging), which then can be used as a prefix in confd.

[hubo1016]

Can you set the keys with prefix?

[template]
src = "myconfig.conf.tmpl"
dest = "/tmp/myconfig.conf"
keys = [
    "/myapp/database",
]

then

{{- range gets "/myapp/database/*/user" }}
[{{- .Key | dir | base }}]
database_user = {{ .Value }}
{{- end }}

[jarikujansuu]

I got same problem using ssm backend. We have multiple services that can be deployed in different flavor, and then there is configuration that is for flavor and then for flavor & service combo.

So I have 2 step process where first replace keys from prefix "/<flavor>/<service>" and then use result of that with prefix "/flavor/"

But unfortunately it fails on first step because it can't find the key because that key was supposed to be replaced in next step. <getv "/db/database">: error calling getv: key does not exist: /db/database