Vault KVs are not dynamically rendered by Consul-Template

[sellymind]

Consul Template version

v0.26.0 (3b7f233a) also I'm tried to use https://github.com/ekristen/consul-template-plugin-savetofile plugin

Configuration

# Copy-paste your configuration files here. Only include what is necessary or
# what you've changed from defaults. Include all referenced configurations.
vault {
  address = "https://<vault_adress>"
  token = <vault_token>
  unwrap_token = false
  renew_token = true
  retry {
    enabled = true
    attempts = 60
    backoff = "300ms"
    max_backoff = "1m"
  }
  ssl {
    enabled = false
  }
}
syslog {
  enabled = true
  facility = "LOCAL5"
}
deduplicate {
  enabled = false
}

# Copy-paste your Consul Template template here

template { destination = "/local/domain_list.txt" create_dest_dirs = true contents = <<EOH {{ $kv_path := "/secret//" }} {{ range secrets $kv_path }} {{ $domain := . }}{{ $savetofile := "/local/daemon/bin/savetofile" }}{{ $domain }} {{ if $domain }} {{ $path := "/local/certs/live/" }} {{ with secret (printf "/secret//%s" .) }}{{ range $k, $v := .Data }} {{ if eq $k "ca.cert" }} {{ $v | plugin $savetofile "create" (print $path $domain "_ecc/" $k) "1000" "1000" "775" }} {{ else }} {{ if eq $k "crt.cert" }} {{ $v | plugin $savetofile "create" (print $path $domain "_ecc/" $k) "1000" "1000" "775" }} {{ else }} {{ if eq $k "priv.key" }} {{ $v | plugin $savetofile "create" (print $path $domain "_ecc/" $k) "1000" "1000" "775" }} {{ else }} {{ if eq $k "fullchain" }} {{ $v | plugin $savetofile "create" (print $path $domain "_ecc/" $k) "1000" "1000" "755" }} {{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }}{{ end }} EOH error_on_missing_key = false perms = 0775 backup = true wait { min = "2s" max = "30s" } } exec { command = "/bin/systemctl reload nginx" }

# Include sample data you reference in the template from Consul or Vault here.

Debug output

2021-08-11T11:16:15.223+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.224+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.225+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.227+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.228+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.229+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.230+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.231+0300 [DEBUG] (cli) receiving signal "urgent I/O condition" 2021-08-11T11:16:15.232+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.233+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.234+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.235+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.237+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.238+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) rendering "(dynamic)" => "/local/adtech/certbot/live/domain_list.txt" 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) diffing and updating dependencies 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) vault.read(secret//) is still needed 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) vault.list(secret//) is still needed 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) vault.read(secret//) is still needed 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) vault.read(secret//) is still needed 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) vault.read(secret//) is still needed 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) vault.read(secret//) is still needed 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) vault.read(secret//) is still needed 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) vault.read(secret//) is still needed ......... 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) vault.read(secret//) is still needed 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) vault.read(secret//) is still needed 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) watching 21 dependencies 2021-08-11T11:16:15.239+0300 [DEBUG] (runner) all templates rendered 2021-08-11T11:16:15.240+0300 [DEBUG] (runner) enabling template-specific quiescence for "fa2b1b30b723a7f6097ca46735dfdf5c" 2021-08-11T11:16:15.240+0300 [INFO] (child) spawning: /bin/touch /tmp/test.txt 2021-08-11T11:16:15.240+0300 [DEBUG] (cli) receiving signal "child exited" 2021-08-11T11:16:15.240+0300 [INFO] (runner) child process died 2021-08-11T11:16:15.240+0300 [ERR] (cli) child process died with exit code 0

Expected behavior

I'm trying to sync web certificates located in Hashicorp Vault to the localhost with Consul Template on it. To sync multiply KVs to multiply files I'm using savetofile plugin as it would be impossible to create template for each certificate and each domain (to many of them) When values are changed in Vault they should also get rendered by consul template and get written to the corresponding files

Actual behavior

With the first run of Consul-Tempalte all values are written to the corresponding files All further changes in Vault are not rendered.

Steps to reproduce

1. Launch Consul Template with the provided config and template
2. Check that all KV from the Vault gets rendered to the corresponding folder and file on the localhost (via "savetofile" plugin)
3. Change value inside the Vault
4. Wait for chages to be writen to the cert files
5. Changes are not propagated to the localhost with the Consul Template running

References

[karl-cardenas-coding]

I'm seeing similar behavior with version 0.27.0, but only in this case it's not honoring the wait value.

[abbasalizaidi]

@eikenb Any updates on this?

I am facing a similar problem. When rotating the static role in the vault no change was recorded by consul-template. This was working in version 0.19.4 (the current version I am using). I am trying to upgrade because I need this fix - https://github.com/hashicorp/consul-template/pull/1451

[eikenb]

Sorry everyone for the delay looking at this but I am gearing up toward working on consul-template 0.28.0 and I'll include looking into this bug with that work. Added it to the milestone and will hopefully start on this soon (series of small but hot issues have put this off a bit but I'm hopefully wrapping those up).

[eikenb]

I'm looking into this now, trying to reproduce it with a simple case. I'm going to use the new writeToFile template function.

If anyone has tips for a simple reproduction case please feel free to speak up.

[karl-cardenas-coding]

@eikenb I got around it by using

  exec {
    command = "/bin/rotate_key.sh"
  } 

   wait {
    min = "10s"
    max = "30s"
  }

Adding the wait block with a 10s minimum solved it for me.

Link to code (HashiCorp internal repository)

[eikenb]

Thanks @karl-cardenas-coding!

[eikenb]

So the issue here has nothing to do with the savetofile and what-not... it is all about consul-template not picking up on updates from Vault. Is this right?

Assuming so and trying for a basic setup to reproduce this, I have this script...

vault kv put secret/foo pass=bar

cat > template.tmpl << EOF
{{ with secret "secret/foo" }}{{ .Data.data.pass }}{{ end }}
EOF

#consul-template -template template.tmpl -once -dry
consul-template -log-level warn -template template.tmpl:out \
    -vault-default-lease-duration=20s \
    -exec 'tail -f out'

Unfortunately this doesn't reproduce the issue. After running that and seeing the 'bar' output I run vault kv put secret/foo pass=zed and after a few seconds (whatever is left of the duration) it sees it and spits out the 'zed'.

It properly updates the value if I change it from the

[eikenb]

Hey @sellymind, sorry for the delay in getting to this.

I have a question. How are you specifying the expiration/duration of the certificates you are using?

For context.. Vault doesn't support blocking queries, so consul-template must poll. It polls at a rate based on secret and its duration. It supports a number of ways that Vault can dictate this duration..

• lease duration (this can be set multiple ways)
• secret TTL
• secret rotation period It uses these to determine the base duration of the secret, then if it is renewable it starts trying to renew it at around 33% of the duration and if it is not renewable it tries to use it as long as possible and doesn't retry until around 85% of the duration. If none of these are set it defaults to a duration of 5 minutes.

[eikenb]

Hey @karl-cardenas-coding, mind going into a bit of details about the issue as you found it. I'm still trying to reproduce it and the presence of wait being set has made no difference in my tests.