pkiCert not rewriting existing certificate

[Inotsacred]

Hi! I have issued certificate by old CA. Create new Vault PKI and target consul template to old .pem file. After start consul I've expect replace old CA certificate instantly or after ttl coming close to end. but consul-template execution doing nothing even certificate became out of date. if i remove old certificate manually and restart consul template - all works fine including renew timer.

Consul Template version

consul-template v0.29.1 (4525703)

Configuration

vault {
  address = "address"
  vault_agent_token_file = "token"
  retry {
    enabled = true
    attempts = 5
    backoff = "250ms"
  }
}

template {
  source      = "/etc/consul_template.d/CN.tpl"
  destination = "/etc/ssl/private/CN.pem"
  perms       = "0400"
  command     = "systemctl reload nginx"
}

{{ with pkiCert "pki/issue/CN" "common_name=CN" "ttl=720h" }}
{{ .Cert }}
{{ .CA }}
{{ if .Data.Key }}
{{ .Key  | writeToFile "/etc/ssl/private/CN.key" "root" "root" "0400" }}
{{ end }}
{{ end }}```

```liquid
# Include sample data you reference in the template from Consul or Vault here.

Command

/usr/local/bin/consul-template -log-file='/var/log/consul-template/' -log-level info -config='/etc/consul_template.d/consul_template.hcl'```

### Debug output

 systemd[1]: Started consul-template.
[DEBUG] (logging) enabling log_file logging to /var/log/consul-template/consul-template.log with rotation every 24h0m0s
184+0300 [INFO] consul-template v0.29.1 (4525703)
184+0300 [INFO] (runner) creating new runner (dry: false, once: false)
185+0300 [DEBUG] (runner) final config: <cutted>
185+0300 [INFO] (runner) creating watcher
185+0300 [DEBUG] (watcher) adding vault-agent.token
185+0300 [TRACE] (watcher) vault-agent.token starting
185+0300 [TRACE] (view) vault-agent.token starting fetch
186+0300 [TRACE] vault-agent.token: READ /etc/vault.d/token
186+0300 [TRACE] vault-agent.token: reported change
186+0300 [TRACE] (view) vault-agent.token marking successful data response
186+0300 [TRACE] (view) vault-agent.token successful contact, resetting retries
186+0300 [INFO] (runner) starting
186+0300 [DEBUG] (runner) running initial templates
186+0300 [DEBUG] (runner) initiating run
186+0300 [DEBUG] (runner) checking template f6dcfdb54a5b3d63585b8ad6d5d190fc
187+0300 [DEBUG] (runner) missing data for 1 dependencies
187+0300 [DEBUG] (runner) missing dependency: vault.pki(pki/issue/CN)
187+0300 [DEBUG] (runner) add used dependency vault.pki(pki/issue/CN) to missing since isLeader but do not have a watcher
187+0300 [DEBUG] (runner) was not watching 1 dependencies
187+0300 [DEBUG] (watcher) adding vault.pki(pki/issue/CN)
187+0300 [TRACE] (watcher) vault.pki(pki/issue/CN) starting
187+0300 [DEBUG] (runner) diffing and updating dependencies
187+0300 [DEBUG] (runner) watching 2 dependencies
187+0300 [TRACE] (view) vault.pki(pki/issue/CN) starting fetch
188+0300 [TRACE] (view) vault.pki(pki/issue/CN) marking successful data response
188+0300 [TRACE] (view) vault.pki(pki/issue/CN) successful contact, resetting retries
294+0300 [TRACE] (view) vault.pki(pki/issue/CN) received data
294+0300 [TRACE] (view) vault.pki(pki/issue/CN) starting fetch
294+0300 [DEBUG] (runner) receiving dependency vault.pki(pki/issue/CN)
294+0300 [DEBUG] (runner) initiating run
294+0300 [DEBUG] (runner) checking template f6dcfdb54a5b3d63585b8ad6d5d190fc
295+0300 [DEBUG] (runner) rendering "/etc/consul_template.d/CN.tpl" => "/etc/ssl/private/CN.pem"
295+0300 [DEBUG] (runner) diffing and updating dependencies
295+0300 [DEBUG] (runner) vault.pki(pki/issue/CN) is still needed
295+0300 [DEBUG] (runner) watching 2 dependencies
295+0300 [DEBUG] (runner) all templates rendered
303+0300 [TRACE] (view) vault-agent.token received data
303+0300 [TRACE] (view) vault-agent.token starting fetch
303+0300 [TRACE] vault-agent.token: READ /etc/vault.d/token
303+0300 [DEBUG] (runner) initiating run
303+0300 [DEBUG] (runner) checking template f6dcfdb54a5b3d63585b8ad6d5d190fc
304+0300 [DEBUG] (runner) rendering "/etc/consul_template.d/CN.tpl" => "/etc/ssl/private/CN.pem"
304+0300 [DEBUG] (runner) diffing and updating dependencies
304+0300 [DEBUG] (runner) vault.pki(pki/issue/CN) is still needed
304+0300 [DEBUG] (runner) watching 2 dependencies
304+0300 [DEBUG] (runner) all templates rendered

### Expected behavior

replace old certificate by another CA
### Actual behavior

What actually happened?

consul-template doing nothing
### Steps to reproduce

1. got another certificate
2. create template
3. wait till certificate end and restart consul-template

[eikenb]

Hey @Inotsacred, thanks for taking the time to file the issue.

I'm not sure this is the problem here but looking at your template I see an issue with the PrivateKey. I'm going to describe it in terms of a fresh PKI fetched from Vault to simplify things. Starting from scratch using that template consul-template would fetch the PKI cert and write the Certificate and the CA to the template destination ("/etc/ssl/private/CN.pem") with the PrivateKey writing to "/etc/ssl/private/CN.key". If consul-template was then restarted it would use the template destination path to read in the Certificate, CA and PrivateKey... but it wouldn't find the PrivateKey as it isn't in that file.

The PrivateKey either needs to be included in that file or with a dummy template destination file where you both include everything and use the writeToFile function to write each part to the different files.

This might be the cause of the reported problem but I'm not sure at this point. But you'll want to address that issue either way.

Please let me know if that does address the problem or not. If not we can take things from there.

Thanks.

[mister2d]

The PrivateKey either needs to be included in that file or with a dummy template destination file where you both include everything and use the writeToFile function to write each part to the different files.

Need an example. I too am running into this issue.

[eikenb]

Taking the example given in the ticket, your template could look something like...

{{ with pkiCert "pki/issue/CN" "common_name=CN" "ttl=720h" }}
{{ .Cert }}
{{ .Key }}
{{ .CA }}
{{ .Key  | writeToFile "/etc/ssl/private/CN.key" "root" "root" "0400" }}
{{ end }}
{{ end }}```

This writes the Cert, CA and Key to the file that the template renders to (the destination) but also writes the Key to the separate file. That way when you reload/restart consul-template it will find all the data needed in the file to refresh the data cache and the TTL monitor. Once in the cache they can be used just as if they had been fetched from Vault originally.

Make sense?

[Inotsacred]

I found workaround with precheck certificate issuer and remove it if its not vault PKI.

certificates renew and etc works fine with my configuration, only migration to vault pki looks strange. sometimes it replace certificate in time, sometimes not.

{{ if .Data.Key }}
{{ .Key  | writeToFile "/etc/ssl/private/CN.key" "root" "root" "0400" }}

I used this condition because when consul template check certificate after service restart it could delete private key file. because vault send back private key only once.

[eikenb]

Hey @Inotsacred, is this resolved to your satisfaction or would you like more help with it? Looking to close this issue if you're good. Thanks.

[Inotsacred]

Hey @Inotsacred, is this resolved to your satisfaction or would you like more help with it? Looking to close this issue if you're good. Thanks.

Hey @eikenb! I've tried your suggestion but this not fix my problem and remove previous legacy certificate before deploy consul-template was only working solution. But i found another problem. It looks very similar. Sometimes consul-template tries to render certificate in threshold interval, but nothing happens. About 1% of TTL it renew it successfully. Im watching on this behavier a while.

template looks like:

 {{ with pkiCert "pki-common/issue/common-name" "common_name=common-name"  "ttl=100m" }}
 {{- .Cert }}
 {{- .Key }}
 {{- .CA -}}
 {{- .Cert | writeToFile "/etc/ssl/private/common-name.pem" "root" "root" "0400" }}
 {{- .CA | writeToFile "/etc/ssl/private/common-name.pem" "root" "root" "0400" "append" -}}
 {{- .Key | writeToFile "/etc/ssl/private/common-name.key" "root" "root" "0400" }}
 {{- end -}}

unsuccessful trace log (~10% TTL left):

 2022-12-17T18:23:18.674+0300 [TRACE] (view) vault.pki(pki/issue/common-name) marking successful data response
 2022-12-17T18:23:18.675+0300 [TRACE] (view) vault.pki(pki/issue/common-name) successful contact, resetting retries
 2022-12-17T18:23:18.685+0300 [TRACE] (view) vault.pki(pki/issue/common-name) received data
 2022-12-17T18:23:18.685+0300 [DEBUG] (runner) receiving dependency vault.pki(pki/issue/common-name)
 2022-12-17T18:23:18.685+0300 [DEBUG] (runner) initiating run
 2022-12-17T18:23:18.703+0300 [DEBUG] (runner) checking template 26ac5b50417fef9d32463eeb37cdc6ee
 2022-12-17T18:23:18.703+0300 [TRACE] (view) vault.pki(pki/issue/common-name) starting fetch
 2022-12-17T18:23:18.728+0300 [DEBUG] (runner) rendering "/etc/consul_template.d/common-name.tpl" => "/etc/ssl/private/common-name.pem"
 2022-12-17T18:23:18.728+0300 [DEBUG] (runner) diffing and updating dependencies
 2022-12-17T18:23:18.728+0300 [DEBUG] (runner) vault.pki(pki/issue/common-name) is still needed
 2022-12-17T18:23:18.728+0300 [DEBUG] (runner) watching 2 dependencies
 2022-12-17T18:23:18.728+0300 [DEBUG] (runner) all templates rendered

successful trace log after 10 mins (1% TTL left):

 2022-12-17T18:33:55.791+0300 [TRACE] (view) vault.pki(pki/issue/common-name) marking successful data response
 2022-12-17T18:33:55.791+0300 [TRACE] (view) vault.pki(pki/issue/common-name) successful contact, resetting retries
 2022-12-17T18:33:55.791+0300 [TRACE] (view) vault.pki(pki/issue/common-name) received data
 2022-12-17T18:33:55.791+0300 [TRACE] (view) vault.pki(pki/issue/common-name) starting fetch
 2022-12-17T18:33:55.791+0300 [DEBUG] (runner) receiving dependency vault.pki(pki/issue/common-name)
 2022-12-17T18:33:55.791+0300 [DEBUG] (runner) initiating run
 2022-12-17T18:33:55.791+0300 [DEBUG] (runner) checking template 26ac5b50417fef9d32463eeb37cdc6ee
 2022-12-17T18:33:55.792+0300 [DEBUG] (runner) rendering "/etc/consul_template.d/common-name.tpl" => "/etc/ssl/private/common-name.pem"
 2022-12-17T18:33:55.796+0300 [INFO] (runner) rendered "/etc/consul_template.d/common-name.tpl" => "/etc/ssl/private/common-name.pem"
 2022-12-17T18:33:55.796+0300 [DEBUG] (runner) appending command ["systemctl reload nginx"] from "/etc/consul_template.d/common-name.tpl" => "/etc/ssl/private/common-name.pem"
 2022-12-17T18:33:55.796+0300 [DEBUG] (runner) diffing and updating dependencies
 2022-12-17T18:33:55.796+0300 [DEBUG] (runner) vault.pki(pki/issue/common-name) is still needed
 2022-12-17T18:33:55.796+0300 [INFO] (runner) executing command "[\"systemctl reload nginx\"]" from "/etc/consul_template.d/common-name.tpl" => "/etc/ssl/private/common-name.pem"
 2022-12-17T18:33:55.796+0300 [INFO] (child) spawning: sh -c systemctl reload nginx
 2022-12-17T18:33:55.834+0300 [DEBUG] (runner) watching 2 dependencies
 2022-12-17T18:33:55.834+0300 [DEBUG] (runner) all templates rendered

graph certificates renew, 100 mins TTL: sometimes it works, sometimes not.

any suggestions?