Continual re-rendering while template output is unchanged

[allanlang]

Consul Template version

0.19.3

Configuration

No configuration file is used.

Template file is defined as:

{{ range service "dashboard" }}
upstream dash{{ range .Tags}}{{.}}{{end}} {
  least_conn;
  server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=0 weight=1;
}{{end}}

server {

  listen 80;

  auth_basic "dashboard";
  proxy_read_timeout 300s;

{{ range service "dashboard" }}
  location /{{ range .Tags}}{{.}}{{end}}/ {
    auth_basic "{{ range .Tags}}{{.}}{{end}}";
    auth_basic_user_file "/data/config/generated/{{ range .Tags}}{{.}}{{end}}-pwd.txt";
    proxy_pass http://dash{{ range .Tags}}{{.}}{{end}}/;
  }
{{end}}

}

Command

/usr/local/bin/consul-template -template "/data/config/nginx-ct.tmpl:/tmp/dashboard.conf:nginx -s reload"

Debug output

https://gist.github.com/allanlang/c1857a111104f4a6ed53ba281898be2c

Expected behavior

Given no changes in the underlying state of "dashboard" services in Consul, the templated file /tmp/dashboard.conf should not be re-rendered, and the nginx service should not be signalled to reload.

Actual behavior

/tmp/dashboard.conf is continually re-rendered with exactly the same content (md5sum matches across multiple renderings). nginx is triggered to reload continually (/var/log/nginx/error.log shows multiple instances of signal process started).

Steps to reproduce

1. Multiple instances of a given service "dashboard" are registered in Consul by Nomad
2. Each service has a single tag which acts as a unique identifier for the instance, and which we want to use to generate a unique URL for that instance under a common nginx server, whose config is templated by consul-template using the template provided.
3. consul-template is scheduled to run under systemd, but the error can be reproduced via the command line in debug mode using:

/usr/local/bin/consul-template -template "/data/config/nginx-ct.tmpl:/tmp/dashboard.conf:nginx -s reload" -log-level="debug"

References

[danieleva]

I have exactly the same issue. My template is : {{ with printf "public_key=%s" (file "/etc/ssh/ssh_host_rsa_key.pub") | secret "ssh-hosts/sign/host" "cert_type=host" "ttl=76h" }}{{ .Data.signed_key }}{{end}} the config file has

template {
  error_on_missing_key = true
  perms = 0600
  source      = "/etc/ssh/ssh_host_rsa_key-cert.pub.tpl"
  destination = "/etc/ssh/ssh_host_rsa_key-cert.pub"
  command = "/usr/bin/systemctl restart sshd"
}

and sshd keeps getting restarted every ~90 seconds

[danieleva]

Found the issue for my use case, the secret returned by the ssh backend has lease=0, so consul-template is correctly renewing it at every run.

"lease_id": "",
"lease_duration": 0,
"renewable": false,

[eikenb]

Thanks for reporting the issue @allanlang.

As reproducing this requires a more complex setup it is going to take some time to get to this issue. I know it has been a while, but if you are still experiencing this and are able to provide a minimal setup to reproduce it, it would be a big help. Thanks.

[dheide951]

Soooo what was the solution to this problem? We are having the same issue. Consul-Template is reloading NGINX way more than it should be.

[eikenb]

Hey @dheide951,

I was not able to reproduce the original issue so haven't made any progress. If you can reproduce it and can provide the relevant details to allow me to do so that would be a big help. You can add your details here or file a new issue (if so, please mention this one).

Thanks.