Allow 'file' retries if file doesn't exist yet

[jsmilani]

The 'file' function will cause consul-template to immediately exit if the file in question doesn't exist, but it would be super useful if it supported retries like every other function. We have 2 containers, both run consul-template but the first reads secrets from Vault and renders a JSON file and the second reads the JSON file via a common volume. We don't want the second container to also read from Vault because it is a generic sidecar and the assumption the JSON file could come from anywhere and it is up to the first container to provide it. The problem is the second container isn't guaranteed to start after the first so there is no guarantee the JSON file exists yet. If there were some retries then the second container could just wait a few seconds for it to become available then render the template.

Consul Template version

consul-template v0.24.1 (c54d0abc)

Configuration

/tmp/config.hcl

log_level = "debug"
template {
  source = "/tmp/oidc.conf.ctmpl"
  destination = "/tmp/oidc.conf"
}

/tmp/oidc.conf.ctmpl

{{ $oidc := file "/tmp/oidc-missing.json" | parseJSON -}}
provider = "oidc"
client_id = "{{ $oidc.id }}"
client_secret = "{{ $oidc.secret }}"

/tmp/oidc.json

{
  "id" : "foo",
  "secret" : "bar"
}

Command

consul-template -config /tmp/config.hcl -once

Expected behavior

What we are hoping consul-template retries using standard exponential backoff until you run cp /tmp/oid.json /tmp/oidc-missing.json so it can finish rendering.

Actual behavior

Consul-template exits immediately with:

2020/04/14 23:00:55.845820 [INFO] consul-template v0.24.1 (c54d0abc)
2020/04/14 23:00:55.845858 [INFO] (runner) creating new runner (dry: false, once: true)
2020/04/14 23:00:55.846330 [INFO] (runner) creating watcher
2020/04/14 23:00:55.846478 [INFO] (runner) starting
2020/04/14 23:00:55.847644 [ERR] (view) file(/tmp/oidc-missing.json): stat /tmp/oidc-missing.json: no such file or directory (exceeded maximum retries)
2020/04/14 23:00:55.847654 [ERR] (runner) watcher reported error: file(/tmp/oidc-missing.json): stat /tmp/oidc-missing.json: no such file or directory
2020/04/14 23:00:55.851655 [ERR] (cli) file(/tmp/oidc-missing.json): stat /tmp/oidc-missing.json: no such file or directory

Steps to reproduce

1. Create the files mentioned in the Configuration section above
2. run consul-template -config /tmp/config.hcl -once

[syberkitten]

I'm having the same issue, this is actually a real blocker.

Can we maybe workaround this by putting this code inside a separate template which is then invoked such if a file does not exist this template fails silently?

(as to allow the process to complete without errors)

[eikenb]

Hey @syberkitten,

I have no good/clean ideas for working around this. Maybe wrap the consul-template in a script that checks for the file first or you could probably do something weird with a blank template configured to run a script on render and have that touch that file to create a blank one if it doesn't exist.

To help promote this fix you can :+1: the issue to indicate it's importance. We use the :+1:'s to help determine what is important to the community and prioritize work.

Thanks.

[kornface13]

I also have this issue. Seems reasonable to just not render the template if the file doesn't exist maybe??