Support loading placeholder content from files (for systemd credentials or docker secrets)

[Sohalt]

It would be nice to support loading secrets (e.g. api tokens for DNS auth) from files (as opposed to env vars).

E.g. using a syntax like {file./foo/bar/baz.txt}, which would be replaced with the contents of the file at /foo/bar/baz.txt.

That would support using systemd credentials or docker secrets to provide those secrets.

[francislavoie]

Interesting idea.

One concern I have is that this placeholder would not be "instantaneous" since it would need to load from a file each time the value is invoked, so it could have negative performance consequences from constantly opening and closing files to read from them at runtime.

To clarify, placeholders are replaced at runtime whenever the config value is read, not just once when the config is loaded. The Caddyfile has support for replacing environment variables once at config adapt time, but that's a special case just for the Caddyfile.

What we could do is cache the value read from these files in memory for the duration the config is active. So the first time the placeholder is replaced, it would load the file, and any subsequent time it would hit the in-memory cache. A config reload would reset the cache. We'd obviously have to clearly document that the values from these file placeholders are not read live.

[Sohalt]

Good to know that env placeholders are only replaced at config adapt time. That was not immediately obvious to me from https://caddyserver.com/docs/conventions#placeholders.

https://github.com/caddy-dns/cloudflare#config-examples shows usage of {env.CF_API_TOKEN} in the Caddy JSON (i.e. not Caddyfile). Would that get replaced? Would it get replaced, if I change the confinguration at runtime with the /config api endpoint and do something like

curl -X POST -H "Content-Type: application/json" localhost:2019/config/apps/tls/automation/policies/0/issuers -d '{
	"module": "acme",
	"challenges": {
		"dns": {
			"provider": {
				"name": "cloudflare",
				"api_token": "{env.CF_API_TOKEN}"
			}
		}
	}
}'

[Sohalt]

Ok, looks like it gets replaced in both cases, using config.json and changing the config with the http api.

I think a similar behavior would make sense for the {file.*} placeholder, which is what you already suggested in your response, if I'm not mistaken.

[emilylange]

There is {env.ENV_VAR} and there is {$ENV_VAR}. The latter is specific to the Caddyfile, will get replaced once at adapt time, and is what @francislavoie was referring to above.

See https://caddyserver.com/docs/caddyfile/concepts#environment-variables

I am sure you are aware of this already, but in case some future reader stumbles upon this, there is also EnvironmentFile= in systemd to read environment vars from a file. Though that's obviously not the requested feature here. Just something I wanted to mention in case someone needs something similar that works right now.

---

Edit: I slipped on crtl+enter again and sent this comment too early :/

[francislavoie]

Actually, another detail I forgot last night -- you can use caddy run with the --envfile option which can pull your secrets from the file and load them for use with the {env.*} placeholders. That should probably be sufficient for most users. Just means you need to format your secrets file such that it's a valid env file, but that should be viable, no?

[Sohalt]

The same applies to the EnvironmentFile option of systemd. It's viable (and what I'm using now) but a bit cumbersome. And does not necessarily compose well with other services using the same secret. But I can understand, if you don't want to complicate the config further.

[mholt]

What's cumbersome about using an env file?

[Sohalt]

I have to remember to put the variable name in the env file and match it up with the variable name used in the config. If another service uses the same secret, I have to load the secret there through the same variable (or create a separate version of the secret without the variable, but then I have to update 2 secrets when I want to rotate)

[mholt]

But wouldn't that be the same workflow if you had the secret in another file? You need to match up the filename with the file in the config.

I'm not really sure/convinced that this feature is what you're looking for, or that the extra complexity is worthwhile.

[Sohalt]

If I had the secret in a file I could use it with an arbitrary number of services supporting systemd-credentials or docker secrets. But as I've said, I can understand if you deem the feature unnecessary and too complex for what is gained.

[polarathene]

FWIW: Same feature request was raised in the Caddy CloudFlare DNS plugin repo a while back.

• One scenario where --envfile does not work, is with caddy-docker-proxy, but as noted in the linked issue, that isn't the responsibility of Caddy to fix (technically am equivalent global setting in a Caddyfile could resolve that, but may not be pragmatic).
• Users wanting to use Docker Secrets may expect the feature to work as it is documented for Docker Swarm, but non-swarm deployment does not function in the same way. The secret support is just for local dev compatibility as noted by a Docker compose maintainer, it's just a bind mount of the file behind the scenes, none of the other secrets features apply (including secure storage and ownership/permission controls). Thus if someone can read the ENV of a container, there's a good chance they can read the file contents of the secret too.
• Main benefit then becomes avoiding risk of secrets leaking via ENV from software that captures the ENV to send / display (eg: in client browser) for debug / troubleshooting (IIRC Django was cited as doing this), often this happens without awareness of the software performing it (third-party packages / modules could likewise be compromised, or performing some sort of telemetry or bug report functionality, where ENV is captured).
• --envfile should avoid that concern by restricting the scope to Caddy (and any third-party modules, but I assume they can read any secrets in config regardless).

---

One concern I have is that this placeholder would not be "instantaneous" since it would need to load from a file each time the value is invoked, so it could have negative performance consequences from constantly opening and closing files to read from them at runtime.

What we could do is cache the value read from these files in memory for the duration the config is active.

AFAIK (at least with Linux) that shouldn't be necessary. Files would be read from disk on initial access, but should generally exist in disk cache in memory.

So long as there is spare memory it's often used for that until a process needs the extra memory to allocate or cache is flushed (can be invoked explicitly). Once the file is read from disk again, it should likewise reside in cache.

I suppose if you have Caddy allocate memory to cache it you at least guarantee under memory constrained scenarios that it's always cached in memory, but otherwise may be redundant.

[mholt]

Ok; I'm not sure I know enough about Docker to make the decision on this one. It'd probably be best if I leave it up to other maintainers/contributors who are familiar/experienced with Docker on this one. :blush:

[polarathene]

Ok; I'm not sure I know enough about Docker to make the decision on this one.

It's not exactly Docker specific.

I don't have links to articles atm that I could direct you to, but you will find secrets management advice discouraging the practice of configuring / passing secrets to software via ENV when using alternative sources like files is viable.

The issue regarding Docker is there is some misinformation about a feature called Docker Secrets, originally only available for a distributed deployment mode known as "Docker Swarm". It provided encrypted storage of secrets at rest and in transit to swarm nodes over the network, the secrets would then be decrypted into an in-memory tmpfs location of the container under /run available as a file. You could also explicitly configure the ownership/permissions of the file so that it was not world readable for example.

A "feature" added support for using the configuration syntax in a local non-swarm development environment. But this didn't use any of the features described above, it only supported sourcing a physical file on disk (not from the encrypted vault at runtime) and making that available as a file in the container (without respecting any configuration to change ownership / permissions). The benefit was less secure / useful, the user needed to be aware of this (often they're not as there is no disclaimer and some articles / community parroting misinformation).

It can be a beneficial practice when done properly.

---

Perhaps it's something a Caddy plugin could enable?

As it's not specific to Docker, perhaps it's better to just leave the issue open for a while to see if any other users of Caddy chime in about the secrets sourced from files as a security benefit over ENV?

I think the main concern is with the risk of ENV leaking accidentally to third-parties, but --envfile at least reduces the scope of that.

[francislavoie]

Perhaps it's something a Caddy plugin could enable?

It could, but only in certain contexts like HTTP routes. It wouldn't work in other contexts like TLS issuance.

We don't have any way for a plugin to hook into global replacer. In fact there is no global replacer really, there's a globalDefaultReplacements function which handles placeholders which require no config to work (e.g. system and time placeholders), and this is registered by default to every replacer instanciated.

One scenario where --envfile does not work, is with caddy-docker-proxy

It should be trivial to add that as a new CLI option to CDP. Out of scope of this issue though.

[francislavoie]

I wrote an implementation in #5463. No caching for now. We'll probably mark it in the docs as experimental, and note that it might have performance implications if used in hot-paths in the config. But this should be good enough for secrets like DNS plugin API keys since those are only read on use which is pretty infrequent.

[mholt]

@Sohalt @polarathene If you still need this, can you please try #5463 so we can verify it has an actual production use case before we merge it? We want to make sure we got it right. Thanks!

[polarathene]

We want to make sure we got it right. Thanks!

I might be able to give it a try today or tomorrow, but I'm not sure if I can test it better beyond providing the secret as a file and confirming the content was read correctly.

---

Question (Resolved)

Can you provide a small Caddyfile example, or at least the syntax expected?

I can only glean file.* from the PR, but not sure what it expects as a file path value, is it an absolute / relative path, or an ENV with a filepath? (EDIT: Nevermind, looks like absolute path)

In containers I've seen that offer the feature, it's often providing a filepath to the same secret ENV, or a variant with a _FILE suffix (eg: mariadb and phpmyadmin](https://github.com/phpmyadmin/docker#variables-that-can-be-read-from-a-file-using-_file) both support _FILE variants, while a popular alternative mariadb image uses FILE__ prefix).

I'm guessing a filepath like shown at the start of this issue? {file./foo/bar/baz.txt} instead of {file.SECRET_BAZ}? I am not aware of any need to require an ENV for sourcing a path so that's fine :+1:

[francislavoie]

It's absolute or relative, and it's relative to Caddy's working dir. What that is depends on how you run Caddy. If you just run caddy run then it'll be wherever you ran that command, if systemd then it's the caddy user's HOME (so /var/lib/caddy), etc.

[hollowshiroyuki]

I'm currently trying to implement an IP blocklist and this feature could be really useful with this use case as I couldn't find a v2 plugin for this. There's a post on the forums suggesting the use of remote_ip https://caddy.community/t/ipfilter-plugin-in-caddy-v-2/8673 but it's unrealistic to put thousands of addresses in the Caddyfile. So I guess it could be something like this ?

@blocked_ip remote_ip {file./ip_list.txt}
abort @blocked_ip

Will it work if the file has one IP/CIDR per line like : https://github.com/jhassine/server-ip-addresses/blob/master/data/datacenters.txt ?

[francislavoie]

@hollowshiroyuki The {file.*} placeholder cannot expand to multiple tokens. The value read the from the file can only ever be a single token. So no, using the file placeholder is not an option here.

What you can do is format the file such that it is a valid Caddyfile config (space separated tokens, or \ to escape each newline), then use import to bring it in.

Or, write your own request matcher plugin which can read from a file.

Either way, it's off-topic from this issue.

[lorenzleutgeb]

I'd appreciate a status update.

Just to get an idea of how ugly a workaround is (using Nix with sops-nix which is based on previously Mozilla, now CNCF tool SOPS):

{ config, pkgs, ... }: {
  sops.secrets."cloudflare/api/token".sopsFile = ./sops/cloudflare.yaml;

  services.caddy.globalConfig = "acme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}";

  systemd.services.caddy.serviceConfig = {
    LoadCredential = "CLOUDFLARE_API_TOKEN:${config.sops.secrets."cloudflare/api/token".path}";

    # Workaround to leak secret into the environment.
    EnvironmentFile = "-%t/caddy/secrets.env";
    RuntimeDirectory = "caddy";
    ExecStartPre = [
      ((pkgs.writeShellApplication {
          name = "caddy-secrets";
          text = "echo \"CLOUDFLARE_API_TOKEN=\\\"$(<\"$CREDENTIALS_DIRECTORY/CLOUDFLARE_API_TOKEN\")\\\"\" > \"$RUNTIME_DIRECTORY/secrets.env\"";
        })
        + "/bin/caddy-secrets")
    ];
  };
}

This might help people using Nix or just plain systemd with credentials.

With reading files directly this would turn into the much nicer:

{ config, pkgs, ... }: {
  sops.secrets."cloudflare/api/token".sopsFile = ./sops/cloudflare.yaml;

  services.caddy.globalConfig = "acme_dns cloudflare {file.${config.sops.secrets."cloudflare/api/token".path}}";
}

Also, I am not sure how one would use {file.*} with systemd credentials. From my understanding of systemd credentials, users are meant to read secrets from $CREDENTIALS_DIRECTORY, so one would really have to use {file.{env.CREDENTIALS_DIRECTORY}/ID}. Please correct me if that's not the case.

[francislavoie]

The latest was https://github.com/caddyserver/caddy/pull/5463#issuecomment-1548615997

I just pushed another commit to try to address those concerns. I'll talk it through with Matt soon and we'll see if we feel comfortable enough with the solution.

[Toomoch]

I'd appreciate a status update.

Just to get an idea of how ugly a workaround is (using Nix with sops-nix which is based on previously Mozilla, now CNCF tool SOPS):

{ config, pkgs, ... }: {
  sops.secrets."cloudflare/api/token".sopsFile = ./sops/cloudflare.yaml;

  services.caddy.globalConfig = "acme_dns cloudflare {env.CLOUDFLARE_API_TOKEN}";

  systemd.services.caddy.serviceConfig = {
    LoadCredential = "CLOUDFLARE_API_TOKEN:${config.sops.secrets."cloudflare/api/token".path}";

    # Workaround to leak secret into the environment.
    EnvironmentFile = "-%t/caddy/secrets.env";
    RuntimeDirectory = "caddy";
    ExecStartPre = [
      ((pkgs.writeShellApplication {
          name = "caddy-secrets";
          text = "echo \"CLOUDFLARE_API_TOKEN=\\\"$(<\"$CREDENTIALS_DIRECTORY/CLOUDFLARE_API_TOKEN\")\\\"\" > \"$RUNTIME_DIRECTORY/secrets.env\"";
        })
        + "/bin/caddy-secrets")
    ];
  };
}

This might help people using Nix or just plain systemd with credentials.

With reading files directly this would turn into the much nicer:

{ config, pkgs, ... }: {
  sops.secrets."cloudflare/api/token".sopsFile = ./sops/cloudflare.yaml;

  services.caddy.globalConfig = "acme_dns cloudflare {file.${config.sops.secrets."cloudflare/api/token".path}}";
}

Also, I am not sure how one would use {file.*} with systemd credentials. From my understanding of systemd credentials, users are meant to read secrets from $CREDENTIALS_DIRECTORY, so one would really have to use {file.{env.CREDENTIALS_DIRECTORY}/ID}. Please correct me if that's not the case.

Great workaround, but I think I found a much easier solution using .env files in sops (you can then reference the secret with {env.secret}):

      sops.secrets."duckdns/token".sopsFile = "${inputs.private}/secrets/duckdns.env";
      sops.secrets."duckdns/token".format = "dotenv";

      systemd.services.caddy.serviceConfig = {
            EnvironmentFile = "${config.sops.secrets."duckdns/token".path}";
      };

[lorenzleutgeb]

@Toomoch Fair point. I guess it boils down to how you want to organize your secrets. I am still hoping for systemd credentials to be supported at some point, and it uses one file per credential (=secret). For example, this allows usage of systemd-creds in the unit (now that I think of it, I probably should add this in the snippet I posted above...). Note that you can avoid that string interpolation, and directly go EnvironmentFile = config.sops.secrets."duckdns/token".path;.

[mholt]

I'm blocking this right now -- totally my bad -- I still have about 3 pages of notification backlog to get through after we had a baby, and this one has security implications I want to make sure we address properly. Sorry about that, it just might take some time.

[kanashimia]

For those who want to use credentials:

Also, I am not sure how one would use {file.*} with systemd credentials. From my understanding of systemd credentials, users are meant to read secrets from $CREDENTIALS_DIRECTORY, so one would really have to use {file.{env.CREDENTIALS_DIRECTORY}/ID}. Please correct me if that's not the case.

I don't think #5463 implemented nested substitution, so you can't really do {file.{env.CREDENTIALS_DIRECTORY}/ID} or {file.{$CREDENTIALS_DIRECTORY}/ID} But you can do it like this: {file./run/credentials/caddy.service/ID}

[francislavoie]

Actually, {file.{$CREDENTIALS_DIRECTORY}/ID} might work because {$ENV} form is pre-processed at config adapt time. Looks really strange though. Probably better to just explicitly use the path instead, would be my recommendation.

[kanashimia]

Yes you are right, tested and it works like that.

[ColCh]

This works really well, trying this for loading dns token in my homelab setup. Thank you very much

I found this issue via google, documentation not mentions this syntax.

Just in case, any plans for updating docs? @mholt

[mohammed90]

Just in case, any plans for updating docs?

This feature is part of v2.8.0, which is only released as beta right now. The docs will be updated when v2.8.0 is released.