Provide an option to get all sensitive information from file (ex.: /run/secrets/my_password)

[endersonmaia]

Do you want to request a feature or report a bug?

Feature

What did you expect to see?

treafik should support defining a file as a source of sensitive informations like API Tokens, passwords, ssh keys and so on, like is already possible for SSL Certificates and such.

For example, Docker Swarm secrets store the secrets on /run/secrets/<secret_name>, and could be used to store CLOUDFLARE_API_KEY DNS provider configuration for ACME (Let's Encrypt) Configuration

Read: https://diogomonica.com/2017/03/27/why-you-shouldnt-use-env-variables-for-secret-data/

Proposal

ENV variables

The suffix _FILE could be added to every ENV variable that already expects a secret maybe is a good starting point, and it's being used in official docker images like this.

Ex.:

export [email protected]
export CLOUDFLARE_API_KEY_FILE="/run/secrets/cloudflare_api_key"

treafik TOML and CLI parameters

To keep consistency, in the same way a suffix is suggested for ENV variables, a suffix _file and File could be used for TOML and CLI parameters respectively.

• TOML

[etcd]
endpoint = "127.0.0.1:2379"
watch = true
prefix = "/traefik"
username = "myuser"
password_file = "/run/secret/etcd_password"

• CLI

traefik ... --consul.httpTokenFile see #3602

Considerations

The parameters some_secret an some_secret_file are exclusives, and should give an error if both are defined.

[endersonmaia]

Looking at the code, and trying to map where the implementation should go, I saw that, for ex., the ACME (Let's Encrypt) Configuration would need an upstream implementation acme/lego#535

[endersonmaia]

Grafana's solution is a great reference, it's using a ENTRYPOINT script.

See: https://github.com/grafana/grafana-docker/blob/4dda68b6ff4dad1f19cfa93b29b9929fb138b97f/run.sh#L49-L61

[endersonmaia]

This was merged to master : https://github.com/xenolf/lego/pull/535

Just wait for the release, and the ENV variables part of this proposal could be achieved by just updating the documentation informing about the _FILE suffix.

[ldez]

@endersonmaia it's already in Traefik since 1.7.3

[endersonmaia]

@ldez, I couldn't find any information about it here on this issue

is there any other issue where this proposal is being tracked ?

[endersonmaia]

couldn't find any information about _FILE suffix support on the documentation either

[dargmuesli]

I think this can be closed now.

[ldez]

I don't think so: only DNS provider configuration (env vars) can use secret.

The passwords inside the static configuration don't use the "secret" approach.

[dargmuesli]

Ah, sorry, I misunderstood the scope.

[pietervanw]

I just tried the _FILE env var extension(s) but it doesn't work for me! Which version of traefik should I have at least? I'm using 1.7.9 (on ARM/Raspberry Pi's) and if I read correctly above, it should be available since 1.7.3.

I'm using the DNS challenge, Route 53 and have an access key which works if I use the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY env vars (only works if you don't add quotes around the env vars though!). This exposes the creds in my compose file though. If I create docker secrets and replace the env vars with AWS_ACCESS_KEY_ID_FILE and AWS_SECRET_ACCESS_KEY_FILE then ACME with LE fails (no valid credentials provider in chain).

This issue might not be the best place to get support to get it working, but I thought I'd ask about the correct version first.

(BTW what is the best / accepted way to get support for this kind of issues? I see a lot of questions on Stack Overflow with 0 responses).

Thanks! --Pieter

[dargmuesli]

How did you specify the created secrets within your compose file? I did it like this:

1. production.yml#L1
2. production.yml#L46
3. production.yml#L41

(notice that those files are merged with the development/stack.yml within my Dargstack setup).

[pietervanw]

Hi Jonas,

When it comes to my secrets definition, my stack looks identical. Externally defined these:

secrets:
  acme_aws_aki:
    external: true
  acme_aws_sak:
    external: true

And then in the Traefik service definition:

    secrets:
      - acme_aws_aki
      - acme_aws_sak
    environment:
# The _FILE reference with Docker Secrets doesn't work in Traefik 1.7.9! 
# To be updated / tried again after upgrading to Traefik 2.0/stable. 
# Keep the Git repo private, these are credentials in config. not nice. 
#      - AWS_ACCESS_KEY_ID_FILE=/run/secrets/acme_aws_aki
#      - AWS_SECRET_ACCESS_KEY_FILE=/run/secrets/acme_aws_sak
      - AWS_ACCESS_KEY_ID=AKI_MY_KEY_ID_QWERQWERQWER
      - AWS_SECRET_ACCESS_KEY=MY_SECRET_ACCESS_KEY_YQRETWERYTERTH
      - AWS_REGION=eu-central-1
      - AWS_HOSTED_ZONE_ID=MY_HOSTED_ZONE_ID

I don't have my config in Github so I can't link to it like you did.. If I comment out the _FILE env vars and remove the 'direct' ones, certificate requests fail. The only difference with your stack is that I don't use CloudFlare but AWS Route53. I've tried a few different ways of creating the Docker secrets:

printf secret | docker secret create my_secret -

and storing the credential (only the credential itself) in a file, then docker secret create my_secret <file>

If I open a shell in the running Traefik container the /run/secrets/... files are there, and they contain only the relevant secret - can't spot any issues or differences there.

Should I open a separate issue for this with more logging / config etc.?

[ldez]

With route53, we use the official aws client, and most of the env var are managed by the aws client, then it's not possible to use _FILE for all env vars.

You can use _FILE only with AWS_HOSTED_ZONE_ID, AWS_POLLING_INTERVAL, AWS_PROPAGATION_TIMEOUT, AWS_TTL, AWS_MAX_RETRIES.

Documentation from AWS client:

AWS Credentials are automatically detected in the following locations and prioritized in the following order:

1. Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, [AWS_SESSION_TOKEN]
2. Shared credentials file (defaults to ~/.aws/credentials)
3. Amazon EC2 IAM role

See also: https://github.com/aws/aws-sdk-go/wiki/configuring-sdk

https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/sessions.html

[pietervanw]

Hi Ludovic,

Thanks for the clarification! The generic statement that lego environment variables can be substituted (with a _FILE suffix) doesn't apply to Route 53 then. It might be a good idea to add that to the DNS provider documentation.

I'll try to figure out another way to keep my AWS credentials outside of my config files.

[pietervanw]

Hi @ldez,

I'm fine with all my comments being marked as off-topic, but one last question: in the lego documentation the same behavior (using _FILE env vars) is explicitly stated to be available for Route 53 as well: https://go-acme.github.io/lego/dns/route53/. What is the rationale behind Traefik using an AWS CLI client instead of the lego-provided DNS provider for Route 53? Am I missing something?

Thanks!

[endersonmaia]

Thanks for the clarification! The generic statement that lego environment variables can be substituted (with a _FILE suffix) doesn't apply to Route 53 then. It might be a good idea to add that to the DNS provider documentation.

@pietervanw you could send an issue or PR with this fix

[ldez]

@pietervanw https://github.com/go-acme/lego/pull/853

[pho3nixf1re]

I have dug around the docs, forums, and here in the issue trackers. Does traefik actually support hiding secrets at all? I see no recommendations on how to provide secret configuration values when using static file configs. Specifically I'm trying to setup Pilot monitoring but there doesn't seem to be a way to actually set the token without comiting it to my source code. What is the current recommendation by the Traefik team for providing secrets until this feature is provided here?

[djpbessems]

I've been trying to find a solution to somehow retrieve Pilot's token from from a secure store as well, instead of it ending up in my versioncontrol...

[ArwynFr]

I've been trying to find a solution to somehow retrieve Pilot's token from from a secure store as well, instead of it ending up in my versioncontrol...

As a workaround, I write the token to my static configuration file during the CD workflow. See working example using github secret

[caiuskong]

Still no solution?

[caiuskong]

I have a simple solution. Traefik static configuration does not support using file and environment variables at the same time, so consider updating it in Dockerfile.

traefik.yaml

providers:
  consul:
    endpoints:
      - "consul-server:8500"
    token: "${CONSUL_HTTP_TOKEN}"

docker-compose.yaml

version: "3"

services:
  traefik-proxy:
    image: my-traefik:v2.10.5
    container_name: traefik-proxy
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - $PWD/.traefik:/etc/traefik
    environment:
      CONSUL_HTTP_TOKEN: ${CONSUL_HTTP_TOKEN}

Dockerfile

FROM traefik:v2.10.5

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

entrypoint.sh

#!/bin/sh

sed -i "s/\${CONSUL_HTTP_TOKEN}/$CONSUL_HTTP_TOKEN/g" /etc/traefik/traefik.yml
traefik

[m-schwob]

@caiuskong nice solve! thank!

Didn't work for me out of the box. I got an error:

sed: can't move '/etc/traefik/traefik.yamlifDJPM' to '/etc/traefik/traefik.yaml': Resource busy

I solved that by mapping the configuration file to a template file:

- "./traefik/traefik.yaml:/etc/traefik/traefik.template.yaml:ro"

and creating a new file using the entrypoint.sh script

I also improved the entrypoint.sh script a little:

#!/bin/sh

# Define paths
TEMPLATE_FILE="/etc/traefik/traefik.template.yaml"
CONFIG_FILE="/etc/traefik/traefik.yaml"

# Replace environment variables in the template and save to actual config file
envsubst < "$TEMPLATE_FILE" > "$CONFIG_FILE"

# Start Traefik
exec /usr/local/bin/traefik "$@"

Since I used envsubst in my script, I had to install it in the Dokerfile:

FROM traefik:latest

# Install gettext for envsubst
RUN apk add --no-cache gettext

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh
ENTRYPOINT ["/entrypoint.sh"]

[ArwynFr]

Both approaches described above provide the secret as an environment variable which defeats the purpose of providing it securely in a filemount and was the initial problem that was reported by OP.

I've been trying to find a solution to somehow retrieve Pilot's token from from a secure store as well, instead of it ending up in my versioncontrol...

As a workaround, I write the token to my static configuration file during the CD workflow. See working example using github secret

Sorry the repo is not available anymore.

You have basically two possible approaches:

Runtime replacement

Make a custom image with an entrypoint that replaces some template/placeholder, like both examples above. But do it using the file contents directly, and avoid storing the secret value in an environment variable.

#!/usr/bin/env bash

sed` -i "s/\${CONSUL_HTTP_TOKEN}/`cat /run/secrets/CONSUL_HTTP_TOKEN`/g" /etc/traefik/traefik.yml

Traefik configuration as a secret

This approach considers the whole configuration as a secret. You can have some compile time procedure that replaces the tempate with the secret value you want. The produced traefik.yml can then be mounted in the container, or stored in a secret management store. The benefit is you can run an official image without custom entrypoint nor additional software in it.

The drawback is that most secrets management tools are limited in what size they can handle. Docker swarm secrets are limited to 500kB and kubernetes ones are limited to 1MB.

[nmengin]

Hello there,

Thank you for the feedback, we have set the status to enhancement.

With the other maintainers, we want to dig deeper on this topic to evaluate the solutions we can bring. We may not address every case together, because the use cases described target many parts of Traefik. We'll follow-up on this issue.