Manually reload tls certificates

[Nepoxx]

Do you want to request a feature or report a bug?

Feature

What did you expect to see?

My tls certificates are generated with Let's Encrypt remotely and are used by Traefik through a glusterfs mount. For that reason, Traefik is unable to properly monitor file changes and thus never knows when certificates are renewed (so it will serve an expired certificate). Having a way to tell Traefik to reload new certificates (or file configs in general) would allow the user to circumvent cases when Traefik is unable to use inotify.

Traefik Version

2.0

Similar to https://github.com/containous/traefik/issues/1623

[ldez]

Hello,

could you give more information about your Traefik configuration?

[RomRider]

Same issue for me, I have this configuration in place:

tls:
  certificates:
    - certFile: /etc/letsencrypt/live/DOMAIN/fullchain.pem
      keyFile: /etc/letsencrypt/live/DOMAIN/privkey.pem

  stores:
    default:
      defaultCertificate:
        certFile: /etc/letsencrypt/live/DOMAIN/fullchain.pem
        keyFile: /etc/letsencrypt/live/DOMAIN/privkey.pem

The path is a NFS volume and the certificates are renewed outside of traefik (because not only used by traefik)

[ldez]

could you give more information about your Traefik configuration (static and dynamic configuration)?

[RomRider]

Sure:

global:
  checkNewVersion: false
  sendAnonymousUsage: false
serversTransport:
  insecureSkipVerify: true

providers:
  docker:
    endpoint: tcp://docker-socket-proxy2:2375
    exposedByDefault: false
    watch: true

  file:
    directory: /configs/
    watch: true

log:
  level: DEBUG
accessLog: {}

api:
  dashboard: true

entryPoints:
  https:
    address: :443
    forwardedHeaders:
      trustedIPs:
        - "127.0.0.1/32"
        - "x.x.x.x"
    proxyProtocol:
      trustedIPs:
        - "127.0.0.1/32"
        - "x.x.x.x"

  publicHttps:
    address: :8443
    forwardedHeaders:
      trustedIPs:
        - "127.0.0.1/32"
        - "x.x.x.x"
    proxyProtocol:
      trustedIPs:
        - "127.0.0.1/32"
        - "x.x.x.x"

TLS:

tls:
  certificates:
    - certFile: /etc/letsencrypt/live/DOMAIN/fullchain.pem
      keyFile: /etc/letsencrypt/live/DOMAIN/privkey.pem

  stores:
    default:
      defaultCertificate:
        certFile: /etc/letsencrypt/live/DOMAIN/fullchain.pem
        keyFile: /etc/letsencrypt/live/DOMAIN/privkey.pem

  options:
    default:
      minVersion: VersionTLS12

Dynamic example:

    labels:
      - traefik.enable=true
      - traefik.docker.network=${TRAEFIK_BACKEND_NETWORK}
      - traefik.http.routers.grafana.rule=Host(`grafana.${PRIVATE_DOMAIN}`)
      - traefik.http.routers.grafana.entryPoints=https
      - traefik.http.routers.grafana.tls=true
      - traefik.http.routers.grafana.tls.options=default
      - traefik.http.services.grafana.loadBalancer.server.port=3000

[ldez]

How you mount the file?

[RomRider]

NFS on the host, then I map the folder in the container:

    volumes:
      - ./traefik.yml:/traefik.yml:ro
      - ./configs:/configs:ro
      - ${DOCKER_DATA_NFS_FOLDER}/certbot/etc-letsencrypt:/etc/letsencrypt:ro

[Nepoxx]

Stack.yml

version: "3.7"
services:
  traefik:
    image: traefik:v2.0
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
      - "8082:8082"
    volumes:
      - type: bind
        source: /var/run/docker.sock
        target: /var/run/docker.sock
      - type: bind
        source: /mnt/gfs/docker/traefik/traefik.yml
        target: /etc/traefik/traefik.yml
      - type: bind
        source: /mnt/gfs/docker/traefik/certs
        target: /certs
      - type: bind
        source: /mnt/gfs/docker/traefik/config
        target: /config
      - type: bind
        source: /mnt/gfs/docker/traefik/acme
        target: /acme
    deploy:
      placement:
        constraints:
          - node.role == manager
    networks:
      traefik-net: {}

networks:
  traefik-net:
    name: traefik-net
    driver: overlay

Where /mnt/gfs is a gluster mount:

localhost:/gfs /mnt/gfs glusterfs defaults,_netdev,backupvolfile-server=venus 0 0

traefik.yml

entryPoints:
  http:
    address: ":80"

  https:
    address: ":443"

  metrics:
    address: ":8082"

providers:
  docker:
    swarmMode: true
    network: traefik-net
    exposedByDefault: false
  file:
    filename: /config/config.yml
    # inotify does not work with network mounts
    watch: false

# API and dashboard configuration
api:
  insecure: true
  debug: true

metrics:
  prometheus:
    entryPoint: metrics

config.yml (mentioned above):

tls:
  certificates:
    - certFile: /certs/_.example.com/_.example.com.cert
      keyFile: /certs/_.example.com/_.example.com.key
http:
  middlewares:
    redirect-to-https:
      redirectScheme:
        scheme: https
  routers:
    http-catchall:
      priority: 0
      entryPoints:
        - http
      middlewares:
        - redirect-to-https@file
      rule: 'hostregexp(`{host:.+}`)'
      service: noop
  services:
    noop:
      loadBalancer:
        servers:
          - url: 'http://127.0.0.1'

Replacing _.example.com.cert and _.example.com.key has no effect on Traefik

[RomRider]

An interesting way to handle that would be to provide an API endpoint to reload certs.

[gtmadev]

I just ran into this today as well when using a glusterfs volume for file configs. In my main config, I have it watch a directory in the glusterfs volume. That directory is where I store dynamic configurations. Some nodes in docker swarm seem to get the updated file config, and others don't, until I manually restart that instance.

It seems that Traefik is unable to know when files are changed or added (when using the watch option) and the directory is a glusterfs volume.

I wonder if this is what I am running into...

https://stackoverflow.com/questions/48877567/traefik-docker-hot-reload-configuration#comment97995443_48877567

[Nepoxx]

@gtmadev I tried bind mounting using folder and that does not help.

This is a pretty major blocker for me, and I'd like to help move this forward. My initial idea is to add support for a kill signal that would trigger a config reload, probably SIGHUP (prometheus does this).

Thoughts?

[martine-stratdat]

This is also something I'd like to see. While Traefik may feel this is covered though it's letsencrypt solution it does not cover all real world cases.

• This includes easy loading of purchased certificates
• Cases where traefik is just one component of TLS use, including transitioning to traefik from other set ups.
• Instances where rate limiting may be come an issue

There two solutions that will work here as I see it:

• An API endpoint
• A watch on certificates or certificate entrties

Given that this is a barrier of entry for Traefik I would like to see a solution given a priority.

[DrEsteban]

We also desperately need this feature, as the current strategies for triggering certificate reload leave a lot to be desired.

[hezten]

I badly need a similar solution too, hope this is getting looked into :-)

[monotek]

Same here with traefik 1.x helm chart. Using existing secret for bought tls certs in kubernetes and telling traefik via ingress rule what the secretname is. Updating the secret has no effect at all. The old cert is still used...

Edit:

Seems our problem was that we had several secrets wiht the same name "my-tls-wildcard-secret" in different namespaces. We had to update all of them before traefik used it.

[ybizeul]

Same issue here.

Traefik version 2.2.0 built on 2020-03-25T17:32:57Z with docker

Before changing certificates :

drwxr-xr-x    2 root     root          4096 Apr 19 15:56 .
drwxr-xr-x    3 root     root          4096 Apr 19 13:24 ..
-rw-r--r--    1 root     root          1062 Apr 19 15:57 nabox.crt
-rw-r--r--    1 root     root          1679 Apr 19 15:57 nabox.key
-rw-r--r--    1 root     root           127 Apr 19 15:53 traefik.yaml

When changing certificates :

time="2020-04-19T14:08:16Z" level=debug msg="Configuration received from provider file: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{\"stores\":{\"default\":{}}}}" providerName=file
time="2020-04-19T14:08:16Z" level=info msg="Skipping same configuration" providerName=file
time="2020-04-19T14:08:16Z" level=debug msg="Configuration received from provider file: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{\"stores\":{\"default\":{}}}}" providerName=file
time="2020-04-19T14:08:16Z" level=debug msg="Configuration received from provider file: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{\"stores\":{\"default\":{}}}}" providerName=file
time="2020-04-19T14:08:16Z" level=info msg="Skipping same configuration" providerName=file
time="2020-04-19T14:08:16Z" level=info msg="Skipping same configuration" providerName=file

And here is the directory :

drwxr-xr-x    2 root     root          4096 Apr 19 15:56 .
drwxr-xr-x    3 root     root          4096 Apr 19 13:24 ..
-rw-r--r--    1 root     root          1062 Apr 19 16:08 nabox.crt
-rw-r--r--    1 root     root          1675 Apr 19 16:08 nabox.key
-rw-r--r--    1 root     root           127 Apr 19 15:53 traefik.yaml

Traefik config :

      - "--log.level=DEBUG"
      - "--api.insecure=true"
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--providers.file.directory=/tmp/ssl/" << File provider here
      - "--providers.file.watch=true"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entrypoint.to=webssl"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--entrypoints.webssl.address=:443"
      - "--entrypoints.graphite.address=:2003"
    volumes:               
      - "/var/run/docker.sock:/var/run/docker.sock:ro"            
      - "/opt/conf/ssl:/tmp/ssl" << Traefik config and SSL files

[MartinMuzatko]

We are generating certificates on request and move them to the right file position. This replaces the inode, which (I believe) breaks traefiks automatic discovery of certificates.

[majorgearhead]

Any word on this possible bug? We programmatically generate our certificates from a known authority and they are replaced for which Traefik cannot see happen. We have to manually bounce trafik to get it to pick up the new certificates.

[Opti98]

We generate out certificates from another authority too, and when we replace the certificates Traefik doesn't see the change. I have discovered if I touch the configuration file it does reload the new certificates, ie "touch certificates.toml"

[Sarke]

I have discovered if I touch the configuration file it does reload the new certificates, ie "touch certificates.toml"

Really? That did not work for me last time I tried. I think Traefik caches the parsed config files and if it's still the same it doesn't re-apply them.

[AndrewSav]

Of course traefik reloads the config file if it monitors it. This issue is for the case when inotify is not available, e.g. remote mount or when you mount a single file with docker. The current functionality covers vast majority of use cases, but not all.

[Ajedi32]

After encountering this problem myself I've done a bit of experimentation, and there seem to be several issues at play here:

First, it appears Traefik doesn't monitor the certificates files for changes at all. I might be wrong about this (I haven't dug around in the source code or anything) but based on my experiments (running Traefik 2.3.2 via the traefik:v2.3 image on Windows Subsystem for Linux) Traefik doesn't watch for changes to the certificate or key files. It does, however, watch for changes to the dynamic config file, in cases where you're using the file provider. If you run touch $file against your dynamic config file, Traefik will then check whether you made changes to the certificate or key files referenced in that config.

Secondly, the above trick where you touch the dynamic config file to reload certificates only works if the certificate in question is referenced under the tls.certificates key. It does not work for certificates that are only listed in the default certificate store. E.g. If you have this config:

tls:
  stores:
    default:
      defaultCertificate:
        certFile: /etc/traefik/certificates/my-site.cer
        keyFile: /etc/traefik/certificates/my-site.key

Then TLS will work as expected, but Traefik will not reload changes to the key or certificate files even if you touch the dynamic config file. No idea why. In order for that trick to work, you'd need a config more like this:

tls:
  certificates:
    - certFile: /etc/traefik/certificates/my-site.cer
      keyFile: /etc/traefik/certificates/my-site.key

  stores:
    default:
      defaultCertificate:
        certFile: /etc/traefik/certificates/my-site.cer
        keyFile: /etc/traefik/certificates/my-site.key

Third, note that even if Traefik does reload the certificates, those changes may not immediately affect the web browser you're testing with if that browser still has an open connection with the Traefik server. In order for the changes to take effect you may need to restart the browser to force it to close any existing connections. (In Chrome, pressing the "Close idle sockets" button in chrome://net-internals/#sockets will also work.)

[Gallardo994]

I am currently deploying Traefik into kubernetes with cert-manager providing wildcard certificates. Wildcard cert is used as default in Traefik via secret mounted together with configmap. According to this issue, should I be concerned with "restarting traefik" each time cert-manager renews the certs?

[redondos]

I am currently deploying Traefik into kubernetes with cert-manager providing wildcard certificates. Wildcard cert is used as default in Traefik via secret mounted together with configmap. According to this issue, should I be concerned with "restarting traefik" each time cert-manager renews the certs?

We run Traefik 1.7 and I don't know the 2.x behavior. Instead of restarting traefik pods to reload certificate secrets we rely on Global Default Backend ingress rules to overcome this limitation.

We create a * ingress rule in a namespace watched by traefik that includes a tls section with all the wildcards we need. cert-manager will manage those certificates and secrets, and traefik will use them for TLS termination.

[zedtux]

No matter the cases where Traefik woud or wouldn't reload the certificates, the initial request is about a way to force Traefik to reload the certs.

Is there any plans to implement it since the year and some months since this issue has been opened?

[bkraul]

Wondering the same thing. We are getting ready to deploy traefik in production with a large number of domains/certs...services that we can't really stop, so a reload for SSL certs is a must...apache and nginx provide this functionality.

[tsarna]

More specifically, I would like that way to be by sending a signal, such as SIGHUP

[bkraul]

A reload option would be awesome tho. SIGHUP would mean the service is being restarted. The point is to have no interruption. Take for instance nginx -s reload or httpd reload neither of them disrupt service to the running web application. But they do update most configuration settings on the fly...including SSL certs.

[tsarna]

@bkraul SIGHUP does not mean the service is being restarted. It's very common for daemons to catch SIGHUP as an indication that configuration should be reloaded. Apache, Nginx, sendmail, BIND, dhcpd, traditional implementations of inetd and cron, etc. all implement configuration reload on SIGHUP, and thus it's well supported by the kind of tools often used for updating configuration files. Although most tools support configuring a different signal today, in my experience SIGHUP is the most commonly implemented.

[MartinMuzatko]

@bkraul @tsarna how do you do sighup via docker? wouldn't an API be better?

[florentchauveau]

Hello all,

@Ajedi32 posted the solution above. It works.

TLDR: traefik does not monitoring the certificate files, it monitors the dynamic config file

Steps:

1. Update your cert file
2. Touch dynamic.yml
3. Et voilà, traefik has reloaded the cert file

There might be a gotcha with the default certificate store. You have to list your certificates twice. My dynamic.yml file looks like this:

tls:
  certificates:
    - certFile: /etc/ssl/any.site.com.crt
      keyFile: /etc/ssl/any.site.com.key
  stores:
    default:
      defaultCertificate:
        certFile: /etc/ssl/any.site.com.crt
        keyFile: /etc/ssl/any.site.com.key

You can test with curl -v, which avoids browser cache.