Cluster wide storage for /etc/letsencrypt

[s4s0l]

First of all great job @hamburml & @vfarcic. I must admit at the beginning, that I haven't run docker-flow even once. Just was looking at the source code to evaluate if it fits my needs. So far I think I'll give it a try and I like it a lot.

I was also going through some open/closed issues ad came across https://github.com/vfarcic/docker-flow-proxy/issues/56, there was a discussion about /etc/letsencrypt. While i was thinking about it some crazy idea came to me so i decided to share it with you. Hope you don't mind bothering you;)

What if docker-flow-letsencrypt or some companion container mounted /var/run/docker.sock of manager node and saved current keys or/and certbot files as secret in swarm? How much storage is needed here?

I've seen sentence "mount /etc/letsencrypt will be improved in the future", do you have any idea how? I'll be facing this problem in near future so maybe I can help. Using Ceph, Gluster or Swift in sole purpose of storing few kb of data is a little overkill for me. I'm going to give https://github.com/cpuguy83/docker-kvfs-driver a try.

[vfarcic]

@s4s0l You beat me to it :). I just released the support for secrets in DFP and was about to suggest the same to @hamburml. The idea is great and we should leverage secrets for certs.

@hamburml I thought to add environment variable CERT-SECRETS that would be a comma separated list of secrets. DFP would create configuration that would use those secrets from the /run/secrets directory. DFLE, on the other hand, would need to issue a docker secret create command that would store the secret. There should probably be a flag in DFLE that you could use to decide whether to PUT the certificate directly to the proxy or store it as a secret. What do you think?

[hamburml]

Hi!

I've seen sentence "mount /etc/letsencrypt will be improved in the future", do you have any idea how?

To be honest, no. But it's also not sufficient to only store the certificates (it is for the proxy, but not for Certbot). Certbot (Let's Encrypt) creates account-information which are reused when a certificate is renewed. We need to store these too. I still wait for official infinit integration https://infinit.sh/docker

@vfarcic @s4s0l The idea of using secrets to send certificates to DFP is nice, I like that. Currently I am full with work so I don't have much free time. Let's see if I can find some free time today or tomorrow. But I have one question: How does DFP knows when the secret is updated (certificate was renewed?).

[vfarcic]

@hamburml At the moment CFP would not know what to do with certs coming from secrets. We can change that. I thought to add env. var. CERT-SECRETS (equivalent to CERTS), reconfigure parameter serviceCertSecret (equivalent to serviceCert), and a new PUT request certSecret (equivalent to cert). I think that DFLE is using the last option (PUT request). If that's correct, and can implement that one first.

The only new thing from those changes to DFP would be that certs would be loaded from /run/secrets directory.

[vfarcic]

Actually, it might be even easier for DFP. I can change configs to always use /run/secrets dir and make a soft link from /certs to /run/secrets. With that change, everything should work no matter whether we use "normal" certs or secrets.

[hamburml]

I think that DFLE is using the last option (PUT request).

Correct, currently PUT request is used to send the certificates to the DFP. I would change DFLE to use secrets instead of PUT request when a env. var. is set. Thanks for the info :)

[vfarcic]

@hamburml Secrets are added to DFP. A user will have to create proxy service with a secret (it can be even an empty file; to be tested). From there on, you can update that secret whenever you fetch a new cert. I don't think we'll even need to reload the proxy config (to be tested). If we do, a reload request with recreate=true should be enough.

[vfarcic]

@hamburml Do you think you'll be able to work on this issue?

[hamburml]

@vfarcic I haven't abandoned this companion service and I really want to work on this matter. I will try to add it this week.

[hamburml]

hamburml/docker-flow-letsencrypt:testing sends certificates via docker secret to DFP when USE_SECRET env. var. is set. It's working on my end, open for testers :). Tomorrow I want to correct the indentation and change the readme.

https://hub.docker.com/r/hamburml/docker-flow-letsencrypt/tags/

letsencrypt-companion:
    image: hamburml/docker-flow-letsencrypt:testing
    networks:
      - proxy
    environment:
      - DOMAIN_1=('haembi.de' 'www.haembi.de' 'blog.haembi.de')
      - DOMAIN_2=('michael-hamburger.de' 'www.michael-hamburger.de' 'blog.michael-hamburger.de')
      - [email protected]
      - PROXY_ADDRESS=proxy
      - USE_SECRET=true
      - CERTBOT_CRON_RENEW=('0 3 * * *' '0 15 * * *')
    volumes:
          - /etc/letsencrypt:/etc/letsencrypt
    deploy:
      labels:
        - com.df.servicePath=/.well-known/acme-challenge
        - com.df.notify=true
        - com.df.distribute=true
        - com.df.port=80
      placement:
        constraints: [node.id == <nodeId>]
      replicas: 1

edit What I don't like is the current way this is handled. Currently everytime a certificate is created or renewed these commands are executed:

docker service update --secret-rm $secretname $PROXY_ADDRESS
docker secret rm $secretname
docker secret create $secretname $folder.combined.pem
docker service update --secret-add $secretname $PROXY_ADDRESS

First command removes old secret from the proxy, removes the secret itself, adds the secret (the new certificate) and adds the secret to the proxy. When the secret is removed from the proxy (first command) and the secret is added (last command) the service is restarted (I don't know if it's really restarted, but it's not reachable for a very short time. When using PUT Request for storing certificates this works without any downtime (or it was so fast it never occurred to me).

[hamburml]

I thought about using docker api directly instead of docker cli. There is an update command but sadly the data itself can't be updated. https://docs.docker.com/engine/api/v1.28/#operation/SecretUpdate

So sad :/ I think it's better to not use secrets for the certificates in the current situation. When a certificate is updated a simple PUT Request sends it to DFP. This happens without a downtime. When the certificate is stored as a docker secret, DFP needs to be stopped, secret removed and recreated with the new certificate file and DFP restarted with the new secret (the 4 commands in the post above).

Any ideas/thoughts about this?

[vfarcic]

I agree. It will be better to use PUT to send DFLE certs to DFP. Secrets are more appropriate for longer lasting information.

[bvis]

What about "secret versioning".

Secrets are immutable but you could emulate secret versioning adding a predefined sufix to each secret, for example: docker secret create ${secretname}.v1 ... and add it to the proxy service. Finally the proxy service will use just the last version of the secret on its list.

When the proxy is restarted only the last version is initialized in the proxy.

[vfarcic]

I'm not sure I understand @bvis. That would mean that when a new certificate comes in, the proxy will need to be restarted so that the new secret is added. I'm probably missing something...

[hamburml]

Yeah, secrets aren't renewed when a container is currently running - a new secret (even a fully new secret with versioning as mentioned) results in a restart of DFP which is a downtime.

[bvis]

You are right. I assumed it won't need to restart when you use --secret-add flag on service update, but it restarts the service anyway.

It can be seen in the official docs: https://docs.docker.com/engine/swarm/secrets/#example-rotate-a-secret

[drozzy]

Hey, is this part of a tagged release?

[hamburml]

No, sorry. I/We abandoned the secret approach and I would favor a new PUT request with the certificate as content. What is currently missing is that DFP is able to request the certificates from DFLE.

For example DFP restarts, doesn't have a certificate but knows through some env. var. that there is a service which can provide certificates. It sends a request to DFLE and DFLE sends all available certs to DFP.

I would like to change the compantion to allow something like this.