Discussion: docker-letsencrypt-dns V3

[adferrand]

This project has been here for quite a while now. It has been designed with a very narrowed scope at the beginning, but you are more and more to use it, with more and more use cases really interesting to handle.

I can see that the initial design became a limitation for the container development. If I do not rethink things now, this design will be the main cause of complexity in the future.

Furthermore, since wildcard certificates are available with LetsEncrypt, managing theses certificates became critical. If I think my container helped a little on that matter, it is clearly not sufficient in term of interoperability with other services. An API is needed.

So I wrote a specification for a new version, with a heavy refactoring and a lot of new features. It is available here : Project V3 specifications.

Before starting to code, I would like your opinion about this specification, if you think it is a good design, what is missing and so on. Do not hesitate to post your comments on this issue !

[githtz]

I just read the V3 specification and I gotta say, you have really been thinking. In my opinion, your concept in itself looks very well thought out.

I especially like that the configuration is no longer done via environment variables. This makes starting the container much easier. In my opinion, the YAML format is perfectly suited for this purpose. I also like the possibility of doing the configuration via a web interface and a REST-Api, even though this feature is less relevant to me. Therefore I would be happy if these additional features could be switched off or run in separate containers.

The automatic migration from V2 to V3 will certainly be a very special challenge, but offers all existing users a comfortable migration and is therefore very welcome.

[astraios]

Hi, I stumbled onto this project because I was looking for a way to handle, in an automated way, my letsencrypt certificates. I manage multiple domains, hosted at different providers, and your "lexicon" integration is great.

I'm sorry to see so few comments here in comparison to your "docker pull" count. IMO you did some serious work here. I wanted to thank you, You literally saved my day today.

Your current V2 can't handle DNS challenge from multiple providers but you seem to solve this "issue" in your V3 design. I like your ideas and your approach seems sound. I like having a core process with an HTTP API. I think your UI should live in a separate package.

Also, having a configuration file is a nice option, it could be stored as a K8S configMap for example.

I hope you don't plan to change how certbot store it's certificates (/etc/letsencrypt). I think it's very convenient this way. I can dive into the container, run certbot commands or whatever. Yes the current V2 can only restart containers but I use consul to store the certificates and deploy them accross my nodes using consul watch. A script of mine browse the /etc/letscencrypt folder and check if the found certificates match the consul k/v entries. If not, consul's k/v entries are updated.

If you have any question, feel free to ask.

[pascalandy]

Just read the V3 specs. Very elegant!

[christian-weiss]

I saw your design concept here: https://github.com/adferrand/docker-letsencrypt-dns/wiki/Project-V3-specifications,-aka-Namedcrypt

In general i like it, but there is one point which i disagree: "a configuration made in only one place, a new file config.yml, and no environment variable"

As a developer / admin i have hundreds of services to manage and instead of having a long collection of certs to be managed in a "central cert config" i prefer to have every thing i need for a service configured next to this service config (incl. which cert is used; e.g. in my docker-compose, helm-chart, kubernetes-resource, shell-script, etc. - so ENV is very welcome). This would help when it comes working in teams, "maintain in parallel", when a user has permissions to manage project A but not project B. And when it comes to processing of "structured data" then you have more trouble on generator- and consumer-side as shell scripts for example do not have native support for yaml files (jq and yq are not available in every context). A set of ENV looks like an approach of the 90s but it is still a good one. So keep letsencrpyt account in a central place but "job config" decentralized but discoverable or pushable.

In my opinion you should switch the perspective from "software developer that designs a service" to "how should my service look like to fit administrator needs" when designing V3.

Required feature set form the perspective of an admin:

• i want to get new certs (if there is no cert)
• i want to get renewed/new certs (if my cert has expired or is about to expire)
• i want to get an instant self-signed cert for the very first start of my application (to avoid that my service will start in a crash-loop while the true-cert-process is taking a while; a chicken-egg-problem)
• i want to get my cert automatically deployed to where my service is expecting it (path)
• i want to let my service reload config (to pickup that cert) automatically (triggered config reload command)
• i want to let my service to check if it is ready for config-reload (e.g. "nginx -t" to test config; to avoid breaking a currently running service) automatically (triggered test command)
• i want a service that can be monitored for its healthyness
• i want a service that can resume after a blackout automatically
• i want a service that adds virtually nothing to the time required for a letsencrypt process
• i want a service that is able to perform all task without extra "wait-loops" or sequences (do other stuff while waiting for DNS propagation of job number 1, e.g. provision TXT-record for job number 2)
• i want a service that is able to create & renew 1000 certs per hour (or lower if there is a rate limit by letsencrypt)
• i want a system that warns me (slack, e-mail, telegram) if this service has discovered a problem (a cert has expired, while service had a downtime, threshold for dns propagation exceeds, max number of retries of X, etc.)
• i want a service that has webhook support to let other systems trigger e.g. a cert renewal or to let the service trigger other webhooks (a jenkins pipeline, github, travis, gitlab, etc.)
• i want a service that makes is possible to have multiple teams using the same service without interfering each other (auto discover "jobs", e.g. in a microservice-world kubernetesless)
• statistic: list of all certs / endpoints
• i want a service that has everything available in its API (nothing is exclusive to the WebUI)
• i want a service that gives me a list of all (configured and discovered) cert endpoints/domains (with export function to be able to monitor it with other tools easily e.g. Qualys SSL Rating see https://www.ssllabs.com/ssltest/analyze.html?d=www.google.de)

For job-discovery see: https://github.com/adferrand/docker-letsencrypt-dns/issues/63#issuecomment-597037663

Last but not least, we should check how we can reduce the attack surface (docker root user required?), full host system path traversal required?, deploy.sh for hundreds of servers in a single place or better a pull system (vault, keybase + token via webhook)?

[adferrand]

First of all, thanks a lot for your feedback. It is always a tremendous effort to understand, as a developer, what are the needs and the point of view of the users of the tool you develop. Even if in this case, I am also a user of my tool, let's face it, it does exactly what I need because, well, I code exactly what I need ^^

Just a short context about this V3 specification: I wrote it two years ago, and my professional experience in clouds technologies since then made me evolve a little on some point, including the problematic of environment variables. More on this later.

The first (chronologically) goal I try to reach here is to refactor my tool before it is too late. Indeed, if I wrote it at the beginning for my own needs (the ones I explain in the README introduction), sharing it to the community proved it usefulness, and hence more use cases to implement, and even more ambitious features that I have in mind.

But at this point, it is still just a bunch of bash scripts, with no proper data structure, testing pyramid, code quality, extensibility and so on. Either it stays as it is and refuse further features, or I push it to the next level, and make it a proper program with what it implies in term of engineering.

The work is almost finished, and you can check the dnsrobocert branch (its future new name) if you are interested.

But it does not mean at all that I will stay with this spec. It is just the solid base I need to code in order to, let's say, start the serious work. Indeed, if it took me two years to actually get this done, it is because I became in the mean time a core developer of the two programs that hold the core logic, lexicon for the DNS APIs, and certbot for the TLS certs. This experience showed me several things that are missing around the wildcards certificates and the DNS challenges, and I will try to respond to that with this project.

[adferrand]

Let's talk about environment variables.

I am a backend developer, and so I am quite concerned by the data model and the objects used to express this model in a program. On that matter environnement variables irritate(d?) me a lot.

It is basically a key value "store", holding only strings, with no persistency on their own. What to do with this, how to express elegant and complex structures of data?

I now a lot of public dockers that, after a while with an accumulation of new fonctionalities because of their popularity, finishe with a list of dozens of environment variables to cover all the possibilities. This becomes very hard for the user, in term of feature discoverability, to read these big lists of variable that you can use, with no validation, or structure control to verify that you do not any mistake.

Worse, specifically for my tool, we have an hybrid data model: part of it is described in env variables, part is in domain.conf that is either weakly structured. That is why I want a coherent data model, that can be expressed as a structure file like json, yaml or toml.

However I agree that you cannot avoid env variables, they are everywhere, and a core way to communicate data to dockers. This cannot just be ignored. Moreover, Kubernetes showed me that you can have a better handling of these variables, and in fact feed their value with better sources, like secrets, configmap, CRDS, and all the templating systems that you can add on top of it like Helm.

Here is my position now: it is not the env variables concept that bothers me, but the fact to design a data model with them. However, having a bijection to environnement variables of a model is OK for me.

Here is what I propose. The model of the tool will be structured, typed, and defined in a json schema. Naturally providing a file will be sufficient to entirely define the behavior of the tool. Let's call this the file provider.

I propose here to write a environnement provider, that allows the tool to configure itself through the environment variables available in the system. It is not about defining a list of variable names here. It is about defining the pattern of variable names that allows to express the data model.

Let's take the acme.email_account for instance. If in yaml, it would be:

acme:
  email_account: [email protected]

The corresponding environment variable name will be DRC__ACME__EMAIL_ACCOUNT. Note the DRC__ prefix, to define a variable namespace and avoid collisions, the double underscore __ defining a field accessor, and the property names written as UPPER_CASE convention.

So for the maps. What about arrays, like in profiles?

They are simply expressed with an index leading to each property of the object available at this index: DRC__PROFILES__1__NAME, DRC__PROFILES__1__PROVIDER and so on.

This way you can entirely avoid a configuration file, and use only environment variables. Of course, both can be used, and express parts of the data model, since they are just a projection of the data model, and can be merged together into a consolidated structure inside the tool

This was for the local providers, the ones that you define inside the local system that is running the tool, and correspond to the current features of letsencrypt-dns.

Tomorrow, I will continue with the remote providers, like what is done with letsencrypt-nginx-proxy-companion in term of environnement variables, or traefik with Docker labels.

[ndbeals]

Did you recently remove that feature (the DRC__ environment variables)? I just transferred servers and updated docker/docker containers and these environment variables don't seem to work (and they are applied inside the container correctly).

[adferrand]

No it is just that I did not implement it yet. I will do it as soon as I have the time, it is the next item in my list.

[ndbeals]

I guess the fact that it was "working" prior was just artefacts from somewhere else?

FWIW: I'd use this. I like to push the config to git, but I don't want to include the username and password, for obvious reasons.

Ever since you removed the environment variable support this DRC__ suggestion seems like the only way to keep my secrets separate from my config, which seems a bit of an anti-pattern TBH.