RFC: new way of loading certificate files

[f1-outsourcing]

1. nothing searches in private #bind 0.0.0.0:8080 ssl crt "${HAPROXY_CRT_DIR}"/dev.marathon.mesos.crt ssl-load-extra-files key "${HAPROXY_KEY_DIR}"/dev.marathon.mesos.key #bind 0.0.0.0:8080 ssl crt "${HAPROXY_CRT_DIR}"/dev.marathon.mesos.crt ssl-load-extra-files key bind 0.0.0.0:8080 ssl crt "${HAPROXY_CRT_DIR}"/dev.marathon.mesos.crt

2. my key does not end on crt.key, just key (It is obvious from the name in front that it is already the key for that domain)

3. Why can't I specify like any other application my own crt and key file name? I don't get why you even want to automate this. Also the certs and private directories are different on distributions.

[ALERT] 210/124445 (30) : parsing [/etc/haproxy/haproxy.cfg:52] : 'bind 0.0.0.0:8080' : No Private Key found in '/etc/ssl/certs/dev.marathon.mesos.crt' or '/etc/ssl/certs/dev.marathon.mesos.crt.key'.

Is there an config option to specify a key dir and a cert dir?

[wlallemand]

Hello,

This is not a bug and works as designed. Indeed the "crt" option takes a basename which will be used to search other files by adding suffixes (".issuer", ".ocsp", ".key" etc.) It won't try to remove the extension of your basename. https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#ssl-load-extra-files

Is there an config option to specify a key dir and a cert dir?

You can only do that with a cert dir with 'crt-base' but we could probably add a 'key-base' too, that could be backported in 2.2.

The problem with a "key" directive is that it will apply on the whole bind line, and that's not a good idea because it will apply on all crt instances. Only a "key" directive in a crt-list file makes sense.

[f1-outsourcing]

Yes would be very nice, I am now scripting to auto generate the pems. I

If you have key and cert dir and specifiy those files, you do not need to do anything more. Except for ca validation.

You should also not force people the use of names. Although most follow standards, it is just never 'one size fits all'. I don't think I have ever seen any application, forcing me to use basename.key.

I think it is even the opposite of logical what is now implemented. In my case all the file names are the same with a nice 3 letter different extension. Furthermore your basename usage is also inconsistent, logically you should apply .crt to it. So in my case it would be dev.marathon.mesos.crt.crt so I could configure it as dev.marathon.mesos and crt and key are appended. But then you have the 3rd argument that your configuration file values are not exactly the same as files on disk.

So I would rethink this ssl configuration, stick to file names easier configuration, less coding, less options for mistakes.

[wlallemand]

I agree with you regarding the use of names, but we had some limitations:

1. You can have several "crt" directives on a bind line
2. The directives on a bind line are supposed to apply on the whole line, not in declaration order

This was the original design of the SSL configuration parsing in HAProxy 1.5, I honestly hates it as most people use separate files and directories instead of PEM. When the OCSP was implemented, we couldn't add a "ocsp" directive because of (2), so a .ocsp file is looked up using the basename in "crt". And every features followed the same design after that, becoming worse with the bundles, which could look after ".ecdsa.ocsp" for example.

I won't be able to change the way we look at files because of backward compatibility. But the introduction of the "ssl-load-extra-files" directive was a first step to disable the auto discover and be able to specify the files with new directives in the future.

Specifying the extra files manually by certificate won't be compatible with the "crt" directive, so we will need to use the "crt-list" keyword for that.

A lot of things changed internally in HAProxy, we now have the concept of certificate storage, which is not really visible in the configuration. But what HAProxy does is that it will store all files related to a "crt' in the certificate storage and then load this storage for each new bind line it is used. So maybe what we need is to be able to declare a new certificate storage in a dedicated section. I don't have any definitive proposal for this, I'll probably open a new ticket with some thoughts.

[f1-outsourcing]

Good, interesting background info!

[wlallemand]

After discussing this with several people here are more thoughts about this:

• the ssl-load-extra-files feature is convenient, people like being able to auto-load a .key but adding this at this end of the filename is not convenient. We could add an option to remove the current extension before looking for the .key.

• some people want to be able to search a key in a separate directory, so the "key-base" will be implemented

• other people would like to put their filenames manually without any auto-discovery, with a "key" and "chain" keyword. However this is not compatible with bind lines. I thought the crt-list was a good way to do it, but after some discussions on future improvements we conclude that it will make the configuration worse. What we need is a way to describe a certificate, and a new section will be dedicated to this.

This is an example of what it could be:

certificates
    crt foobar.crt key foobar.key chain foobar.chain
    crt /etc/ssl/certs/site1.crt key /etc/ssl/private/site1.key

listen
    bind *:443 ssl crt foobar.crt # use the crt described in the certificates section

[WRMSRwasTaken]

How about using the resolvers section as a model to be able to define a name per certificates section?

certificates foobar
    crt /etc/ssl/certs/foobar.crt
    key /etc/ssl/private/foobar.key

So you'd just have to pass foobar to other directives (like for example making it compatible with crt-list or being able to specify multiple named certificates to the bind line)

[wlallemand]

We thought about it first, but that's not really convenient for people having thousands of certificates.

What I suggested will be compatible with crt-list, however if we want to use a different name we could use an optional "name" parameter, for example:

certificates
    crt foobar.crt key foobar.key chain foobar.chain
    crt /etc/ssl/certs/site1.crt key /etc/ssl/private/site1.key name site1

listen
    bind *:443 ssl crt foobar.crt crt site1 # use the crt described in the certificates section

[wtarreau]

I'm overall fine with this but I'm convinced that we really need to be able to name the section and probably to make it mandatory, like this:

certificates mysite
    crt foobar.crt key foobar.key chain foobar.chain
    crt /etc/ssl/certs/site1.crt key /etc/ssl/private/site1.key name site1

listen
    bind *:443 ssl crt foobar.crt crt @mysite:site1 # use the crt described in this certificates section

This would allow to easily load all the certs mentioned in such a group (e.g. crt @mysite), and more importantly will ease config management at scale. I don't think it adds any complexity at all, and should avoid difficulties imposed by a centralized list when it comes to multiple applications having their own certs. Typically an application that needs to be deployed could simply be shipped with a single "certificates" section mentioning the various paths and certs.

What I like with this format is that we can later think about extending it, for example by adding:

• import : loads a file into the section
• crt-path : preset the path to load all "crt" files
• key-path : preset the path to load all "key" files etc.

[wlallemand]

We talked about this, face to face with @wtarreau and decided to change a little bit the format:

# we declare a "mysite" scope
crt-store mysite
    crt foobar.crt key foobar.key chain foobar.chain
    crt /etc/ssl/certs/site1.crt key /etc/ssl/private/site1.key name site1

# or use the default scope
crt-store
    crt cert1.crt key cert1.key name cert1  

# the @:cert1 is used to specify the certificate "cert1" in the global scope
# @mysite:foobar.crt take the "mysite" scope
listen
    bind *:443 ssl crt @mysite:foobar.crt crt @mysite:site1 crt @:cert1

[wlallemand]

I've backported 8e8581e to 2.2.

[serimin]

It works well. But this option does not seem to be compatible with "set ssl cert, etc" command.

[wlallemand]

The problem is that the path of each file is not stored, only the path of the "crt" is indexed. So when you are doing a commit this is not a problem, but if you do a "set" with a ".key", haproxy is not able to know what was the extension of the corresponding crt.

So it's a little bit difficult to fix with the current architecture, you will still need to set ".key" at the end of the current path, and we will have the same problem with the new "crt-store" keyword.

I'll really don't want to introduce this in a 2.2 or 2.3 release without having a clean solution, so I'll revert the patch before it become a problem for the backward compatibility of the "set ssl" command.

[wlallemand]

What I could do is change a little bit the behavior, we could force the "crt" file to have a ".crt" extension as a requirement for this feature. It's less flexible but it make more sense and it's easy to replace the extension by ".crt" before the lookup

[wlallemand]

The above commit should fix this, I also added a reg-test which test this with "set ssl cert".

[wlallemand]

Unfortunately I won't backport this to 2.2, the CLI still has the bundle code and I don't want to risk to break it, so I'm still reverting this in 2.2.

[hholst80]

The current workflow is also inconsistent with the highly opinionated K8s workflow. kubectl will by default create two keys called "tls.crt" and "tls.key" for a TLS secret.. Not "tls.crt.key", thus not working out of the box.

Workaround (bash shell):

kubectl create secret tls haproxy --cert=<(cat fullchain.pem privkey.pem) --key=privkey.pem

[hholst80]

For public information, why can't we just have a "key" keyword in the bind statement, explicitly specifying a key file?

The directives on a bind line are supposed to apply on the whole line, not in declaration order

I do not know the internal magic how the suffixes are resolved but I do not think the above statement is true or at least how it is incompatible with the suggested syntax below.

    bind [...] ssl crt /tls/fullchain.pem key /tls/privkey.pem crt /etc/tls/another.crt

This would resolve 2 certificates (fullchain.pem and another.crt) and 2 keys (privkey.pem and another.{key,crt.key} [must exists if another.crt is not a cert bundle]).

Another alternative could be to introduce a new keyword (which intuitively seems backwards but it gets the job done)

    bind [...] ssl crtkey /tls/fullchain.pem /tls/privkey.pem crt /etc/tls/another.crt

[wlallemand]

The current workflow is also inconsistent with the highly opinionated K8s workflow. kubectl will by default create two keys called "tls.crt" and "tls.key" for a TLS secret.. Not "tls.crt.key", thus not working out of the box.

This was fixed in 2.3 with the "ssl-load-extra-del-ext" keyword.

For public information, why can't we just have a "key" keyword in the bind statement, explicitly specifying a key file?

The directives on a bind line are supposed to apply on the whole line, not in declaration order

I do not know the internal magic how the suffixes are resolved but I do not think the above statement is true or at least how it is incompatible with the suggested syntax below.

    bind [...] ssl crt /tls/fullchain.pem key /tls/privkey.pem crt /etc/tls/another.crt

This would resolve 2 certificates (fullchain.pem and another.crt) and 2 keys (privkey.pem and another.{key,crt.key} [must exists if another.crt is not a cert bundle]).

Unfortunately there is no way to know which key is associated with which crt using this syntax, because there is no "declaration order" on a bind line, all options are applied for the whole line. That means if you change the order of the crt and key options it must behave the same way, so that won't work if you have multiple key and crt keywords on the same line.

Another alternative could be to introduce a new keyword (which intuitively seems backwards but it gets the job done)

    bind [...] ssl crtkey /tls/fullchain.pem /tls/privkey.pem crt /etc/tls/another.crt

I think it's kind of confusing but it's an interesting idea, we tend to avoid having multiple arguments to a keyword but we should maybe reconsider something like this.

[WRMSRwasTaken]

If you plan to implement the said crtkey <certfile> <keyfile> or something in this direction, having an alternative to crt-list, for example: crtkey-list <file> with two file paths per line would be a good idea aswell imho.

[wlallemand]

I don't think think adding a new crtkey-list keyword is a good idea, each line generates only one SSL context, you can't have multiple key and crt on the same line, so just having a key keyword that works only on a crt-list section could be worth considering. It's not the same problem as a bind line.

[rustyx]

Can't bind options order be preserved? In that case the key keyword can be required to be either never specified or specified exactly as many times as the crt keyword. Then the correspondence when it is specified is always 1-to-1, in the same order, e.g. crt a key ka crt b key kb or key ka crt a key kb crt b or crt a crt b, but never crt a key ka crt b.

[wlallemand]

That would be the best option if the options order was preserved from the start, but it does not make sense to preserve order only for one keyword, it would be confusing. The key keyword is only part of the problem, people will eventually ask to set other files or options, for a specific certificate. The best "simple" way to implement this without breaking anything would be to do it in a crt-list file, because each line contains a certificate, but this would be kind of overkill for someone who just have one or two certificates to use.

But your comment make me thing of something, but it would need to be discussed with the other maintainers because it's kind of touchy. What if the SSL options are applied in their own scope, which could be defined with an ssl keyword (or a new one if it breaks things).

bind *:443 ssl crt /etc/ssl/certs/foobar.pem key /etc/ssl/private/foobar.key ocsp /etc/ssl/ocsp/foobar.ocsp ssl crt /etc/ssl/certs/site1.pem key /etc/ssl/private/site1.key ocsp /etc/ssl/privte/site1.ocsp

This way we could emit an error if there are multiple "key" or "ocsp" keywords in the same scope. It will be kind of a crt-list, but instead of a newline character, it's an "ssl" keyword. If it's confusing for backward compatibility we could decide of a new keyword to do it (what about tls ?)

[nakoo]

bind *:443 ssl crt /etc/ssl/certs/foobar.pem key /etc/ssl/private/foobar.key ocsp /etc/ssl/ocsp/foobar.ocsp ssl crt /etc/ssl/certs/site1.pem key /etc/ssl/private/site1.key ocsp /etc/ssl/privte/site1.ocsp

This way we could emit an error if there are multiple "key" or "ocsp" keywords in the same scope. It will be kind of a crt-list, but instead of a newline character, it's an "ssl" keyword. If it's confusing for backward compatibility we could decide of a new keyword to do it (what about tls ?)

+1 for this. I'm trying to use docker secret to store my TLS certificates and ocsp files. But since docker secret only supports file, without folder support, it was impossible to provide ocsp as static file unless using socat which needs additional packages that I want to avoid.

[wlallemand]

Unfortunately this would break a lot of configuration so that is not possible, we will probably proceed this way:

• add the "key", "ocsp" etc keywords to crt-list (since it's only one SSL context per line it is possible easily)
• add a support for "inline" crt-list (so we could use the same syntax without breaking the old one)

[nakoo]

Unfortunately this would break a lot of configuration so that is not possible, we will probably proceed this way:

• add the "key", "ocsp" etc keywords to crt-list (since it's only one SSL context per line it is possible easily)
• add a support for "inline" crt-list (so we could use the same syntax without breaking the old one)

Thank you for the answer. crt-list syntax is much cleaner. Can I expect it to be working like this? I checked the docs just in case but seems doesn't support yet.

In crt-list file

example.com.pem [key example.com.pem.key ocsp example.com.pem.ocsp]

In haproxy.cfg

bind *:443,[::]:443 ssl crt-list /etc/haproxy/certs/crt-list strict-sni

[wlallemand]

example.com.pem [key example.com.pem.key ocsp example.com.pem.ocsp]

It could work like this indeed, but nothing was pushed in the master repository yet, we'll probably try to have at least these two keywords for 2.6.

[wlallemand]

Unfortunately the changes are too deep in the configuration parser, and will require to change a lot of things, it's a bit too late to be integrated into 2.6.

[sigv]

Is this expected to land in 2.7?

[quality-leftovers]

I'd like to add here that the ability to specifiy the key as seperate configuration itemg would enable supporting hardware security modules via OpenSSL key engine (which is deprecated) or OpenSSL 3 store API with just a few lines of code.

When changing the configuration please bear in mind that the OpenSSL 3 API supports URIs for key locations. So it would be great if URIs would be supported in addition to "plain" file paths.

See #71

[scryptio]

What is the current state about this topic? Its been 3 years and unfortunately I cannot find new / further information about this. I once read it was coming in 2.6, then that it was planned for 2.7. Now 2.8 is on the horizon. An official statement would be great about whether this topic is still on the list, being worked on or not considered.