Certificate Compression (RFC8879)

[tmshort]

Replacement of #9155 (github doesn't allow change of source repositories)

Add support for brotli compression Add support for Zstandardi compression Add support for RFC8879 Add support for callback API, which allows the user to handle either compression expansion or pre-compressed server certificates.

Checklist

• [X] documentation is added or updated
• [X] tests are added or updated

[openssl-machine]

This PR is in a state where it requires action by @openssl/otc but the last update was 30 days ago

[tmshort]

Not sure why address_ub_santitizer is failing. I tried running it locally (same options, etc), and it succeeds. The error seems to be in a file this PR does not touch?

[tmshort]

Thank you @mattcaswell! I will be updating this shortly.

[tmshort]

Ping @mattcaswell ?

[mattcaswell]

Ping @mattcaswell ?

Sorry - been away on vacation for the last 2 weeks. Will try and pick this up when I am caught up.

[tmshort]

I removed BUF_MEM from the APIs.

[tmshort]

Should we run some tests with this code and asan?

Done.

[tmshort]

ping @mattcaswell ?

[tmshort]

I added most of the fixes as fixups/additional commits; hopefully to make review easier.

[tmshort]

OTC want to discuss the design of this feature so there maybe some more comments that come out of that. I'd like to discuss it at next week's meeting if possible (although we weren't quorate this week due to vacations). I can ask OTC if you can attend too if you would like to do that...although the meeting time is likely to be dreadful for your timezone (OTC meet every Tuesday 0800UTC - 1000UTC).

Yes, that would be 4am to 6am my time. I could possibly attend, given enough warning.

While I like the flexibility in this implementation (simple compress API, get/set compress API, compress callback, auto-compression), I can certainly understand the desire to simplify it, but I would hate for it to be dumbed down.

In terms of use cases, I see the following (these are the extreme cases):

• Simple website: Set the certificate in the SSL_CTX, call SSL_CTX_compress_certificate() API with the preferred algorithm, done.
• Less simple website: multiple SSL_CTX's are set up as above or via SSL_CTX_get_cert_to_compress() and SSL_CTX_set_comp_cert(), and assigned during the early/servername callback.
• Complex: CDN fronting many domains: Set the certificate via SSL_use_certificate() in the SSL during the early/servername callback from external database. Use the compression callback to determine if compression is desired, and look up a compressed certificate based on algorithm. Set compressed cert if available, or compress and cache in external database. If it expands, note in external database and return no compression from callback.

I suppose the callback might not be necessary in the CDN case where the certificate and any compressed forms are set in the same callback (early or servername). This might require compressing the certificate multiple times (outside OpenSSL) and that a compressed version ends up unused, but it's a one-time compression, assuming that the result is cached externally.

I had suggested an OSSL_COMP_CERT object that could be referenced (rather than copied), this might make ownership clearer, but I didn't want to go down that path, yet.

The elephant in the room with respect to compression is the client cert; it cannot be pre-compressed due to the context data received from the server, so it has to be done at run-time, dynamically. And if the client has to do it that way, why can't the server?

[mattcaswell]

OTC: Receiving of compressed certificates should be "on" by default Transmission of compressed certificates should be "off" by default, unless you have already cached some certificates using SSL_CTX_compress_cert() or you have set the callback. We should only negotiate compression algorithms for transmission for which we have a cached certificate (unless you are using the callback).

[tmshort]

OTC: Receiving of compressed certificates should be "on" by default Transmission of compressed certificates should be "off" by default, unless you have already cached some certificates using SSL_CTX_compress_cert() or you have set the callback. We should only negotiate compression algorithms for transmission for which we have a cached certificate (unless you are using the callback).

I was actually thinking of getting rid of the callback; it's not strictly necessary, and complicates the API. I think (server-side) pre-compression can do everything necessary. Unless the @openssl/otc really wants the callback, the following is how I propose to update this PR (basically in agreement, except the callback, but with additional details).

Client Side

Pre-compression is not an option. User assigns certificates, and specifies compression algorithm preferences/options (via SSL_{CTX_}set1_cert_comp_preferences() if desired). Compression is done dynamically per the configuration, due to the context received from the server.

Server Side

There are three basic use cases:

1. Easy-to-use compression done during initialization
2. External store of compressed certs set in SSL (e.g. CDNs/virtual hosting)
3. External store of compressed certs set in SSL_CTX (e.g. load-balanced servers)

Easy-to-use option

For (1), the APIs are simple, and apply to both SSL_CTX and SSL, although they are less useful for SSL:

1. Assign certificates, chains, etc, as normal
2. Set compression algorithm preferences (optional) via SSL_{CTX_}set1_cert_comp_preferences()
3. Compress certificates via SSL_{CTX_}compress_certs(), this API takes an algorithm to specify the compression algorithm to use to compress, with algorithm = 0 meaning to use all algorithms configured in step 2. Trying to compress with an algorithm not specified in step 2, or not enabled in the library, will generate an error.

No (server-side) state-machine compression will be performed; if a certificate hasn't already been compressed, then it won't be. Certificate expansion will be handled properly, meaning the expanded certificate won't be used. The SSL_{CTX_}compress_certs() API would compress all certificates assigned to an SSL or SSL_CTX (note the plural). While the APIs for (2) and (3) apply only to the most recently added certificate.

Provide compressed data to SSL

For (2), the APIs are a bit more complex, and both apply to SSL_CTX and SSL, although they are less useful for SSL_CTX:

1. Set compression algorithm preferences (optional) via SSL_set1_cert_comp_preferences()
2. Assign certificates, chains, etc, as normal
3. Assign one-or-more compressed certificate to the "current certificate" (i.e. the last one set) via SSL_set1_compressed_cert(). This certificate data is retrieved from an external store, not unlike the certificate itself.
4. Go back to (2) to assign a different type of certificate (e.g. ECDSA vs RSA)

This is most useful during the early or servername callbacks; based on the servername indication, a certificate can be loaded into an SSL via SSL_set1_compressed_cert().

This will require, for instance, setting the ECDSA cert/key/comp-cert(s). Then setting the RSA cert/key/comp-cert(s). It's not one-step because there may be multiple compressed versions of the installed certificate

Provide compressed data to SSL_CTX

For (3), it's similar to (2), but is done at initialization time of an SSL_CTX like (1). In this case, the certificate and compressed form are distributed to each server as they are set up. Compression is done in one place (e.g. a certificate server) rather than locally. This would be done via SSL_CTX_set1_compressed_cert().

Helper routines

While users could certainly read the RFC to figure out how to format the data to be compressed, the SSL_{CTX_}get_cert_to_compress() helper API can be used to get the properly formatted data. It may be more useful to simply provide the compressed data instead, and avoid having the user try to compress it themselves (e.g. SSL_{CTX_}get_compressed_cert()).

Enabling

The certificate compression feature defaults to "on".

A client merely needs to have the compression algorithm in its library to either transmit a compressed certificate or receive one.

In addition to having the compression algorithm in its library, a server needs to explicitly pre-compress (via the APIs above) a certificate in order to transmit it. A server will accept compressed certificates.

The feature can explicitly be disabled via the SSL_OP_NO_TX_CERTIFICATE_COMPRESSION (which disables transmitting compressed certificates) and the SSL_OP_NO_RX_CERTIFICATE_COMPRESSION (which disables transmitting the extension, and thus indicating no support for receiving compressed certificates) options.

[mattcaswell]

Unless the @openssl/otc really wants the callback

The impression I got from other OTC members was that a simpler API is better. It's always possible to add additional API functions later - but taking them away is hard. My guess is that removing the callback altogether would be acceptable.

In addition to having the compression algorithm in its library, a server needs to explicitly pre-compress (via the APIs above) a certificate in order to transmit it. A server will accept compressed certificates.

If I understand correctly the feature is "on" but won't be used by a server if there are no pre-compressed certificates available to send. Just a normal (uncompressed) certificate will be sent back to the client.

For client certs you can't precompress anyway, so this is always on if the library has support for the compression algorithms and SSL_OP_NO_TX_CERTIFICATE_COMPRESSION is not being used.

If my understanding above is correct then I think the way you have described things is more-or-less in line with the sentiment in the OTC meeting, so I think this would be acceptable.

[tmshort]

If my understanding above is correct then I think the way you have described things is more-or-less in line with the sentiment in the OTC meeting, so I think this would be acceptable.

Exactly!

[hlandau]

So long as this simpler solution can work with the server using our existing callback options to select a certificate dynamically, with the server not necessarily knowing the set of certificates or SNI names it might receive in advance, I think that solution is fine.

[tmshort]

So long as this simpler solution can work with the server using our existing callback options to select a certificate dynamically, with the server not necessarily knowing the set of certificates or SNI names it might receive in advance, I think that solution is fine.

Yes, that should continue to work (and one of my goals).

[mattcaswell]

Looks like a rebase is required.

[tmshort]

Looks like a rebase is required.

(Un)fortunately, I've become a rebase master...

[tmshort]

@mattcaswell this should have all updates now for the "simplified" API.

[tmshort]

Regarding crypto/objects/objects.pl which gitbub won't let me comment on directly.

Not sure if this is really necessary.

One of the problems with the FIPS module is that it includes many definition files that aren't used just by FIPS. For example, NIDs which are used for things other than encryption (e.g. Compression). So although this does not contain any code in the FIPS module, it would be considered a FIPS change. That seems counter-intuitive.

This avoids that, but it's not really scaleable. If there were some way to isolate the non-FIPS definitions, etc., from FIPS, it would go a long way to avoiding the dreaded (fips changed) label.

I can certainly remove it, but it will suddenly get that label.

[tmshort]

All comments should be addressed, now

[mattcaswell]

Needs another rebase too.

[tmshort]

I suspect the failures are due to obsolete or old installs of zstd/brotli on Windows. I think review can continue while that gets resolved.

[tmshort]

I suspect the failures are due to obsolete or old installs of zstd/brotli on Windows. I think review can continue while that gets resolved.

Looks to have been due to an update to vcpkg configuration on windows-latest.

[mattcaswell]

Is this ready for re-review or is there still outstanding work to be done following from my previous round of comments?

[tmshort]

Is this ready for re-review or is there still outstanding work to be done following from my previous round of comments?

@mattcaswell, Yes.

This comment is still outstanding. Docs need updating to say they return an "empty" method on failure - or something like that

I replied to this comment directly in the diff, it's not showing up in the comment thread‽‽

[tmshort]

ping @mattcaswell

[mattcaswell]

ping @mattcaswell

I know you're never going to believe me after 16 days of inactivity - but I was actually partway through reviewing this when your ping came through!! Honest!

[mattcaswell]

Ping @openssl/committers for second review

[tmshort]

ping @mattcaswell

I know you're never going to believe me after 16 days of inactivity - but I was actually partway through reviewing this when your ping came through!! Honest!

I didn't want to ping you just before the releases... and then there's the unrelease... sigh.