[DISCUSSION] Improve API of X.509/TLS cert verify callback

[mpg]

Currently, our X.509 chain verification functions can accept a callback that can for example perform additional checks, or clear flags found by our checks. However, this callback is currently called once for each certificate in the chain, making it hard to check properties of the entire chain and act on flags for the entire chain based on that.

Quoting from a comment from a few years ago:

• It's something I had in mind ever since introducing mbedtls_x509_crt_verify_chain: this would be a great structure to pass to the verify callback. There has been discussions above that the verify callback could in principle reconstruct the chain across calls, but that's at least extremely inconvenient (if not downright impossible) if you consider that a single ssl_conf, hence a single callback context, can be used for multiple connections (potentially concurrent).

• One potential drawback with exposing mbedtls_x509_crt_verify_chain is that it would make a currently-internal type part of the API. But I think the added flexibility for users would be well worth it. (We don't have to share exactly that structure, we could isolate a public type, and keep a private type which would potentially extend it.)

• One question that's been on my mind is whether this should be a new option, or completely replace the existing verify callback, as it's strictly more powerful. The main argument for replacing rather than adding would be to avoid having to maintain and test multiple interfaces. The main argument for adding is we can do it anytime, while replacing could only be done in 3.0 (which is already full) of 4.0 (which is likely a bit far in the future).

Well, now 4.0 is in the near future, so it's probably time to revisit this.

This task is to decide what we want to do, and then break it down into a series of estimated tasks.

[awakecoding]

Here is my what I'd like to see with external certificate validation:

A simple callback type that contains the raw X.509 certificate chain exactly as provided by the server, with no automatic CA chain completion or validation. The X.509 certificates should be provided as an array of byte buffers along with an array of the byte buffer sizes - this is the most reusable format that can then be used to reconstruct X.509 certificate objects in various external application libraries. Keep in mind it should be easy to load the certificates in something like the X509Certificate2 class in .NET, or any X.509 classes from other libraries and languages that would be used for external validation on top of mbedtls. I'd say keep the certificate chain in the same order as it appears over the wire, but if we want to enforce a specific order, I suggest using leaf-first ordering.

The current verification callback can be set using either mbedtls_ssl_set_verify or mbedtls_ssl_conf_verify and it accepts a void pointer for application-specific data. I suggest we keep that for our new external validation callback, but it would be important to provide enough context-specific information for proper validation, starting with the hostname to be used for validation against the SANs. The issue with mbedtls_ssl_set_verify is that it sets a callback at a higher level than mbedtls_ssl_conf_verify which is config-specific. One workaround is to registry the application pointer with a context-specific structure that contains information like the hostname used for validation. However, since we pretty much always need the hostname, I suggest adding it explicitly in the callback type. As for the two places where we can currently register the callback, maybe we could include the mbedtls_ssl_config pointer in the callback to expose the entire config for external validation, but it would require calling more mbedtls APIs to extract config-specific parameters. This would still be a good way to expose pretty much anything else.

If the idea of passing the X.509 certificates as an array of byte buffers is not suitable, then I suggest going with an array of mbedtls_x509_crt*, but it would be a lot easier if we can pass byte buffers for external languages and libraries which would likely just call mbedtls APIs to extract the raw bytes underneath anyway.

Last but not last, I would like to see an example where external certificate is used by an application to manually call the same mbed TLS validation APIs used internally, automatically. The goal would be to provide the best way to perform the same kind of validation done internally, but with the possibility of extending the validation with things like certificate validation exceptions (trust on first use like SSH, where you remember the cert hash, etc). Once you validate externally, you can take full control over the process and extend it in any way you like, or you can fully offload the verification process to platform-specific APIs like those used by SChannel on Windows, etc. The two primary goals of external validation being to extend what's done internally with custom logic, or replace the internal logic entirely to use an external validation API provided by another stack than mbedtls.