Support Authority Information Access (AIA) certificate extension

[DerGuteMoritz]

Summary

Some web servers present TLS certificates which don't contain the full authority chain to validate them. Instead, they use the Authority Information Access (AIA) certificate extension for referring to an externally stored certificate. Major browsers support this extension and will fetch external certificates when encountering such an AIA block in a certificate during validation. Lychee however will fail with a certificate validation error.

Reproduction

At the time of writing, https://citeseerx.ist.psu.edu/ presents the following first certificate (most information elided for readability):

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            5f:00:7c:72:0c:3c:41:e7:d7:12:eb:6a:7a:df:32:f5
        Signature Algorithm: sha384WithRSAEncryption
        Issuer: C=US, O=Internet2, CN=InCommon RSA Server CA 2
        Validity
            Not Before: Feb  7 00:00:00 2025 GMT
            Not After : Mar 10 23:59:59 2026 GMT
        Subject: C=US, ST=Pennsylvania, O=The Pennsylvania State University, CN=*.ist.psu.edu
        [...]
        X509v3 extensions:
            [...]
            Authority Information Access: 
                CA Issuers - URI:http://crt.sectigo.com/InCommonRSAServerCA2.crt
                OCSP - URI:http://ocsp.sectigo.com
        [...]

While the certificate chain contains some further certificates, the issuer's (C=US, O=Internet2, CN=InCommon RSA Server CA 2) is not among them. That one has to be fetched via the URL given under X509v3 extensions -> Authority Information Access -> CA Issuers. For further details, see the spec linked in the references below.

Here is how Lychee currently fails:

$ echo https://citeseerx.ist.psu.edu/ | lychee - 
  1/1 ━━━━━━━━━━━━━━━━━━━━ Finished extracting links                                                                                                                          Issues found in 1 input. Find details below.

[stdin]:
   [ERROR] https://citeseerx.ist.psu.edu/ | Network error: error sending request for url (https://citeseerx.ist.psu.edu/) Maybe a certificate error?

🔍 1 Total (in 0s) ✅ 0 OK 🚫 1 Error

While browsers like Firefox (tested with version 136.0.1) indicate that the certificate is perfectly valid.

References

• The AIA extension is specified in RFC 5280, section 4.2.2.1
• https://github.com/danilobellini/aia is a Python implementation that could serve as inspiration
• Same issue in curl's TODO list

[whentze]

Note that that sites like that work in Firefox not because Firefox implements AIA (and they don't intend to), but because it has a set of pre-loaded intermediate certificates.

Both rustls and openssl have no AIA support either, and it seems quite hard to pull off - it essentially means the TLS client (a dependency of HTTPS clients like reqwest) needs access to a whole HTTPS client itself. This is what killed un-stapled OCSP, and I think may spell doom for AIA as well.

That's not to say that this cannot be implemented. Just that there are other potential solutions than "implement AIA".

[mre]

Okay, that's helpful. From what I can see, this functionality should be implemented in upstream dependencies. I doubt that we'll be able to handle this case anytime soon.

[thomas-zahner]

@mre Is right. As we don't implement any encryption/certificates ourselves it is an issue of either the SSL library used (openSSL) or soon rustls.

Also see the relevant curl discussion and rustls discussion.