botan 1.15 with boost asio HTTPS problem

[drppetrov]

I have web client based on boost-beast which is relying on botan for TLS. However, handshake fails with ay other site but yours: https://botan.randombit.net/news.html

I am attaching the wire-shark log, which shows clearly what is the difference, the Certificate Status package. See attachment. Any hints are welcome!

[randombit]

Botan 1.15 is not something that exists. Possibly this is a Boost or Beast version?

Also I'm really not sure why it would happen to work for botan.randombit.net and not other sites, considering that site is hosted on a nginx server so the peer in that case is OpenSSL.

In your WORKS side it looks like QUIC is being negotiated, so I'm quite certain that is not Botan as we do not currently support TLS v1.3 or the necessary other pieces needed to support QUIC.

What specifically are you seeing in your failed case? Can you try running your client against a local server (eg openssl s_server or botan tls_http_server) and seeing if you get any useful debug output?

[drppetrov]

My apologies, I am using botan 2.15 !

However, the problem is not in QUIC. It is some protocol I am not aware of, certainly noting to do with botan. What fail is the handshake operation and in particular the certificate verification step. In case the status message is transmitted the connection continues normally and output is correct. Otherwise I get a certificate verification exception (TLS_Exception(Alert::BAD_CERTIFICATE)), against botan website too if I bypass the if step in the code:

    if(state.server_hello()->supports_certificate_status_message())
         {
         state.set_expected_next(CERTIFICATE_STATUS); // optional
         }
      else
         {
         try
            {
            auto trusted_CAs = m_creds.trusted_certificate_authorities("tls-client", m_info.hostname());

            callbacks().tls_verify_cert_chain(server_certs,
                                              {},
                                              trusted_CAs,
                                              Usage_Type::TLS_SERVER_AUTH,
                                              m_info.hostname(),
                                              policy());
            }
         catch(TLS_Exception&)
            {
            throw;
            }
         catch(std::exception& e)
            {
            throw TLS_Exception(Alert::INTERNAL_ERROR, e.what());
            }
         }
      }

p.s. I am using the default implementation from the example:

	class CredentialsManager : public Botan::Credentials_Manager 
	{
	public:

		CredentialsManager()
		{}
	
		auto trusted_certificate_authorities(const std::string&, const std::string&) -> std::vector<Botan::Certificate_Store*> override
		{
			return { &m_certStore };
		}

	private:

		Botan::System_Certificate_Store m_certStore;
	};

[reneme]

Most probably you are seeing BAD_CERTIFICATE because Botan's default TLS policy has require_cert_revocation_info() enabled. Hence, the certificate verification fails if neither a valid OCSP response nor a CRL are available for the TLS server certificate in question.

We can rule out an available CRL as you use the Botan::System_Certificate_Store where CRL retrieval is currently not implemented (macOS, Windows).

Regarding OCSP: If OCSP stapling is enabled and supported, the TLS server will provide a valid OCSP response. That's what if(state.server_hello()->supports_certificate_status_message()) is about. I.e. once the else branch in your referenced listing is taken, we can be certain, that no certificate revocation info will be available at this point.

That said, Botan provides (fairly rudimentary) support for fetching an OCSP response (see http_util) during the TLS handshake's certificate verification step. Though, this might be disabled at build time, not supported on your platform or fail due to local network particularity. For instance, http_util doesn't support HTTP proxies.

Bottom line, assuming that your Botan::TLS::Policy requires revocation info, http_util is unable to provide an OCSP revocation info (for any reason above), and CRL retrieval is not implemented in Botan::System_Certificate_Store. Then, your stack will depend on the TLS server to provide a stapled OCSP response in the handshake. Otherwise, the handshake is aborted with BAD_CERTIFICATE.

It would be great if you could verify my hypothesis by overriding the Botan::TLS::Policy you are currently using to make ::require_cert_revocation_info() return false. This should "fix" your TLS handshakes. For production usage that's not the way to go, though. We'll look into options afterwards.

Also, Botan::TLS::Policy::to_string() allows you to dump your active TLS policy which would be beneficial for debugging this issue.

[reneme]

One other hypothesis (assuming the http_util works as expected): You might be seeing a clock skew issue. Given the OCSP response (fetched from the CA) is created instantaneously (with a precise time stamp) and your local clock is just a little slow. Botan might then error out with "OCSP not yet valid", as the fetched response has a time stamp that lies in the future for your local machine. Such things result in flaky handshake failures, usually.

[drppetrov]

Thank you @reneme for your detailed explanation. I think I would need some time to digest all that. I will definitely try what you and @randombit suggested and come back to report.

[lieser]

As you haven't replied yet I'm assuming your questions are answered. If not please reopen the issue.