Outgoing HTTPS connections fail with OpenSSL 3.2+

[maskit]

ATS fails to establish outgoing HTTPS connections if it uses OpenSSL 3.2+.

SocketManager::sendto() returns EINVAL (22) because ats_ip_size(dst) returns 0 here. https://github.com/apache/trafficserver/blob/433bb35e3334e21d7040d23c12bca06e4aa81a95/src/iocore/net/BIO_fastopen.cc#L121-L124

Workaround:

diff --git a/src/iocore/net/SSLNetVConnection.cc b/src/iocore/net/SSLNetVConnection.cc
index 4d0e2d9a2..b088026bf 100644
--- a/src/iocore/net/SSLNetVConnection.cc
+++ b/src/iocore/net/SSLNetVConnection.cc
@@ -209,6 +209,9 @@ SSLNetVConnection::_make_ssl_connection(SSL_CTX *ctx)
   if (likely(this->ssl = SSL_new(ctx))) {
     // Only set up the bio stuff for the server side
     if (this->get_context() == NET_VCONNECTION_OUT) {
+      SSL_set_fd(ssl, this->get_socket());
+
+#if !defined(BIO_SOCK_TFO)
       BIO *bio = BIO_new(const_cast<BIO_METHOD *>(BIO_s_fastopen()));
       BIO_set_fd(bio, this->get_socket(), BIO_NOCLOSE);
 
@@ -217,6 +220,7 @@ SSLNetVConnection::_make_ssl_connection(SSL_CTX *ctx)
       }
 
       SSL_set_bio(ssl, bio, bio);
+#endif
     } else {
       this->initialize_handshake_buffers();
       BIO *rbio = BIO_new(BIO_s_mem());

[zwoop]

This seems like a priority for v10.0.0

[ywkaras]

Verified the problem on Ubuntu. The tls_verify Au test fails.

[bryancall]

I am seeing the same issue on Fedora 40

[ywkaras]

It looks like the problem is a consequence of the f_tcp_fastopen option not set: https://github.com/apache/trafficserver/blob/e31eca4565db721c8f2032d5baa161affeb7073a/src/iocore/net/SSLNetVConnection.cc#L215

[Cyborgscode]

@jeredfloyd backport fix works. Tested on production system.