`SSLClientServerTest.*` tests fail with OpenSSL 3.2.1

[mgorny]

When attempting to run the test suite on Gentoo Linux amd64, I'm seeing most of the SSLClientServerTest.* tests fail:

[  FAILED  ] 6 tests, listed below:
[  FAILED  ] SSLClientServerTest.ClientCertPresent
[  FAILED  ] SSLClientServerTest.ClientEncryptedCertPresent
[  FAILED  ] SSLClientServerTest.MemoryClientCertPresent
[  FAILED  ] SSLClientServerTest.MemoryClientEncryptedCertPresent
[  FAILED  ] SSLClientServerTest.TrustDirOptional
[  FAILED  ] SSLClientServerTest.CustomizeServerSSLCtx

This is on 548dfff0aef25e36e971af96b49ce7fbb72d840e (but I also reproduce the same result on 0.15.3).

Detailed results:

[----------] 8 tests from SSLClientServerTest
[ RUN      ] SSLClientServerTest.ClientCertPresent
test.cc:5160: Failure
Value of: res
  Actual: false
Expected: true
[  FAILED  ] SSLClientServerTest.ClientCertPresent (7 ms)
[ RUN      ] SSLClientServerTest.ClientEncryptedCertPresent
test.cc:5160: Failure
Value of: res
  Actual: false
Expected: true
[  FAILED  ] SSLClientServerTest.ClientEncryptedCertPresent (7 ms)
[ RUN      ] SSLClientServerTest.MemoryClientCertPresent
test.cc:5243: Failure
Value of: res
  Actual: false
Expected: true
[  FAILED  ] SSLClientServerTest.MemoryClientCertPresent (6 ms)
[ RUN      ] SSLClientServerTest.MemoryClientEncryptedCertPresent
test.cc:5243: Failure
Value of: res
  Actual: false
Expected: true
[  FAILED  ] SSLClientServerTest.MemoryClientEncryptedCertPresent (7 ms)
[ RUN      ] SSLClientServerTest.ClientCertMissing
[       OK ] SSLClientServerTest.ClientCertMissing (5 ms)
[ RUN      ] SSLClientServerTest.TrustDirOptional
test.cc:5308: Failure
Value of: res
  Actual: false
Expected: true
[  FAILED  ] SSLClientServerTest.TrustDirOptional (6 ms)
[ RUN      ] SSLClientServerTest.SSLConnectTimeout
[       OK ] SSLClientServerTest.SSLConnectTimeout (1006 ms)
[ RUN      ] SSLClientServerTest.CustomizeServerSSLCtx
test.cc:5428: Failure
Value of: res
  Actual: false
Expected: true
[  FAILED  ] SSLClientServerTest.CustomizeServerSSLCtx (7 ms)
[----------] 8 tests from SSLClientServerTest (1056 ms total)

Please let me know how can I help you debug this.

Full output: test.txt

[mgorny]

This is with OpenSSL 3.2.1, Brotli 1.1.0.

[mgorny]

This seems specific to OpenSSL 3.2.1. With 3.1.5, they work.

[yhirose]

@mgorny thanks for the report. It's interesting. Are you able to run the tests with 3.2.0 to see what will happen?

[mgorny]

Sure, just done that. I've getting the same failures as with 3.2.1.

[yhirose]

Thanks a lot. It seems like something has changed between 3.1.x and 3.2.0 about certificate handling... At this point, I have no idea why the problem came from. I'll do some research about 3.2 when I have time. Please let me know if you find any helpful information regarding this matter. Thanks!

[mgorny]

The release notes are here: https://www.openssl.org/news/openssl-3.2-notes.html

My first thought was "The default SSL/TLS security level has been changed from 1 to 2", but I've tried adding SSL_CTX_set_security_level(..., 1) everywhere and that didn't help. I don't see anything else that would look like a breaking change.

[Tachi107]

Hi, I can confirm this issue is impacting Debian too. It as been reported as Debian bug #1069357. I've been able to reproduce these failures on cpp-httplib versions as old as 0.10.8. It is unclear how severe this issue is.

[yhirose]

I tested with the openssl version 3.0.13 and confirmed that the unit test works, but it fails with the version 3.3.0 on my MacBook. They are both installed by Homebrew.

I also generated .pem files with openssl cli 3.0.13, and made a unit test executable with the 3.3.0 header fines and libraries. In this condition, the unit test still works fine.

So it sems like .pem files generated by openssl cli 3.3.0 can no longer be accepted by the current usage of SSL_ functions in cpp-httplib... I have no clue at this point, but I'll continue to look into it when I have time.

[Tachi107]

Nice find! I'll use that workaround in the Debian package in the meantime :)

[yhirose]

Maybe due to https://github.com/openssl/openssl/issues/1418..

[yhirose]

It seems like the following code to make certificates is no longer correct with the recent OpenSSL versions...

https://github.com/yhirose/cpp-httplib/blob/ae63b89cbf70481ae60515dfd95467e91eecd992/test/Makefile#L65-L77

@davidgfnet, @PixlRainbow, @ottigeda, @sebblanchet, @jimmy-park, do you have any thoughts on this issue?

[Tachi107]

Maybe due to openssl/openssl#1418..

Why though? That issue got fixed in OpenSSL 3.0, but this only manifests on OpenSSL 3.2 an newer.

Edit: it might be helpful to run a git bisect on OpenSSL between 3.2 and 3.1 to see which commit effectively introduced the issue.

[opoplawski]

Any progress here? We're hitting this in Fedora as well.

[yhirose]

Nothing so far. Any helpful information would be greatly appreciated.

[Tachi107]

I've run a git bisection and identified the OpenSSL commit which introduced the regression: https://github.com/openssl/openssl/commit/342e3652c791bdb06e08abcc169b4456c83ccd00. I've used the following git bisect script:

#!/bin/sh

set -efuxo pipefail

git clean -x -d -f 2>&1 >/dev/null || exit 125
git restore . || exit 125
./Configure --prefix=/tmp/tmp.ApsDLGFvVT/osslprefix --debug no-tests || exit 125
make -j 2 || exit 125
make install_sw || exit 125
cd ../cpp-httplib || exit 125
rm -rf build || exit 125
PATH="$(realpath ../osslprefix/bin):$PATH" meson setup --buildtype=debug -Db_lto=false -Dcpp-httplib_openssl=enabled -Dcpp-httplib_zlib=disabled -Dcpp-httplib_brotli=disabled -Dcpp-httplib_compile=true -Dcpp-httplib_test=true --pkg-config-path $(realpath ../osslprefix/lib64/pkgconfig/) build || exit 125
ninja -C build || exit 125
GTEST_COLOR=yes meson test -C build --verbose --test-args='--gtest_filter=SSLClientServerTest.*'

The commit has changed the default X.509 certificate format from v1 to v3, which is what is breaking the tests. I don't know why tests are broken by this change, but the work around is pretty easy.

I've submitted #1940 to fix this for the Meson build system.

[yhirose]

@Tachi107 fantastic job! I'll make the same change to test/Makefile.

[yhirose]

I made the same fix in test/Makefile.

@sum01 you may want to make the same fix that we did in test/meson.build and test/CMakeLists.txt.