erlang
Inconsistent behavior building on Linux/MacOS against OpenSSL 3.x
Describe the bug
Building from source with fips enabled using --with-ssl=[OpenSSL 3.x dir] on macos fails at Checking for FIPS_mode_set()... no
An identical build on Linux completes successfully (Checking for FIPS_mode_set()... yes) which is confusing, as it seems FIPS_mode_set doesn't actually exist in OpenSSL 3.x.
To Reproduce
mkdir /tmp/erlang-test
cd /tmp/erlang-test
git clone https://github.com/openssl/openssl
pushd openssl
./Configure enable-fips --prefix=/tmp/erlang-test/openssl-3.x
make && make install
popd
git clone https://github.com/erlang/otp
pushd otp
./configure --enable-fips --with-ssl=/tmp/erlang-test/openssl-3.x
Expected behavior
I would expect to see an OpenSSL 3.x friendly check, rather than a FIPS_mode_set() check which seems to only be valid when using OpenSSL 1.x.
As the FIPS_mode_set check is there however, I also expect current Linux builds --with-ssling OpenSSL 3.x to fail, which does not match my experience.
Affected versions I encountered this while attempting to build 26.2 with FIPS enabled, but it looks like anything newer is the same. I'm not sure which older versions this behaviour is present in.
@udkyo Could you try #8277 where I have removed the check for FIPS_mode_set which was incorrect in multiple ways. It's currently based on OTP-26.2.3.
Looks good @sverker. Using that PR on OTP-26.2.3 I'm able to build on MacOS + Linux and enable FIPS on each. Thank you very much!