AVX512 Serpent amalgamation tests fail with GCC

[arckoor]

When running ci_build.py --cc='gcc' amalgamation on a machine that supports the avx512 instruction set, various tests fail: 659 tests failed (in block_ciphers cipher_modes cryptobox stream_ciphers). Tested on the master branch with commit 823bcb4. I used act to emulate the Github runners like so: act -P ubuntu-22.04=catthehacker/ubuntu:full-22.04 -j linux &> act-log.txt, and commented out everything but the gcc/amalgamation job in ci.yml. I tested on different machines with different underlying cpu models, and all failed in the same way. act-log-08-linux.txt act-log-12-linux.txt

[randombit]

This is almost certainly a miscompilation. There was a similar one affecting Serpent AVX2 with GCC in amalgamation build, for which a workaround was found. Unfortunately reducing the amalgamation to a test case that GCC devs will accept is non trivial.

I would suggest any of

• Use Clang instead
• Disable serpent_avx512 module
• Consider using --minimized-build and then explicitly enabling the features you need/use (eg --enable-modules=ecdsa,sha2_32,aes,gcm,x509,...)

[arckoor]

Yea, with clang it does work without issue. As far as I know we currently do not need the feature at all, I just happened to notice while testing for #4443. Feel free to close / ping me again to test should any potential fixes arise.

[thesamesam]

Unfortunately reducing the amalgamation to a test case that GCC devs will accept is non trivial.

If you can turn it into a standalone file, even if huge, that has some main() which aborts on a bad condition, I can try reduce it. (That would include taking the amalgamation and just adding a main which calls the right test function, if that works.)