randombit
Universal binary support on MacOS
Currently, ,building as a universal binary breaks everything. For example if I try with vcpkg and botan (3.7.1, so a bit dated), I get errors like:
build/include/internal/botan/internal/simd_32.h:310:19: error: unknown type name 'uint8x16_t'; did you mean 'uint16_t'?
310 | const uint8x16_t mask = vld1q_u8(maskb);
| ^~~~~~~~~~
| uint16_t
build/include/public/botan/types.h:99:12: note: 'uint16_t' declared here
99 | using std::uint16_t;
| ^
In file included from /Users/m1/vcpkg/buildtrees/botan/src/3.7.1-3f4b712006.clean/src/lib/block/aes/aes_vperm/aes_vperm.cpp:16:
build/include/internal/botan/internal/simd_32.h:310:37: error: use of undeclared identifier 'vld1q_u8'
310 | const uint8x16_t mask = vld1q_u8(maskb);
| ^
build/include/internal/botan/internal/simd_32.h:311:62: error: use of undeclared identifier 'vreinterpretq_u8_u32'
311 | return SIMD_4x32(vreinterpretq_u32_u8(vqtbl1q_u8(vreinterpretq_u8_u32(m_simd), mask)));
| ^
build/include/internal/botan/internal/simd_32.h:313:64: error: use of undeclared identifier 'vreinterpretq_u16_u32'
313 | return SIMD_4x32(vreinterpretq_u32_u16(vrev32q_u16(vreinterpretq_u16_u32(m_simd))));
| ^
build/include/internal/botan/internal/simd_32.h:317:23: error: use of undeclared identifier 'vshlq_n_u32'
317 | vorrq_u32(vshlq_n_u32(m_simd, static_cast<int>(ROT)), vshrq_n_u32(m_simd, static_cast<int>(32 - ROT))));
| ^
build/include/internal/botan/internal/simd_32.h:317:67: error: use of undeclared identifier 'vshrq_n_u32'
317 | vorrq_u32(vshlq_n_u32(m_simd, static_cast<int>(ROT)), vshrq_n_u32(m_simd, static_cast<int>(32 - ROT))));
| ^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [build/obj/lib/block_aes_vperm.o] Error 1
To be clear, weird breaks happen like this with OpenSSL, LibreSSL, and libffi, but thought I would raise it here for Botan at least since it would be nice to get this fixed if possible. I'm not entirely sure how to detect when universal compilation is active either during configure or preprocessing time myself; I'm no skilled MacOS dev. But I hope it can possibly be resolved! In the meantime, lipo'ing the libraries would work, it's just not the most ideal solution.
It's somewhat tricky for us because we have a lot of system dependent code. Is there a solution for this? [Edit: By which I mean, what do other projects do here?]
One thing to try would be --cpu=generic. That disables all SIMD/hw specific code and likely will work or at least get you quite close. With the downside of course of some common operations being much slower than otherwise :/
Some new-to-me information about how this works from https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary
When you build a universal binary, Xcode compiles your source files twice—once for each architecture. After linking the binaries for each architecture, Xcode then merges the architecture-specific binaries into a single executable file using the lipo tool. If you build the source files yourself, you must call lipo as part of your build scripts to merge your architecture-specific binaries into a single universal binary.
That's interesting in that there is no (additional) magic happening from Xcode beyond lipo
@ethindp Can I ask what is not-ideal about running lipo over two separate builds? Is it just the issue of having to do two builds (configure.py + make) vs just a one-shot thing with XCode? I'm wondering if the fix here isn't just to offer an additional script that handles this for the user.
This used to work though. I was using it until the Mac build of Botan broke completely a few months ago. (I've been working on other things recently, so haven't looked to see if it's been fixed. The issue I raised hasn't had any updates though, so I assume not.)
