[Mac] Fail to build universal (fat) library for arm64 and x86_64 on M1 Mac

[SergeiNevdakh]

I'm trying to build static universal lib on Mac M1 (arm64) for 2 architectures: arm64 and x86_64 to support both intel and M1 Macs.

Configure with options: ./configure.py --module-policy=modern --cc-abi-flags="-force_cpusubtype_ALL -mmacosx-version-min=10.14 -arch x86_64 -arch arm64" --disable-shared-library --disable-modules=aes_armv8,sha1_armv8,sha2_32_armv8 --disable-neon (Several modules are explicitly disabled here, because configure script detects them as available but they won't build anyway)
Then build fails

clang++  -fstack-protector -pthread -stdlib=libc++ -force_cpusubtype_ALL -mmacosx-version-min=10.14 -arch x86_64 -arch arm64 -std=c++11 -D_REENTRANT  -O3 -DBOTAN_IS_BEING_BUILT -Wall -Wextra -Wpedantic -Wshadow -Wstrict-aliasing -Wstrict-overflow=5 -Wcast-align -Wmissing-declarations -Wpointer-arith -Wcast-qual  -Ibuild/include -c src/lib/utils/cpuid/cpuid_arm.cpp -o build/obj/lib/utils_cpuid_arm.o
src/lib/utils/cpuid/cpuid_arm.cpp:208:50: error: unknown use of instruction mnemonic without a size suffix
   auto neon_probe  = []() noexcept -> int { asm("and v0.16b, v0.16b, v0.16b"); return 1; };
                                                 ^
<inline asm>:1:2: note: instantiated into assembly here
        and v0.16b, v0.16b, v0.16b
        ^
src/lib/utils/cpuid/cpuid_arm.cpp:209:50: error: out of range literal value in '.word' directive
   auto aes_probe   = []() noexcept -> int { asm(".word 0x4e284800"); return 1; };
                                                 ^
<inline asm>:1:8: note: instantiated into assembly here
        .word 0x4e284800
              ^
src/lib/utils/cpuid/cpuid_arm.cpp:210:50: error: out of range literal value in '.word' directive
   auto pmull_probe = []() noexcept -> int { asm(".word 0x0ee0e000"); return 1; };
                                                 ^
<inline asm>:1:8: note: instantiated into assembly here
        .word 0x0ee0e000
              ^
src/lib/utils/cpuid/cpuid_arm.cpp:211:50: error: out of range literal value in '.word' directive
   auto sha1_probe  = []() noexcept -> int { asm(".word 0x5e280800"); return 1; };
                                                 ^
<inline asm>:1:8: note: instantiated into assembly here
        .word 0x5e280800
              ^
src/lib/utils/cpuid/cpuid_arm.cpp:212:50: error: out of range literal value in '.word' directive
   auto sha2_probe  = []() noexcept -> int { asm(".word 0x5e282800"); return 1; };
                                                 ^
<inline asm>:1:8: note: instantiated into assembly here
        .word 0x5e282800
              ^
5 errors generated.

[devnexen]

Just curious, would building separately arm64 and x86_64 and gluing them with lipo work for you ?

[SergeiNevdakh]

@devnexen, It does produce a fat library, indeed. But I'm still not sure that it will be OK on later stages. I've just started trying to make universal app, and building separately require some big adjustments to the build system.

And second issue is the configure script generates build.h with some defines. How do I configure it to use with multiple archs?

I mean, I can configure it to build for arm, then configure to build for intel. Make both libs and lipo them into fat lib. Then in order to use it in some app that is built with 2 architectures, it would be great if configure produces build.h that works with both.

[mouse07410]

Frankly, I don't understand why one may want or need a FAT binary or library. If you're on M1 - build library for M1, and link against it. If on x86 - build for that. Why would you need an M1-capable binary on x86, or vs. versa?

[devnexen]

there might be a need for easy deployment for mac devices, there are still intel variants.

[mouse07410]

Yes, there sure still are Intel variants of Mac, and I have several of them at work. But I never build fat binaries, almost-always compile for the native architecture, sometimes (rarely) - cross-compile. In general, fat binaries seem to be more trouble than they're worth, plus they're space-hogs.

[SergeiNevdakh]

Guys, I think that arguing about pros and cons of fat binaries is out of scope. Documentation tells that it's possible to build fat: https://botan.randombit.net/handbook/building.html#on-macos Yes, it's pretty outdated, for intel/ppc. However it's still there, and there are no significant differences between intel/ppc fat and intel/arm fat.

Considering that fat binaries is a way Apple goes, there's a demand to have this feature. So It would be great to either have this option working, or have it explicitly stated in docs that it's impossible.

[devnexen]

things have changed since the "ppc old days" tough. digging into apple docs, this is the way to do with M1 devices

https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary

You can theoretically do a xcode project and automate this with a proper apple dev license but behind the scene it just does all the steps above.

Might need doc clarifications indeed.

[briankendall]

I'm currently converting a project that uses botan from x86_64 only to an x86_64/arm64 universal app. Though it sounds like from last year's conversation that the issue of whether supporting universal builds is important has been settled, I'll add that it is absolutely essential for many macOS apps that they be built as a universal app. In fact, until such a time that Apple stops supporting x86_64 altogether (and that will hopefully be quite a long time from when I've written this), all macOS apps should be compiled as universal unless there is some reason preventing them from doing so, such as supporting an arm64 only feature.

With that out of the way, I quickly discovered that just using lipo to combine together an x86_64 and arm64 build of botan is not enough, as build.h has platform specific code in it, and will throw errors when compiled for the wrong architecture.

So I created a universal build of botan with the following steps:

1. Created two separate build directories and release directories named x86_64 and arm64
2. Ran configure.py in both directories, specifying the correct architecture as per the botan documentation for each respective directory and a different installation directory for each using --prefix
3. ran make and make install
4. Used lipo to combine together the two resultant libraries, e.g. lipo -create -output libbotan-2-x86_64-arm64.a dist/x86_64/lib/libbotan-2.a dist/arm64/lib/libbotan-2.a
5. Installed the x86_64 version of the include directory into my project
6. Renamed build.h to build_x86_64.h
7. Copied build.h from the arm64 include directory into my project at the same location as build_x86_64.h, and named it build_arm64.h
8. Created a new build.h in my project containing the following:

#if defined(__x86_64__)
    #include "build_x86_64.h"
#elif defined(__aarch64__)
    #include "build_arm64.h"
#else
    #error Unsupported architecture for botan
#endif

That seems to have done the trick! Hopefully that will aid someone else in the same endeavor. And hopefully at some point soon botan will properly support arm64/x86_64 universal builds.

[securitykernel]

Thanks @briankendall for the writeup!

We should either add a helper script or update the instructions in the handbook. Any opinions?