swift-nio CMake build causes runtime crash from Atomics

SwiftNIO doesn't support CMake by default, so I tried make it to happen. Upon calling MultiThreadedEventLoopGroup(), I ran into a runtime crash from Atomics.ManagedAtomic It could have something do do with a STATIC library overrides. Same trouble on both Mac and Linux.

Actual behavior

💣 Program crashed: Bad pointer dereference at 0x0000000000000038

Thread 0 "main" crashed:

0 MultiThreadedEventLoopGroup.init(threadInitializers:canBeShutDown:threadNamePrefix:metricsDelegate:selectorFactory:) + 203 in main at /home/duck/BrainServer/lib/BrainLib/deps/swift-nio/Sources/NIOPosix/MultiThreadedEventLoopGroup.swift:67:42

    65│ 
    66│     private let myGroupID: Int
    67│     private let index = ManagedAtomic<Int>(0)                                                                                                                                                                                                
      │                                          ▲
    68│     private var eventLoops: [SelectableEventLoop]
    69│     private let shutdownLock: NIOLock = NIOLock()

1 main + 28 in main at /home/duck/BrainServer/src/main.swift:11:13

     9│     // thread (each EventLoop) can support a large number of connections!
    10│ 
    11│ let group = MultiThreadedEventLoopGroup(numberOfThreads: 2)                                                                                                                                                                                  
      │             ▲
    12│ /*
    13│ let globalChatHandler = ServerChatRoomsHandler(chats: [Chat(messages: Chat.viewMessages)])

Atomics.ManagedAtomic.__allocating_init(τ_0_0) -> Atomics.ManagedAtomic<τ_0_0> (@Atomics.ManagedAtomic.__allocating_init(τ_0_0) -> Atomics.ManagedAtomic<τ_0_0>:8)
NIOPosix.MultiThreadedEventLoopGroup.init(threadInitializers: Swift.Array<(NIOPosix.NIOThread) -> ()>, canBeShutDown: Swift.Bool, threadNamePrefix: Swift.String, metricsDelegate: Swift.Optional<NIOPosix.NIOEventLoopMetricsDelegate>, selectorFactory: () throws -> NIOPosix.Selector<NIOPosix.NIORegistration>) -> NIOPosix.MultiThreadedEventLoopGroup (\home\duck\BrainServer\lib\BrainLib\deps\swift-nio\Sources\NIOPosix\MultiThreadedEventLoopGroup.swift:67)
NIOPosix.MultiThreadedEventLoopGroup.__allocating_init(threadInitializers: Swift.Array<(NIOPosix.NIOThread) -> ()>, canBeShutDown: Swift.Bool, threadNamePrefix: Swift.String, metricsDelegate: Swift.Optional<NIOPosix.NIOEventLoopMetricsDelegate>, selectorFactory: () throws -> NIOPosix.Selector<NIOPosix.NIORegistration>) -> NIOPosix.MultiThreadedEventLoopGroup (\home\duck\BrainServer\lib\BrainLib\deps\swift-nio\Sources\NIOPosix\MultiThreadedEventLoopGroup.swift:0)
NIOPosix.MultiThreadedEventLoopGroup.__allocating_init(numberOfThreads: Swift.Int, canBeShutDown: Swift.Bool, metricsDelegate: Swift.Optional<NIOPosix.NIOEventLoopMetricsDelegate>, selectorFactory: () throws -> NIOPosix.Selector<NIOPosix.NIORegistration>) -> NIOPosix.MultiThreadedEventLoopGroup (\home\duck\BrainServer\lib\BrainLib\deps\swift-nio\Sources\NIOPosix\MultiThreadedEventLoopGroup.swift:203)
NIOPosix.MultiThreadedEventLoopGroup.__allocating_init(numberOfThreads: Swift.Int) -> NIOPosix.MultiThreadedEventLoopGroup (\home\duck\BrainServer\lib\BrainLib\deps\swift-nio\Sources\NIOPosix\MultiThreadedEventLoopGroup.swift:140)
main (\home\duck\BrainServer\src\main.swift:11)
__libc_start_call_main (@__libc_start_call_main:29)
__libc_start_main_impl (@__libc_start_main@@GLIBC_2.34:43)
_start (@_start:15)

Steps to reproduce

Download BugReproduction.zip
run build.sh
run binary at build/src/main

May 24 '24 14:05 ThyOwen

Thanks for filing this. I'm not a CMake expert, @etcwilde knows it a lot better than me. Evan, do you have thoughts?

May 24 '24 15:05 Lukasa

FWIW, it may be worth looking at swift-certificates which has a working cmake build. It's also a good example of how we would want CMake support in NIO to function, with SwiftPM remaining the source of truth and the CMake build script being derived from it.

May 24 '24 15:05 Lukasa

Thank you for your response. There are some other projects with similar build issues and all of them use alternative build... whatevers with Atomics. #https://forums.swift.org/t/exc-bad-access-instantiating-atomic-primitives/65909/2 #https://github.com/apple/swift-atomics/issues/86?ref=https://coder.social #https://github.com/tuist/tuist/issues/5794

May 24 '24 21:05 ThyOwen

The error occurs because __swift_instantiateConcreteTypeFromMangledName returns a null address (0), which the program then attempts to use. While I'm not a Swift runtime internals expert, I believe this function reads Swift-specific metadata and creates an object from it. A return value of null likely indicates that the final binary is missing some critical metadata due to Swift atomics, as others have mentioned.

The reason the final binary might lack important metadata is that the linker isn't including it.

For example, on my Linux system, the linker invocation looks like this and it fails to run correctly:

"/usr/bin/ld.gold" \
  --hash-style=gnu \
  --eh-frame-hdr \
  -m \
  elf_x86_64 \
  -dynamic-linker \
  /lib64/ld-linux-x86-64.so.2 \
  -o \
  src/main \
  /usr/lib/gcc/x86_64-redhat-linux/14/../../../../lib64/Scrt1.o \
  ...
  -rpath \
  /usr/libexec/swift/5.10.1/lib/swift/linux \
  /usr/libexec/swift/5.10.1/lib/swift/linux/x86_64/swiftrt.o \
  src/CMakeFiles/main.dir/main.swift.o \
  /usr/libexec/swift/5.10.1/lib/swift/linux/x86_64/swiftrt.o \
  lib/BrainLib/deps/libNIO.a \
  lib/BrainLib/deps/libNIOEmbedded.a \
  ...
  -ldispatch \
  -lswiftDispatch \
  -lm \
  -lpthread \
  -lutil \
  -ldl \
  src/CMakeFiles/main.dir/main.swift.o \
  ...

To include all object files, I added the --whole-archive flag for library files. This linker invocation works fine, for instance:

"/usr/bin/ld.gold" \
  --hash-style=gnu \
  --eh-frame-hdr \
  -m \
  elf_x86_64 \
  -dynamic-linker \
  /lib64/ld-linux-x86-64.so.2 \
  -o \
  src/main \
  /usr/lib/gcc/x86_64-redhat-linux/14/../../../../lib64/Scrt1.o \
  ...
  -rpath \
  /usr/libexec/swift/5.10.1/lib/swift/linux \
  /usr/libexec/swift/5.10.1/lib/swift/linux/x86_64/swiftrt.o \
  src/CMakeFiles/main.dir/main.swift.o \
  /usr/libexec/swift/5.10.1/lib/swift/linux/x86_64/swiftrt.o \
  -z muldefs --whole-archive --start-group \
  lib/BrainLib/deps/libNIO.a \
  lib/BrainLib/deps/libNIOEmbedded.a \
  ...
  -ldispatch \
  -lswiftDispatch \
  --end-group --no-whole-archive \
  -lm \
  -lpthread \
  -lutil \
  -ldl \
  src/CMakeFiles/main.dir/main.swift.o \
  ...

However, I'm still unsure why some symbols are missing. To diagnose the issue effectively, I recommend testing with a minimal set of libraries. (I tested using my project, https://github.com/bc-lee/test-bazel-swift-atomics to ensure Bazel compatibility with Swift atomics.) I hope you can fix your CMake project to work. Good luck!

Jul 10 '24 12:07 bc-lee