Segfault when calling `dlopen` on certain Qt frameworks

[carlocab]

I have Qt installed via Homebrew on macOS 12 (x86_64), and running the following script causes a segmentation fault (ruby 2.6.8p205 (2021-07-07 revision 67951) [x86_64-darwin21] installed via rbenv, Fiddle 1.1.0 installed via gem):

require 'fiddle'

dylibs = [
  '/usr/local/lib/Qt3DCore.framework/Qt3DCore',
  '/usr/local/lib/Qt3DQuickAnimation.framework/Qt3DQuickAnimation',
  '/usr/local/lib/Qt3DQuickExtras.framework/Qt3DQuickExtras'
]

dylibs.each do |dylib|
  puts dylib
  Fiddle.dlopen(dylib).close
end

Running ruby crash.rb produces the following output (truncated for brevity):

/usr/local/lib/Qt3DCore.framework/Qt3DCore
/usr/local/lib/Qt3DQuickAnimation.framework/Qt3DQuickAnimation
/usr/local/lib/Qt3DQuickExtras.framework/Qt3DQuickExtras
/Users/carlocab/.rbenv/versions/2.6.8/lib/ruby/gems/2.6.0/gems/fiddle-1.1.0/lib/fiddle.rb:61: [BUG] Segmentation fault at 0x000000010f18945c
ruby 2.6.8p205 (2021-07-07 revision 67951) [x86_64-darwin21]

-- Crash Report log information --------------------------------------------
   See Crash Report log file under the one of following:
     * ~/Library/Logs/DiagnosticReports
     * /Library/Logs/DiagnosticReports
   for more details.
Don't forget to include the above Crash Report log file in bug reports.

-- Control frame information -----------------------------------------------
c:0007 p:---- s:0028 e:000027 CFUNC  :initialize
c:0006 p:---- s:0025 e:000024 CFUNC  :new
c:0005 p:0014 s:0020 e:000019 METHOD /Users/carlocab/.rbenv/versions/2.6.8/lib/ruby/gems/2.6.0/gems/fiddle-1.1.0/lib/fiddle.rb:61
c:0004 p:0020 s:0015 e:000014 BLOCK  crash.rb:11 [FINISH]
c:0003 p:---- s:0011 e:000010 CFUNC  :each
c:0002 p:0023 s:0007 E:001288 EVAL   crash.rb:9 [FINISH]
c:0001 p:0000 s:0003 E:001650 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
crash.rb:9:in `<main>'
crash.rb:9:in `each'
crash.rb:11:in `block in <main>'
/Users/carlocab/.rbenv/versions/2.6.8/lib/ruby/gems/2.6.0/gems/fiddle-1.1.0/lib/fiddle.rb:61:in `dlopen'
/Users/carlocab/.rbenv/versions/2.6.8/lib/ruby/gems/2.6.0/gems/fiddle-1.1.0/lib/fiddle.rb:61:in `new'
/Users/carlocab/.rbenv/versions/2.6.8/lib/ruby/gems/2.6.0/gems/fiddle-1.1.0/lib/fiddle.rb:61:in `initialize'

-- Machine register context ------------------------------------------------
 rax: 0x000000010f189450 rbx: 0x0000000000000000 rcx: 0x0000000000000120
 rdx: 0x0000000000000000 rdi: 0x00006000034343b0 rsi: 0x00006000034065b0
 rbp: 0x00007ff7b2096b30 rsp: 0x00007ff7b2096ae0  r8: 0x0000000000000065
  r9: 0x000000010fdd2556 r10: 0x0000000100000600 r11: 0x0000000000000000
 r12: 0x000000010f1ef450 r13: 0x000000010f1be298 r14: 0x00007ff7b2096b48
 r15: 0x000000010fe450c0 rip: 0x000000010fa9ddf7 rfl: 0x0000000000010206

-- C level backtrace information -------------------------------------------
/Users/carlocab/.rbenv/versions/2.6.8/lib/libruby.2.6.dylib(rb_vm_bugreport+0x82) [0x10e562652]
/Users/carlocab/.rbenv/versions/2.6.8/lib/libruby.2.6.dylib(rb_bug_context+0x1d6) [0x10e3bb906]
/Users/carlocab/.rbenv/versions/2.6.8/lib/libruby.2.6.dylib(sigsegv+0x51) [0x10e4c9511]
/usr/lib/system/libsystem_platform.dylib(_sigtramp+0x1d) [0x7ff811cb5e2d]
/usr/local/Cellar/qt/6.2.1/lib/QtCore.framework/Versions/A/QtCore(_ZNK9QMetaType8idHelperEv+0xb7) [0x10fa9ddf7]
/usr/local/Cellar/qt/6.2.1/lib/Qt3DQuick.framework/Versions/A/Qt3DQuick(_GLOBAL__sub_I_quick3dbuffer.cpp+0x26) [0x10f1cd8b6]

The full output can be found here: output.txt The log referenced in the error message: crash.log.tar.gz

I can only produce the segfault when the dylibs array is dlopened in the order given. Dropping one of them or reversing the array allows the script to run correctly.

As a sanity check, I wrote what I think should be the equivalent program in C:

#include <dlfcn.h>
#include <stdio.h>

int main()
{
    char dylibs[][65] = {
        "/usr/local/lib/Qt3DCore.framework/Qt3DCore",
        "/usr/local/lib/Qt3DQuickAnimation.framework/Qt3DQuickAnimation",
        "/usr/local/lib/Qt3DQuickExtras.framework/Qt3DQuickExtras",
    };

    for (int i = 0; i < 3; ++i) {
        printf("%s\n", dylibs[i]);
        void *handle;
        handle = dlopen(dylibs[i], RTLD_LAZY | RTLD_GLOBAL);
        if (handle == NULL) {
            printf("failed to load %s\n", dylibs[i]);
            return 1;
        }
        dlclose(handle);
    }

    return 0;
}

This runs without any problems:

❯ make crash && ./crash; echo $?
cc     crash.c   -o crash
/usr/local/lib/Qt3DCore.framework/Qt3DCore
/usr/local/lib/Qt3DQuickAnimation.framework/Qt3DQuickAnimation
/usr/local/lib/Qt3DQuickExtras.framework/Qt3DQuickExtras
0

[kou]

I tried this with Ruby installed by Homebrew but I couldn't reproduce this:

$ $(brew --prefix ruby)/bin/ruby -v /tmp/a.rb
ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-darwin20]
/usr/local/lib/Qt3DCore.framework/Qt3DCore
/usr/local/lib/Qt3DQuickAnimation.framework/Qt3DQuickAnimation
/usr/local/lib/Qt3DQuickExtras.framework/Qt3DQuickExtras

Could you also try with Ruby installed by Homebrew not rbenv?

[carlocab]

This seems to affect some builds of Ruby 2.6 but is fixed in Ruby 3.0.

❯ ruby -v crash.rb
ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-darwin21]
/usr/local/lib/Qt3DCore.framework/Qt3DCore
/usr/local/lib/Qt3DQuickAnimation.framework/Qt3DQuickAnimation
/usr/local/lib/Qt3DQuickExtras.framework/Qt3DQuickExtras

I get the above output for Homebrew Ruby 3.0 and Ruby 3.0 installed via rbenv.

This doesn't affect Homebrew Ruby 2.6:

❯ /usr/local/opt/[email protected]/bin/ruby -v crash.rb
ruby 2.6.8p205 (2021-07-07 revision 67951) [x86_64-darwin21]
/usr/local/Cellar/[email protected]/2.6.8/lib/ruby/2.6.0/rubygems/defaults/operating_system.rb:64: warning: method redefined; discarding old default_specifications_dir
/usr/local/Cellar/[email protected]/2.6.8/lib/ruby/2.6.0/rubygems/basic_specification.rb:37: warning: previous definition of default_specifications_dir was here
/usr/local/lib/Qt3DCore.framework/Qt3DCore
/usr/local/lib/Qt3DQuickAnimation.framework/Qt3DQuickAnimation
/usr/local/lib/Qt3DQuickExtras.framework/Qt3DQuickExtras

It does affect the Ruby provided with macOS, which comes with Fiddle.

[kou]

Thanks for confirming this.

I can also confirmed that the system Ruby crashes:

$ ruby -v -r fiddle /tmp/a.rb
ruby 2.6.3p62 (2019-04-16 revision 67580) [universal.x86_64-darwin20]
/usr/local/lib/Qt3DCore.framework/Qt3DCore
/usr/local/lib/Qt3DQuickAnimation.framework/Qt3DQuickAnimation
/usr/local/lib/Qt3DQuickExtras.framework/Qt3DQuickExtras
dyld: Assertion failed: (0 && "LoadedImage not found by num"), function findLoadedImage, file /System/Volumes/Data/SWE/macOS/BuildRoots/d7e177bcf5/Library/Caches/com.apple.xbs/Sources/libdyld/dyld-852.2/dyld3/ClosureBuilder.cpp, line 713.

[1]    25767 abort      ruby -v -r fiddle /tmp/a.rb

I think that system trace log (strace on Linux) will be useful to confirm difference with Ruby 3.0 and Ruby 2.6. It seems that we can use dtruss for it on macOS (I'm not familiar with macOS) but I couldn't get it...:

$ sudo dtruss ruby /tmp/a.rb
dtrace: system integrity protection is on, some features will not be available

dtrace: failed to execute ruby: (os/kern) failure

Could you get system trace logs for Ruby 2.6 and Ruby 3.0?

[kou]

Stale.