macOS: System image file fails consistency check on binary startup

[ghyatzo]

Upon creating an app, the process goes on smoothly, untill it is time to actually call the binary. I get the error: ERROR: System image file failed consistency check: maybe opened the wrong version?

Julia Version 1.8.2
Commit 36034abf260 (2022-09-29 15:21 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.3.0)
  CPU: 10 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 8 on 8 virtual cores
Environment:
  JULIA_NUM_THREADS = 8
  JULIA_DEPOT_PATH = /Users/cshen/.local/julia

I installed julia through juliaup. the juliaup folder is in ~/.local/juliaup and the julia versions in ~/.local/julia/juliaup. the binaries in ~/.local/juliaup/bin are symlinked to ~/.local/bin which is in the PATH. I also have julia aliased to julia --project=@.

Could it be that the symlinking throws some pathing off somewhere?

[FerreolS]

Similar issue here on

Julia Version 1.8.4
Commit 00177ebc4fc (2022-12-23 21:32 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin21.4.0)
  CPU: 16 × Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, skylake)
  Threads: 16 on 16 virtual cores
Environment:
  JULIA_NUM_THREADS = auto

the app execute smoothly when it is in the folder where it has been compiled but fail with ERROR: System image file failed consistency check: maybe opened the wrong version? when relocated anywhere else. Using the example of the package

> Code/Julia/PackageCompiler.jl/examples/MyAppCompiled/bin/MyApp 
  Downloaded artifact: MKL
ARGS = String[]
Base.PROGRAM_FILE = "Code/Julia/PackageCompiler.jl/examples/MyAppCompiled/bin/MyApp"
...

> cp -r Code/Julia/PackageCompiler.jl/examples/MyAppCompiled .
> MyAppCompiled/bin/MyApp                                     
ERROR: System image file failed consistency check: maybe opened the wrong version?

[PhilReinhold]

I'm running into this issue as well, just trying to build any sysimage, e.g.

PackageCompiler.create_sysimage(String[], sysimage_path="test_image.so")

then

julia --sysimage=test_image.so gives the same error as above.

Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 12 × Apple M2 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 1 on 8 virtual cores
Environment:
  JULIA_PKG_USE_CLI_GIT = true

[kevmoor]

Same issue, tried on Julia 1.9.3, 1.9.2, 1.10.0-beta2, all with PackageCompiler version 2.1.9, also tried on Julia 1.9.2 with PackageCompiler version 2.1.7 (the mix I had when I last had it running a few weeks ago), this versioninfo() below is from this latest combination.

Julia Version 1.9.2
Commit e4ee485e909 (2023-07-05 09:39 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin22.4.0)
  CPU: 10 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1)
  Threads: 1 on 8 virtual cores
Environment:
  JULIA_PKG_USE_CLI_GIT = true

[KwatMDPhD]

I'm seeing the same error when building https://github.com/KwatMDPhD/GSEA.jl/commit/79364ca00160a034f8afe1387df6107adfbdbc1a

https://github.com/comonicon/Comonicon.jl/issues/264

[sloede]

I'm seeing the same error when building KwatMDPhD/GSEA.jl@79364ca

comonicon/Comonicon.jl#264

This happens on macOS only?

[KwatMDPhD]

I have only tried on macOS Sonoma 14.0. If you have a non-mac machine, you can clone the above commit and try julia --project deps/build.lj app.

[jayscook]

I'm still getting the error on PackageCompiler 2.1.11 (which specifically addresses this issue) on macOS Ventura (13.6). Running Julia 1.8.5, not sure if 1.9 is required for the fix.

[sloede]

Thanks for reporting. Just to clarify: There is no fix available as of yet (also not in PC v2.1.11)

[jayscook]

Thanks for reporting. Just to clarify: There is no fix available as of yet (also not in PC v2.1.11)

Thanks for clarifying! I assumed that the closed issue indicated that it was fixed.

Please let me know if I can contribute to a fix in some way.

[sloede]

Please let me know if I can contribute to a fix in some way.

At the moment we are lacking an understanding of why the consistency check fails. If you manage to find out about this, we could start looking for a remedy 🙂

[KwatMDPhD]

I'm using 1.9.3

Would be great if this error can be fixed...

[ctarn]

I was able to build apps using Julia 1.9.0, PackageCompiler 2.1.7, and macOS 13. Recently I upgraded Julia to 1.9.3, PackageCompiler to 2.1.2, and macOS to 14, and the error occurred.

I tried to restore Julia to 1.9.0 and PackageCompiler to 2.1.7, and the error still occurs. So I guess macOS should cause it since Xcode Command Line Tools was also upgraded to 15, which is required by PackageCompiler to compile executables.

Unfortunately, when I tried to downgrade Xcode Command Line Tools, it warned that older version Xcode C. L. T. can not be installed on macOS 14... And I can not downgrade the OS... :(

I would recommend anyone (@jayscook ?) still using macOS 13 try to install an older version (v14.3?)of Xcode. It can be downloaded from: https://developer.apple.com/download/all/

[bjarthur]

i can't build sysimages on apple silicon either. please let me know how i can help fix it.

[hycakir]

I can confirm it is related to clang version. I have compiled MyLib with gcc from homebrew and it works.

[ctarn]

it really works! but remember to set the compiler to gcc-13 instead of just gcc since gcc is just an alias of clang by default.

you can specify the compiler by running an export in your shell:

export JULIA_CC="gcc-13"

[sloede]

Great catch! Could you please report which version of clang/gcc you are each using by default on your systems that works (using gcc --version), which version did not work, and the output of your Julia versioninfo()?

[ctarn]

clang v15 from Xcode CLT doesn't work

$ gcc --version
Apple clang version 15.0.0 (clang-1500.0.40.1)
Target: x86_64-apple-darwin23.1.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

(clang v14 should work, but older version Xcode CLT can not be installed on latest macOS)

gcc v13 from brew works

$ gcc-13 --version
gcc-13 (Homebrew GCC 13.2.0) 13.2.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

versioninfo

julia> versioninfo()
Julia Version 1.9.3
Commit bed2cd540a1 (2023-08-24 14:43 UTC)
Build Info:
  Official https://julialang.org/ release
Platform Info:
  OS: macOS (x86_64-apple-darwin22.4.0)
  CPU: 16 × Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-14.0.6 (ORCJIT, skylake)
  Threads: 1 on 16 virtual cores
Environment:
  JULIA_CC = gcc-13

[jayscook]

Good catch!

I've installed gcc-13 with brew install gcc@13 and set it as an env var but am still getting the error after restarting the shell. Any thoughts on what I'm missing here?

$ gcc-13 --version
gcc-13 (Homebrew GCC 13.2.0) 13.2.0
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

julia> versioninfo()
Julia Version 1.8.5
Commit 17cfb8e65ea (2023-01-08 06:45 UTC)
Platform Info:
  OS: macOS (arm64-apple-darwin21.5.0)
  CPU: 10 × Apple M1 Pro
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, apple-m1)
  Threads: 1 on 8 virtual cores
Environment:
  JULIA_CC = gcc-13

[ctarn]

Since the versions of Julia and LLVM are different, maybe you should try older versions of gcc or upgrade Julia to 1.9.

[sloede]

I've installed gcc-13 with brew install gcc@13 and set it as an env var but am still getting the error after restarting the shell. Any thoughts on what I'm missing here?

Hm, this is unfortunate. It also means we're back to square one, since I don't understand now (again) what the necessary conditions are to trigger the error.

[hycakir]

I've installed gcc-13 with brew install gcc@13 and set it as an env var but am still getting the error after restarting the shell. Any thoughts on what I'm missing here?

julia> versioninfo()
Julia Version 1.8.5
...
  JULIA_CC = gcc-13

This is expected as per https://github.com/JuliaLang/julia/issues/49581. You should pick an earlier version of GCC/Clang or newer version of Julia.

[sloede]

@hycakir Thanks for letting us know, I didn't know about this!

@jayscook Could you please confirm that it works with gcc-12? If yes, we might want to add a note about this behavior to the docs.

[KwatMDPhD]

How can we make this work with the latest mac gcc? I would be nice to not have to downgrade.

[sloede]

How can we make this work with the latest mac gcc? I would be nice to not have to downgrade.

TBH, I do not know, since I neither know the rationale behind this check nor do I understand what it actually checks 😅 It seems this check was introduced in https://github.com/JuliaLang/julia/commit/4e7ec790b587f4fe26ce7620c83d3e3197b3c871, specifically here.

@vtjnash can you maybe shed some light on this and help us understand

• what exactly causes this error to appear when loading sysimages built with clang/gcc versions that are "too new" in some sense,
• if there is anything we can do about it on PC side, and if not
• what could be done on the Julia side to remedy this issue for building custom sysimages?

[vtjnash]

It simply checks if the symbol jl_RTLD_DEFAULT_handle_pointer exists, which is something added by codegen to the image, and that it linked to jl_RTLD_DEFAULT_handle. You may want to try setting LD_DEBUG to find out why this did not generate a completed dylib for you.

[sloede]

It simply checks if the symbol jl_RTLD_DEFAULT_handle_pointer exists, which is something added by codegen to the image, and that it linked to jl_RTLD_DEFAULT_handle. You may want to try setting LD_DEBUG to find out why this did not generate a completed dylib for you.

Thanks for the explanation! So you mean we should set LD_DEBUG when calling Julia with the broken sysimage? Can you tell us which value should we set the variable to? Also, this error does only seem to occur on macOS - there's no LD_DEBUG available IIRC, maybe you have an alternative?

Sorry for the many questions, we just don't have any enough experience to tackle this at the moment.

[vtjnash]

See man dyld for the available env options

[sloede]

I've been trying to reproduce the problem on my M1 Macbook, but without success so far - all my sysimages or apps I build seem to work.

@KwatMDPhD @bjarthur @hycakir @ctarn Can someone who had this problem please provide an MWE with a sysimage/app as small as possible and the exact commands you use to build it? In addition, please also post the output of gcc --version and Julia's versioninfo() such that we understand which environment is used exactly.

[sloede]

See man dyld for the available env options

Thanks! One more question: At the moment I seem to fail to reproduce the error on my system for both sysimages and apps. This is weird, because when using nm on any of the libjulia{"",-codegen,-internal}.dylib libraries, neither has the symbol jl_RTLD_DEFAULT_handle_pointer, and only libjulia-internal.dylib has jl_RTLD_DEFAULT_handle.

Where should this jl_RTLD_DEFAULT_handle_pointer be located and do you have an idea why it doesn't throw an error on my system although it does not exist?

[ctarn]

@sloede please see https://github.com/ctarn/MiniApp.jl/releases/tag/PC738