Crash in Julia 1.5 and later

[fingolfin]

(This comes from comments on https://github.com/JuliaPackaging/Yggdrasil/issues/2160 but I think it deserves its own full issue):

There are at least two issues with libcxxwrap_julia_jll v0.8.5 in Julia 1.5 and later

1. It fails to load on macOS with error Symbol not found: __Unwind_Resume ; this is addressed in https://github.com/JuliaPackaging/Yggdrasil/pull/2190 ; see also https://github.com/JuliaPackaging/Yggdrasil/pull/2199 and

2. On Linux there is a segfault; this segfault can also be seen on this repo in the nightly CI tests, which fail with the same error (the CI tests here only test with Julia 1.4 and nightly, it might be useful to also test 1.5 there?). This is a backtrace:

...
Running tests from containers.jl...

signal (11): Segmentation fault
in expression starting at /home/mhorn/.julia/packages/CxxWrap/ZOkSN/test/containers.jl:21
_ZN5jlcxx6detail11CallFunctorINS_10ConstArrayIdLl1EEEJEE5applyEPKv at /home/mhorn/.julia/artifacts/860a8b2216bd059600ed7c44cdaa3bb81b23ff1c/lib/libjlcxx_containers.so (unknown line)
const_vector at /home/mhorn/.julia/packages/CxxWrap/ZOkSN/src/CxxWrap.jl:590
macro expansion at /home/mhorn/.julia/packages/CxxWrap/ZOkSN/test/containers.jl:29 [inlined]
macro expansion at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.5/Test/src/Test.jl:1115 [inlined]
top-level scope at /home/mhorn/.julia/packages/CxxWrap/ZOkSN/test/containers.jl:23
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:834
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:913
jl_load_rewrite at /buildworker/worker/package_linux64/build/src/toplevel.c:914
include at ./client.jl:457
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:117
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:206
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:157 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:566
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:492
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:492
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:660
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:840
jl_parse_eval_all at /buildworker/worker/package_linux64/build/src/ast.c:913
jl_load_rewrite at /buildworker/worker/package_linux64/build/src/toplevel.c:914
include at ./client.jl:457
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2231 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/src/julia.h:1690 [inlined]
do_call at /buildworker/worker/package_linux64/build/src/interpreter.c:117
eval_value at /buildworker/worker/package_linux64/build/src/interpreter.c:206
eval_stmt_value at /buildworker/worker/package_linux64/build/src/interpreter.c:157 [inlined]
eval_body at /buildworker/worker/package_linux64/build/src/interpreter.c:566
jl_interpret_toplevel_thunk at /buildworker/worker/package_linux64/build/src/interpreter.c:660
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:840
jl_toplevel_eval_flex at /buildworker/worker/package_linux64/build/src/toplevel.c:790
jl_toplevel_eval_in at /buildworker/worker/package_linux64/build/src/toplevel.c:883
eval at ./boot.jl:331
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
exec_options at ./client.jl:272
_start at ./client.jl:506
jfptr__start_53898.clone_1 at /home/mhorn/julia-1.5.3/lib/julia/sys.so (unknown line)
_jl_invoke at /buildworker/worker/package_linux64/build/src/gf.c:2214 [inlined]
jl_apply_generic at /buildworker/worker/package_linux64/build/src/gf.c:2398
jl_apply at /buildworker/worker/package_linux64/build/ui/../src/julia.h:1690 [inlined]
true_main at /buildworker/worker/package_linux64/build/ui/repl.c:106
main at /buildworker/worker/package_linux64/build/ui/repl.c:227
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at /home/mhorn/julia-1.5.3/bin/julia (unknown line)
Allocations: 16112006 (Pool: 16107521; Big: 4485); GC: 15

Applying c++filt to _ZN5jlcxx6detail11CallFunctorINS_10ConstArrayIdLl1EEEJEE5applyEPKv gives jlcxx::detail::CallFunctor<jlcxx::ConstArray<double, 1l>>::apply(void const*).

It might be helpful to attack this with rr, but my early attempts failed due to a lack of instructions, and then I had to work on other stuff.

[benlorenz]

I did some debugging but I still don't know why that happens but it does offer some workarounds, i.e. setting -DCMAKE_CXX_FLAGS_RELEASE="-O2".

I tried several configurations to get a better backtrace and could reproduce that crash only in some rather specific cases:

• using the existing 0.8.5+0 binaries: crashes
• building libcxxwrap_julia manually (with my own gcc 9): works
• building libcxxwrap_julia with binarybuilder (uses gcc 7): crashes
• setting preferred_gcc_version = 8: works
• setting preferred_gcc_version = 9: works
• setting preferred_gcc_version = 7 + cmake target Debug: works
• setting preferred_gcc_version = 7 + cmake target Debug + -O2: works
• setting preferred_gcc_version = 7 + cmake target Debug + -O3: crashes
• setting preferred_gcc_version = 7 + -O2: works

And with that -O3 Debug option I got a slightly better backtrace:

#0  0x00007fffda2768cf in jlcxx::ConvertToJulia<jlcxx::ConstArray<double, 1l>, jlcxx::ConstArrayTrait>::operator() (arr=..., this=<optimized out>)
    at /workspace/srcdir/libcxxwrap-julia/include/jlcxx/const_array.hpp:95
#1  jlcxx::convert_to_julia<jlcxx::ConstArray<double, 1l> > (cpp_val=...) at /workspace/srcdir/libcxxwrap-julia/include/jlcxx/type_conversion.hpp:745
#2  jlcxx::detail::ReturnTypeAdapter<jlcxx::ConstArray<double, 1l>>::operator()(void const*) (this=<optimized out>, functor=<optimized out>)
    at /workspace/srcdir/libcxxwrap-julia/include/jlcxx/module.hpp:47
#3  jlcxx::detail::CallFunctor<jlcxx::ConstArray<double, 1l>>::apply(void const*) (functor=<optimized out>) at /workspace/srcdir/libcxxwrap-julia/include/jlcxx/module.hpp:72

const_array.hpp:95 is a JL_GC_POP() call but that it only happens with -O3 points to something rather annoying to debug.

So far for today, maybe someone else wants to have a look again.

[fingolfin]

@benlorenz thanks for that, that helps a lot. So perhaps we can just rebuild this with GCC 8 or 9. It may "just" be a compiler bug, after all.

[fingolfin]

I made https://github.com/JuliaPackaging/Yggdrasil/pull/2236 let's see if that helps

[fingolfin]

That indeed seems to have fixed it, great! Now I guess the CI in this repository should be switched to use GCC 8+, too?

[barche]

Alright, great news that it works with the newer GCC, but I'm still quite nervous that this is a bug in libcxxwrap-julia. We'll see if it resurfaces elsewhere somehow, for now this seems like a good solution, thanks for the help!