signal (6) if numba is imported after julia

[ivirshup]

If numba is imported after a PyJulia session is started, the process fails and exits. This does not occur if numba is imported first. Example:

Python 3.7.3 (default, Mar 27 2019, 09:23:15) 
[Clang 10.0.1 (clang-1001.0.46.3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from julia import Main
>>> import numba
Assertion failed: (PassInf && "Expected all immutable passes to be initialized"), function addImmutablePass, file /Users/buildbot/miniconda3/conda-bld/llvmdev_1556270736866/work/lib/IR/LegacyPassManager.cpp, line 849.

signal (6): Abort trap: 6
in expression starting at no file:0
__pthread_kill at /usr/lib/system/libsystem_kernel.dylib (unknown line)
Allocations: 9604414 (Pool: 9602251; Big: 2163); GC: 20
[1]    93402 abort      python3

I'm using julia v1.1.1, PyJulia v0.4.1, and numba v0.44.1.

[tkf]

Maybe you can fix it by loading some of numba's C extensions using RTLD_DEEPBIND before importing numba and PyJulia.

See: https://github.com/JuliaLang/julia/issues/12644#issuecomment-314546572 https://github.com/JuliaPy/pyjulia/issues/301

[ivirshup]

Sorry for the late response, I must have missed the notification.

Swapping the order of the imports seems like an easier work-around to me. Would you expect any bad behavior from this?

[tkf]

Swapping the imports may be an OK workaround. But I'd say I'd be nervous to use PyJulia and Numba if I can invoke a segfault by swapping the imports. It seems due to LLVM compatibility. So something in Julia could be messed up by loading Numba first.

I wonder if doing https://github.com/JuliaLang/julia/issues/12644#issuecomment-314546572 for libjulia could help. Maybe try something like the following?

import ctypes
import os
from julia.api import JuliaInfo
juliainfo = JuliaInfo.load()
ctypes.PyDLL(juliainfo.libjulia_path, ctypes.RTLD_GLOBAL | os.RTLD_DEEPBIND)

from julia import Main
import numba

[ihnorton]

Seems like pyjulia and/or PyCall could also try using dlmopen (with RTLD_DEEPBIND) to isolate libraries to a single namespace (at least on GLIBC platforms). See

• http://man7.org/linux/man-pages/man3/dlopen.3.html#NOTES
• example w/ multiple python interpreters in single process: https://gist.github.com/dutc/eba9b2f7980f400f6287

glibc dlmopen has (or at least has in widely-available versions) a limitation that libraries within a namespace cannot reference libraries previously loaded with RTLD_GLOBAL, but this seems like a desirable property for interop use-cases (see https://patchwork.ozlabs.org/patch/496559/).

[tkf]

Thanks, I didn't know dlmopen. But, IIUC, we can't use this approach. This is because PyCall running inside Julia (libjulia) needs to access Python namespace. That is to say, ccall(:PyObject_Call, ...) in PyCall has to refer to the symbol in the global main namespace. If we open libjulia with dlmopen, I suppose this is not possible?

[ihnorton]

dlmopen still returns a handle, so you can use dlsym against that handle (or pass it as the 2nd argument to ccall)

[ihnorton]

I guess it wouldn't help with the original issue posted here, because macOS doesn't support dlmopen (though it does have a similar feature). Might be worth trying on linux if that is still a problem.

[tkf]

I see, thanks. I think it's doable but there are a few challenges:

• PyCall does not use ccall with explicit handle. It's either ccall(:PyObject_Call, ...) or ccall((:PyObject_Call, static_path), ...). I guess we can change it to use dlsym (possibly with caching), provided that there is no loss in performance.

• Passing the handle to PyCall. PyCall needs to know if it should dlopen libpython or not inside __init__. It's probably doable with a special environment variable. Another possibility is to add (say) LibPython.jl as a dependency of PyCall which lets you initialize Python runtime in a configurable manner (including passing an existing handle).