pyjulia How to print out error information when debugging PyJulia?

There seems to be some incompatibilities between PyJulia and other packages, and I would like to do some debugging:

first do import <another_package> then

import julia
j = julia.Julia()

would crash the Python kernel (in Jupyter notebook) or give "Segmentation Fault" (in terminal); on the other hand, first do

import julia
j = julia.Julia()

then import <another_package> gives no error.

I suspect the package might interfere with the communication between Python and Julia, but the error message isn't printed out to Python so that I can't debug. Any idea how to print out the error messages, or hints on why the error occurs? Thanks!

May 28 '19 01:05 chengrunyang

This is likely due to dynamically loaded external libraries like libstdc++. Essentially, initializing PyJulia as early as possible is the safest way to work with other Python packages at the moment.

https://pyjulia.readthedocs.io/en/latest/troubleshooting.html#error-due-to-libstdc-version
https://github.com/JuliaPy/pyjulia/issues/180
https://github.com/JuliaPy/pyjulia/issues/223

how to print out the error messages

"Segmentation Fault" seems to be a reasonable error message here. If you want to debug more, you can start Python inside gdb.

May 28 '19 04:05 tkf

Thanks! The workaround of initializing PyJulia as early as possible works in Jupyter notebook on Mac OS X. However, in Linux command line, this still doesn't work even if I first initialize PyJulia and then the Python package it has conflict with, with the error:

signal (11): Segmentation fault
while loading no file, in expression starting on line 0
_ZN4llvm13StringMapImpl15LookupBucketForENS_9StringRefE at /opt/julia_files/julia-903644385b/bin/../lib/julia/libLLVM-3.9.so (unknown line)
unknown function (ip: 0x7f6e228438e0)
unknown function (ip: 0x7f6e22843a83)
_ZN4llvm2cl6Option11addArgumentEv at /opt/julia_files/julia-903644385b/bin/../lib/julia/libLLVM-3.9.so (unknown line)
unknown function (ip: 0x7f6dee8febad)
unknown function (ip: 0x7f6e2e6ca732)
unknown function (ip: 0x7f6e2e6cf1fe)
_dl_catch_exception at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f6e2e6ce7c9)
unknown function (ip: 0x7f6e2dea6f95)
_dl_catch_exception at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_dl_catch_error at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
unknown function (ip: 0x7f6e2dea7734)
dlopen at /lib/x86_64-linux-gnu/libdl.so.2 (unknown line)
_PyImport_FindSharedFuncptr at python3 (unknown line)
_PyImport_LoadDynamicModuleWithSpec at python3 (unknown line)
unknown function (ip: 0x5fb767)
PyCFunction_Call at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x50253f)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
_PyFunction_FastCallDict at python3 (unknown line)
_PyObject_FastCallDict at python3 (unknown line)
_PyObject_CallMethodIdObjArgs at python3 (unknown line)
PyImport_ImportModuleLevelObject at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x511d77)
PyCFunction_Call at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x50253f)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
_PyFunction_FastCallDict at python3 (unknown line)
_PyObject_FastCallDict at python3 (unknown line)
_PyObject_CallMethodIdObjArgs at python3 (unknown line)
PyImport_ImportModuleLevelObject at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x511d77)
PyCFunction_Call at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x50253f)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
_PyFunction_FastCallDict at python3 (unknown line)
_PyObject_FastCallDict at python3 (unknown line)
_PyObject_CallMethodIdObjArgs at python3 (unknown line)
PyImport_ImportModuleLevelObject at python3 (unknown line)
unknown function (ip: 0x510dc3)
PyCFunction_Call at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x50253f)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
_PyFunction_FastCallDict at python3 (unknown line)
_PyObject_FastCallDict at python3 (unknown line)
_PyObject_CallMethodIdObjArgs at python3 (unknown line)
PyImport_ImportModuleLevelObject at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x511d77)
PyCFunction_Call at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x50253f)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
_PyFunction_FastCallDict at python3 (unknown line)
_PyObject_FastCallDict at python3 (unknown line)
_PyObject_CallMethodIdObjArgs at python3 (unknown line)
PyImport_ImportModuleLevelObject at python3 (unknown line)
unknown function (ip: 0x510dc3)
PyCFunction_Call at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x50253f)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
_PyFunction_FastCallDict at python3 (unknown line)
_PyObject_FastCallDict at python3 (unknown line)
_PyObject_CallMethodIdObjArgs at python3 (unknown line)
PyImport_ImportModuleLevelObject at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x511d77)
PyCFunction_Call at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x50253f)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
_PyFunction_FastCallDict at python3 (unknown line)
_PyObject_FastCallDict at python3 (unknown line)
_PyObject_CallMethodIdObjArgs at python3 (unknown line)
PyImport_ImportModuleLevelObject at python3 (unknown line)
unknown function (ip: 0x510dc3)
PyCFunction_Call at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x50253f)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
_PyFunction_FastCallDict at python3 (unknown line)
_PyObject_FastCallDict at python3 (unknown line)
_PyObject_CallMethodIdObjArgs at python3 (unknown line)
PyImport_ImportModuleLevelObject at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x511d77)
PyCFunction_Call at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x50253f)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
_PyFunction_FastCallDict at python3 (unknown line)
_PyObject_FastCallDict at python3 (unknown line)
_PyObject_CallMethodIdObjArgs at python3 (unknown line)
PyImport_ImportModuleLevelObject at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x511d77)
PyCFunction_Call at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x50253f)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x502208)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
_PyFunction_FastCallDict at python3 (unknown line)
_PyObject_FastCallDict at python3 (unknown line)
_PyObject_CallMethodIdObjArgs at python3 (unknown line)
PyImport_ImportModuleLevelObject at python3 (unknown line)
unknown function (ip: 0x510dc3)
PyCFunction_Call at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
unknown function (ip: 0x50253f)
unknown function (ip: 0x502f3c)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
_PyFunction_FastCallDict at python3 (unknown line)
_PyObject_FastCallDict at python3 (unknown line)
_PyObject_CallMethodIdObjArgs at python3 (unknown line)
PyImport_ImportModuleLevelObject at python3 (unknown line)
_PyEval_EvalFrameDefault at python3 (unknown line)
unknown function (ip: 0x504c27)
PyEval_EvalCode at python3 (unknown line)
unknown function (ip: 0x634d51)
unknown function (ip: 0x4a38c4)
PyRun_InteractiveLoopFlags at python3 (unknown line)
PyRun_AnyFileExFlags at python3 (unknown line)
Py_Main at python3 (unknown line)
main at python3 (unknown line)
__libc_start_main at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
_start at python3 (unknown line)
Allocations: 3299862 (Pool: 3298544; Big: 1318); GC: 5
Segmentation fault

I guess this is because of the conflict between PyJulia and libLLVM (instead of libstdc++). Is there a similar solution for this? Thanks!

May 30 '19 23:05 chengrunyang

The stack trace says julia-903644385b. Are you using Julia 0.6 or something? Is it the same version you are using on macOS?

Aside: Maybe you can use this trick: https://github.com/JuliaLang/julia/issues/12644#issuecomment-314546572 https://github.com/JuliaPy/PyCall.jl/issues/476

May 31 '19 00:05 tkf

Yes, Julia 0.6 for both. Does a newer version have the fix?

May 31 '19 01:05 chengrunyang

Sorry, I don't really understand how to do the dlopen workaround. It looks like something to be done in C, not Python or Julia. Is there any guideline for this? Thanks!

May 31 '19 16:05 chengrunyang

dlopen is a function in Julia.

https://docs.julialang.org/en/v0.6/manual/calling-c-and-fortran-code/ https://docs.julialang.org/en/v0.6/stdlib/libdl/#Base.Libdl.dlopen

It may be possible to do this in Python's ctypes with the os.RTLD_DEEPBIND flag.

May 31 '19 17:05 tkf

Do you mean I should do something like

import julia
j = julia.Julia()
j.using("Libdl")

and then something like j.dlopen(os.RTLD_DEEPBIND)?

Not sure how to call dlopen in Python via PyJulia.

Thanks!

May 31 '19 18:05 chengrunyang

There's nothing special in PyJulia. You can always Main.eval any Julia code; e.g.,

from julia import Main
Main.eval("""
using Libdl
Libdl.dlopen("PATH/TO/LIB.so", Libdl.RTLD_DEEPBIND)
""")

I mentioned ctypes because

import ctypes
import os
ctypes.CDLL("PATH/TO/LIB.so", os.RTLD_DEEPBIND)

may work as well. (I didn't test the code snippets.)

May 31 '19 18:05 tkf

In Linux, I got

ImportError: cannot import name 'Main'

when doing from julia import Main, and

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: PATH/TO/LIB.so: cannot open shared object file: No such file or directory

when doing ctypes.CDLL("PATH/TO/LIB.so", os.RTLD_DEEPBIND). Is there some installations I am missing?

May 31 '19 20:05 chengrunyang

ImportError: cannot import name 'Main'

What PyJulia version are you using?

when doing ctypes.CDLL("PATH/TO/LIB.so", os.RTLD_DEEPBIND)

Did you replace PATH/TO/LIB.so to the actual shared library?

May 31 '19 20:05 tkf

Sorry for being dumb. For the first, I upgraded PyJulia to v0.4.1 and solved the problem; for the second, I am not sure which .so file I should use, among a bunch of such files in the lib directory:

There are

libjulia-debug.so.0.6    libjulia.so      libjulia.so.0.6.0
libjulia-debug.so  libjulia-debug.so.0.6.0  libjulia.so.0.6

in lib;

libamd.so                   libmbedcrypto.so
libarpack.so                libmbedcrypto.so.0
libarpack.so.2              libmbedcrypto.so.2.3.0
libarpack.so.2.0.0          libmbedtls.so
libcamd.so                  libmbedtls.so.10
libccalltest.so             libmbedtls.so.2.3.0
libccalltest.so.debug       libmbedx509.so
libccolamd.so               libmbedx509.so.0
libcholmod.so               libmbedx509.so.2.3.0
libcolamd.so                libmpfr.so
libcurl.so                  libmpfr.so.4
libcurl.so.4                libmpfr.so.4.1.5
libcurl.so.4.4.0            libopenblas64_.so
libdSFMT.so                 libopenblas64_.so.0
libfftw3f.so                libopenlibm.so
libfftw3f.so.3              libopenlibm.so.2
libfftw3f.so.3.5.6          libopenlibm.so.2.3
libfftw3f_threads.so        libopenspecfun.so
libfftw3f_threads.so.3      libopenspecfun.so.1
libfftw3f_threads.so.3.5.6  libopenspecfun.so.1.3
libfftw3.so                 libpcre2-8.so
libfftw3.so.3               libpcre2-8.so.0
libfftw3.so.3.5.6           libpcre2-8.so.0.5.0
libfftw3_threads.so         libpcre2-posix.so
libfftw3_threads.so.3       libpcre2-posix.so.1
libfftw3_threads.so.3.5.6   libpcre2-posix.so.1.0.1
libgcc_s.so.1               libquadmath.so.0
libgfortran.so.3            libspqr.so
libgit2.so                  libssh2.so
libgit2.so.0.25.1           libssh2.so.1
libgit2.so.25               libssh2.so.1.0.1
libgmp.so                   libstdc++.so.6
libgmp.so.10                libsuitesparseconfig.so
libgmp.so.10.3.2            libsuitesparse_wrapper.so
libLLVM-3.9.1.so            libumfpack.so
libLLVM-3.9.so              sys-debug.so
libLLVM.so                  sys.so

in lib/julia.

May 31 '19 22:05 chengrunyang

No worries. PATH/TO/LIB.so needs to point to whatever the .so file contained in the Python package you've been having trouble with.

Jun 02 '19 01:06 tkf

First of all, the solution

import ctypes
import os
ctypes.CDLL("/opt/julia_files/julia-903644385b/lib/julia/libstdc++.so.6", os.RTLD_DEEPBIND)

works in my Linux Docker container with Python 3.6.7: I can switch the order between

import julia
j = julia.Julia()

and import <some_other_packages> without getting Segmentation Fault. Thanks!

Then, it seems the semantics of the Main.eval() solution needs to be changed a little bit: running

from julia import Main
Main.eval("""
using Libdl
Libdl.dlopen("/opt/julia_files/julia-903644385b/lib/julia/libstdc++.so.6", Libdl.RTLD_DEEPBIND)
""")

gives

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/usr/local/lib/python3.6/dist-packages/julia/core.py", line 1149, in eval
    ans = self._call(src)
  File "/usr/local/lib/python3.6/dist-packages/julia/core.py", line 1082, in _call
    self.check_exception(src)
  File "/usr/local/lib/python3.6/dist-packages/julia/core.py", line 1132, in check_exception
    .format(exception, src))
julia.core.JuliaError: Exception 'ArgumentError: Module Libdl not found in current path.
Run `Pkg.add("Libdl")` to install the Libdl package.' occurred while calling julia code:

using Libdl
Libdl.dlopen("/opt/julia_files/julia-903644385b/lib/julia/libstdc++.so.6", Libdl.RTLD_DEEPBIND)

While Pkg.add("Libdl") gives

ERROR: unknown package Libdl
macro expansion at ./pkg/entry.jl:53 [inlined]
(::Base.Pkg.Entry.##1#3{String,Base.Pkg.Types.VersionSet})() at ./task.jl:335
Stacktrace:
 [1] sync_end() at ./task.jl:287
 [2] macro expansion at ./task.jl:303 [inlined]
 [3] add(::String, ::Base.Pkg.Types.VersionSet) at ./pkg/entry.jl:51
 [4] (::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}})() at ./pkg/dir.jl:36
 [5] cd(::Base.Pkg.Dir.##4#7{Array{Any,1},Base.Pkg.Entry.#add,Tuple{String}}, ::String) at ./file.jl:70
 [6] #cd#1(::Array{Any,1}, ::Function, ::Function, ::String, ::Vararg{String,N} where N) at ./pkg/dir.jl:36
 [7] add(::String) at ./pkg/pkg.jl:117

I think this might be because ``Libdlalready exists asBase.Libdlin Julia, which is the result I get when typingLibdl` in Julia's terminal. Thus I changed the command in Python to

from julia import Main
Main.eval("""
Base.Libdl.dlopen("PATH/TO/LIB.so", Libdl.RTLD_DEEPBIND)
""")

and this gives c_void_p(19006544). Afterwards there is still Segmentation Fault in Python.

Jun 02 '19 17:06 chengrunyang

I'm surprised that lib/julia/libstdc++.so.6 helps, because that's not what I meant. But if it works then it's all good?

Anyway, just FYI, what I meant was

from julia import Main

Main.eval("""
Libdl.dlopen("PATH/TO/LIB.so", Libdl.RTLD_DEEPBIND)
""")

import SOME_OTHER_PACKAGE

where PATH/TO/LIB.so probably is located in PYTHON_ENV/lib/pythonX.Y/site-packages/SOME_OTHER_PACKAGE/ or somewhere similar for the dependencies of SOME_OTHER_PACKAGE. For example, for matplotlib, it could be lib/python3.7/site-packages/matplotlib/ft2font.cpython-37m-x86_64-linux-gnu.so (I randomly picked it up).

I think this might be because ``Libdlalready exists asBase.Libdlin Julia, which is the result I get when typingLibdl` in Julia's terminal.

Ah, you are right. I forgot that you don't need using Libdl in Julia 0.6. It's already in the default namespace. You can replace Base.Libdl with Libdl in your code.

Jun 03 '19 01:06 tkf

I see. Do you mean I should try out .so files to see which one works?

Also, regarding the solution of calling dlopen in Python, in some environments there is no os.RTLD_DEEPBIND in Python, which means

import os
os.RTLD_DEEPBIND

gives

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'os' has no attribute 'RTLD_DEEPBIND'

Do you know the reason for that? Sorry I haven't gained enough understanding about dlopen to figure it out.

Jun 05 '19 03:06 chengrunyang

in some environments there is no os.RTLD_DEEPBIND in Python

os.RTLD_DEEPBIND does not exist in Python older than 3.3 (and non-Unix OS?).

See: https://docs.python.org/3/library/os.html#os.RTLD_DEEPBIND

Do you mean I should try out .so files to see which one works?

Yeah, I think that's pretty much the case.

I just noticed that Python has sys.setdlopenflags in Unix. Maybe you can try sys.setdlopenflags(os.RTLD_DEEPBIND)? (This is a very random suggestion)

Jun 05 '19 04:06 tkf

Regarding os.RTLD_DEEPBIND (thanks for the reference), I cannot find it on Mac OS X with Anaconda Python 3.7.3. Might it be because this is Anaconda Python, so that it can't access to some system information (as opposed to system Python)?

Jun 05 '19 05:06 chengrunyang

Maybe it's not available in macOS. I don't think it's related to Anaconda distribution (I can access os.RTLD_DEEPBIND in Python installed via conda in Linux).

Jun 05 '19 05:06 tkf

I see. Thanks for helping me check this! The bridge between Julia and Python seems hard to build, and also unstable...

Jun 05 '19 05:06 chengrunyang