Weird behavior importing lists from python to Julia

[droodman]

Affects: JuliaCall

Describe the bug I'm getting weird crashes when I transfer a list from Python to Julia. This does not happen with PyJulia. Minimum working example below.

>>> from juliacall import Main as jl
>>> jl.x = [1., 2.]
>>> jl.seval("show(x)")
Any["s", "h", "o", "w", "(", "x", ")"]

>>> jl.x = 1
>>> jl.seval("show(x)")
1

>>> from julia import Main
>>> Main.x = [1., 2.]
>>> Main.eval("show(x)")
[1.0, 2.0]

Your system

• Windows 11 Pro 22H2
• Julia 1.9.2; Python 3.11, JuliaCall 0.9.13

C:\Users\drood>pip list
Package          Version
---------------- -------
fortls           2.13.0
json5            0.9.14
julia            0.6.1
juliacall        0.9.13
juliapkg         0.1.10
numpy            1.25.0
packaging        23.1
pip              23.2
psutil           5.9.5
semantic-version 2.10.0
setuptools       65.5.0

[github-actions[bot]]

This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days.

[cjdoris]

This is indeed mysterious.

>>> from juliacall import Main as jl
>>> jl.x = [1,2,3]
>>> jl.x
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\chris\.julia\packages\PythonCall\qTEA1\src\jlwrap\any.jl", line 195, in __getattr__
    return self._jl_callmethod($(pyjl_methodnum(pyjlany_getattr)), k)
SystemError: <built-in method _jl_callmethod of ModuleValue object at 0x0000023033B938E0> returned NULL without setting an exception

This could be hard to debug. I guess just don't assign to a module for now!

[MilesCranmer]

Ran into this too. Do you know what the issue is?

With PyJulia:

[ins] In [1]: from julia import Main as jl

[ins] In [2]: jl.x = [(-1, -1), (-1, -1), (-1, -1), (-1, -1)]

[ins] In [3]: jl.typeof(jl.x)
Out[3]: <PyCall.jlwrap Vector{Tuple{Int64, Int64}}>

With juliacall:

[ins] In [1]: from juliacall import Main as jl

[ins] In [2]: jl.x = [(-1, -1), (-1, -1), (-1, -1), (-1, -1)]

[ins] In [3]: jl.typeof(jl.x)
--------------------------------------------------------------------
SystemError                        Traceback (most recent call last)
Cell In[3], line 1
----> 1 jl.typeof(jl.x)

File ~/.julia/packages/PythonCall/wXfah/src/jlwrap/any.jl:195, in __getattr__(self, k)
    193         raise AttributeError(k)
    194     else:
--> 195         return self._jl_callmethod($(pyjl_methodnum(pyjlany_getattr)), k)
    196 def __setattr__(self, k, v):
    197     try:

SystemError: <built-in method _jl_callmethod of ModuleValue object at 0x104a29f90> returned NULL without setting an exception

[MilesCranmer]

Did a python -m pdb run... So, this is the code being executed in the PyJulia version:

https://github.com/JuliaPy/pyjulia/blob/8ab68f4c7844f3a61f6b3e9799cbbaaf8c590926/src/julia/core.py#L206

    def __setattr__(self, name, value):
        if name.startswith('_'):
            super(JuliaMainModule, self).__setattr__(name, value)
        else:
            juliapath = remove_prefix(self.__name__, "julia.")
            setter = '''
            PyCall.pyfunctionret(
                (x) -> Base.eval({}, :({} = $x)),
                Any,
                PyCall.PyAny)
            '''.format(juliapath, jl_name(name))
            self._julia.eval(setter)(value)

    help = property(lambda self: self._julia.help)
    eval = property(lambda self: self._julia.eval)
    using = property(lambda self: self._julia.using)

Compared the juliacall version:

https://github.com/JuliaPy/PythonCall.jl/blob/13f596d6a7d60ef7bfcee2d538cd895f59826d95/src/JlWrap/any.jl#L211-L219

        def __setattr__(self, k, v):
            try:
                ValueBase.__setattr__(self, k, v)
            except AttributeError:
                if k.startswith("__") and k.endswith("__"):
                    raise
            else:
                return
            self._jl_callmethod($(pyjl_methodnum(pyjlany_setattr)), k, v)

---

@cjdoris should this __setattr__ be overridden in init_module() in module.jl? I tried to do it myself but I couldn't figure out the logic of pyjl_methodnum.

[MilesCranmer]

Okay this was quite a journey through pointer land so I might be looking at the wrong thing. So I wasn't not sure where _jl_callmethod is actually being called... I tried to find ValueBase but it led me to

    setptr!(pyjlbasetype, incref(Cjl.PyJuliaBase_Type[]))
    pyjuliacallmodule.ValueBase = pyjlbasetype

So it looks like ValueBase is actually PyJuliaBase_Type. This gets created at this line:

    o = PyJuliaBase_Type[] = C.PyPtr(pointer(_pyjlbase_type))

This then led me to

    _pyjlbase_type[] = C.PyTypeObject(
        name = pointer(_pyjlbase_name),
        basicsize = sizeof(PyJuliaValueObject),
        # new = C.POINTERS.PyType_GenericNew,
        new = @cfunction(_pyjl_new, C.PyPtr, (C.PyPtr, C.PyPtr, C.PyPtr)),
        dealloc = @cfunction(_pyjl_dealloc, Cvoid, (C.PyPtr,)),
        flags = C.Py_TPFLAGS_BASETYPE | C.Py_TPFLAGS_HAVE_VERSION_TAG,
        weaklistoffset = fieldoffset(PyJuliaValueObject, 3),
        # getattro = C.POINTERS.PyObject_GenericGetAttr,
        # setattro = C.POINTERS.PyObject_GenericSetAttr,
        methods = pointer(_pyjlbase_methods),
        as_buffer = pointer(_pyjlbase_as_buffer),
    )

then it looks like it is actually defined under _pyjlbase_callmethod_name

const _pyjlbase_callmethod_name = "_jl_callmethod"

which references

        C.PyMethodDef(
            name = pointer(_pyjlbase_callmethod_name),
            meth = @cfunction(_pyjl_callmethod, C.PyPtr, (C.PyPtr, C.PyPtr)),
            flags = C.Py_METH_VARARGS,
        ),

which finally brings me to

function Cjl._pyjl_callmethod(f, self_::C.PyPtr, args_::C.PyPtr, nargs::C.Py_ssize_t)

So my understanding is that

    if Cjl.PyJuliaValue_IsNull(self_)

is where the bug is thrown? Does that mean the module is being cast into a NULL pointer??

[cjdoris]

I'm pretty sure this issue is already fixed by this commit https://github.com/JuliaPy/PythonCall.jl/commit/45a75c1ecbf6ae8f5ffd63693083598b8b688358 and is just waiting for the next release - which I've been delaying until I get more test coverage because I recently did a big refactor and want to be more confident I didn't break anything.

[MilesCranmer]

How do I work with the dev version of PythonCall.jl from within ipython? I tried pip install -e . on the git repository; but importing juliacall instead installed the PythonCall.jl from the registry (without that fix). I tried tweaking the meta.json file but had no luck. I manually updated the Project.toml file in the Julia env but it seemed to cause other issues.

Any tips?

[cjdoris]

The instructions are in the docs here: https://juliapy.github.io/PythonCall.jl/stable/juliacall/#Installation

[MilesCranmer]

~~Ah, sorry. I didn’t expect the dev instructions to be so early in the page — was checking much deeper in the docs…... mea culpa~~

Actually seems like it's currently broken....