Example python package using juliacall

[cjdoris]

[msundvick]

Possibly related: I was just looking into trying to make a combined python-julia project (with the python and julia code in the same git repository, and neither are published). Any ideas for how this should work? I've got an example where I manually modify the "juliacalldeps.json" file in a build step, but is there something smarter that could be done?

[cjdoris]

What exactly does the build step do? I imagine you need to refer to the directory of the package. I've been thinking about adding some templating so that ${__DIR__} expands to the directory. Would that suffice?

[msundvick]

Ok, to make this easier I thought I'd actually publish an example (and also contribute to this issue a little more directly). You can see what I'm currently doing here https://github.com/msundvick/ffi_examples.

Specifically, the relevant build step is in build.py:

    with open("juliacalldeps.json", "r") as f:
        jl = json.load(f)

    jl["packages"]["FfiExamples"]["path"] = os.getcwd()
    with open("juliacalldeps.json", "w") as f:
        json.dump(jl, f)

    import juliacall

As for whether ${__DIR__} will work... Maybe? I'm guessing this will basically do the same thing as the replacement that I've got going here. Is there any reason that relative paths couldn't be used instead?

Edit: I've tried to strip it down a bit; there's a bunch of other stuff in that repo: https://github.com/msundvick/ffi_examples/tree/just-julia.

[cjdoris]

Oh right yeah, the path should definitely be evaluated relative to the directory containing the file. I'll make a separate issue for that.

[cjdoris]

I've fixed the relative paths issue on the master branch.

[henry2004y]

I'm also expecting an example python package that uses JuliaCall, i.e. a demonstration of Python wrapper over a Julia package. More specifically, I'm expecting demos of

• type conversions, e.g. ways to check and convert Julia structs into native Python objects. I see many methods for converting Julia types into Python from the Julia side Wrap-Julia_values, but I haven't found methods for converting Julia types into Python from the Python side (I guess like jlconvert from the naming conventions?)
• general ways of handling Julia codes in Python. (There is now a seval, as the correspondence of pyeval and pyexec.)

Maybe I can also help on these as well.

[cjdoris]

There is no jlconvert because that wouldn't be idiomatic - Python has no equivalent of convert.

Instead, JuliaCall takes advantage of multiple inheritance so that a Julia Dict is wrapped as a DictValue, which is both an AnyValue (i.e. a Julia object) and a Mapping (i.e. behaves like a Python dict). If you want an actual Python dict you can do dict(x) which is idiomatic.

This system can be extended by defining new subtypes of AnyValue and overloading pyjl to wrap particular Julia types as the new Python type.

It would be great if you could contribute an example package.

[henry2004y]

I learn something more! So here is a situation where I want to pass a Julia object (currently shown as AnyValue) to a Python function. I have a Julia package which defines a struct in MyPkg like

struct MyType
   name::String
   dir::String
   fid::IOStream
   variable::Vector{String}
   #...
end

function load(...)
   # some operations...
   return MyType(...)
end

and in Python

from juliacall import Main as jl
jl.seval("using MyPkg")
file = "example_file_name"
meta = jl.load(file) # <class 'juliacall.AnyValue'>, shown by type(meta)

Now I may need some Python functions which operates on this Julia type

def plot(meta: MyType):
    var = jl.readvariable(meta, "myvar")
    x = np.arange(meta.coordmin[0], meta.coordmax[0], meta.dcoord[0])
    plt.plot(x, var)
    plt.show()

Without the type annotation in Python, this can work, since all the methods for meta lives in the Julia package which understands MyType and JuliaCall does the automatic conversion for me. However, say if I do need a concrete type even for the Python function, it seems that I need new subtypes of AnyValue, as your said

This system can be extended by defining new subtypes of AnyValue and overloading pyjl to wrap particular Julia types as the new Python type.

If you can quickly provide a concrete example of how to do this, I think I may easily come up with a Python interface for my data processing package that uses Matplotlib for plotting :smile: .

[fzeiser]

I'd be great to have some example(s) on this. I started with a very basic repo, but still I have difficulties. Maybe someone here has a good idea?

https://github.com/fzeiser/juliacall_test

I have a very simple routine I want to call in julia from python:

module Foo
export foo

using PythonCall

function foo(a)
	println(typeof(a))
	b = pyconvert(Array, a)
	println(typeof(b))
end

end

If I try to call this script with juliacall and jl.seval(f'include("[...]/Foo.jl")') it will work! If i use using Foo it will not.

I think that it would be nice to have some basic examples of this kind. and of course -- any help on my particular example is appreciated :).

The issue with using ... arises when compiling a julia modules that uses using PythonCall; one can see that the Bar module can be run with using Bar.... For python call_using.py I get following problem:

   Resolving package versions...
    Updating `~/pyenv/algo/julia_env/Project.toml`
  [fdd1085a] + Foo v0.0.0 `~/repos/tmp/julia/Foo`
    Updating `~/pyenv/algo/julia_env/Manifest.toml`
  [fdd1085a] + Foo v0.0.0 `~/repos/tmp/julia/Foo`

signal (11): Segmentation fault
in expression starting at /home/u54671/repos/tmp/julia/Foo/src/Foo.jl:4
_dl_lookup_symbol_x at /usr/src/debug/glibc-2.34-29.fc35.x86_64/elf/dl-lookup.c:850
do_sym at /usr/bin/../lib64/libc.so.6 (unknown line)
dlsym_doit at /usr/bin/../lib64/libc.so.6 (unknown line)
_dl_catch_exception at /usr/bin/../lib64/libc.so.6 (unknown line)
_dl_catch_error at /usr/bin/../lib64/libc.so.6 (unknown line)
_dlerror_run at /usr/bin/../lib64/libc.so.6 (unknown line)
dlsym at /usr/bin/../lib64/libc.so.6 (unknown line)
jl_dlsym at /usr/bin/../lib64/julia/libjulia-internal.so.1 (unknown line)
#dlsym#1 at ./libdl.jl:59 [inlined]
dlsym at ./libdl.jl:57 [inlined]
init_pointers at /home/u54671/.julia/packages/PythonCall/XgP8G/src/cpython/pointers.jl:283
init_pointers at /home/u54671/.julia/packages/PythonCall/XgP8G/src/cpython/pointers.jl:283 [inlined]
init_context at /home/u54671/.julia/packages/PythonCall/XgP8G/src/cpython/context.jl:40
__init__ at /home/u54671/.julia/packages/PythonCall/XgP8G/src/cpython/CPython.jl:21
unknown function (ip: 0x7efd19fa4ac3)
unknown function (ip: 0x7efd7e6020af)
jl_init_restored_modules at /usr/bin/../lib64/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7efd5ef37ce2)
unknown function (ip: 0x7efd5f02e51c)
unknown function (ip: 0x7efd5f036c66)
unknown function (ip: 0x7efd5efca6d0)
unknown function (ip: 0x7efd5eae5ec6)
unknown function (ip: 0x7efd7e6e9363)
unknown function (ip: 0x7efd7e6053e0)
unknown function (ip: 0x7efd7e604cd1)
unknown function (ip: 0x7efd7e604fae)
jl_toplevel_eval_in at /usr/bin/../lib64/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7efd5efbc9ba)
unknown function (ip: 0x7efd5ef3743a)
unknown function (ip: 0x7efd5ebea44e)
unknown function (ip: 0x7efd5ebea76b)
unknown function (ip: 0x7efd7e5e6b7b)
unknown function (ip: 0x7efd7e5e63dd)
unknown function (ip: 0x7efd7e5e73ae)
unknown function (ip: 0x7efd7e5e7ac1)
unknown function (ip: 0x7efd7e604591)
jl_toplevel_eval_in at /usr/bin/../lib64/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7efd5e6394e7)
unknown function (ip: 0x7efd7e5e6b7b)
unknown function (ip: 0x7efd7e5e63dd)
unknown function (ip: 0x7efd7e5e73ae)
unknown function (ip: 0x7efd7e5e7ac1)
unknown function (ip: 0x7efd7e604591)
unknown function (ip: 0x7efd7e604fae)
jl_toplevel_eval_in at /usr/bin/../lib64/julia/libjulia-internal.so.1 (unknown line)
unknown function (ip: 0x7efd5f0e5897)
unknown function (ip: 0x7efd5ec03ea7)
unknown function (ip: 0x7efd5ec04018)
unknown function (ip: 0x7efd7e6283e9)
jl_repl_entrypoint at /usr/bin/../lib64/julia/libjulia-internal.so.1 (unknown line)
main at /usr/bin/julia (unknown line)
__libc_start_call_main at /usr/bin/../lib64/libc.so.6 (unknown line)
__libc_start_main at /usr/bin/../lib64/libc.so.6 (unknown line)
_start at /usr/bin/julia (unknown line)
Allocations: 552910 (Pool: 552426; Big: 484); GC: 1
Traceback (most recent call last):
  File "/home/u54671/repos/tmp/python/call_using.py", line 8, in <module>
    jl.seval("using Foo")
  File "/home/u54671/.julia/packages/PythonCall/XgP8G/src/jlwrap/module.jl:19", line 7, in seval
juliacall.JuliaError: Failed to precompile Foo [fdd1085a-9f79-4a83-9a23-f2e3e5a2036c] to /home/u54671/.julia/compiled/v1.7/Foo/jl_UWZHpi.
Stacktrace:
 [1] pyjlmodule_seval(self::Module, expr::PythonCall.Py)
   @ PythonCall ~/.julia/packages/PythonCall/XgP8G/src/jlwrap/module.jl:13
 [2] _pyjl_callmethod(f::Any, self_::Ptr{PythonCall.C.PyObject}, args_::Ptr{PythonCall.C.PyObject}, nargs::Int64)
   @ PythonCall ~/.julia/packages/PythonCall/XgP8G/src/jlwrap/base.jl:62
 [3] _pyjl_callmethod(o::Ptr{PythonCall.C.PyObject}, args::Ptr{PythonCall.C.PyObject})
   @ PythonCall.C ~/.julia/packages/PythonCall/XgP8G/src/cpython/jlwrap.jl:47

[I tried to add a juliapkg.json, but that did not help / or I did it wrong]

[fzeiser]

Big surprise for me: I got it working -- but I'm not sure what the general message of this should be / what I did wrong in the first place.

I added a juliapkg.json, but that didn't solve the issue:

> python call_using.py
[juliapkg] Locating Julia ^1.7
[juliapkg] Querying Julia versions from https://julialang-s3.julialang.org/bin/versions.json
[juliapkg] Using Julia 1.7.2 at /usr/bin/julia
[juliapkg] Using Julia project at /home/u54671/pyenv/algo/julia_env
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.develop([Pkg.PackageSpec(name="Foo", uuid="fdd1085a-9f79-4a83-9a23-f2e3e5a2036c", path=raw"/home/u54671/repos/tmp/julia/Foo")])
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
 [...]
signal (11): Segmentation fault
[...]
   @ PythonCall ~/.julia/packages/PythonCall/XgP8G/src/jlwrap/base.jl:62
 [3] _pyjl_callmethod(o::Ptr{PythonCall.C.PyObject}, args::Ptr{PythonCall.C.PyObject})
   @ PythonCall.C ~/.julia/packages/PythonCall/XgP8G/src/cpython/jlwrap.jl:47

. I then tried to run the same commands as juliapkg seemingly tries to run in the terminal

subprocess.CalledProcessError: Command '['/usr/bin/julia', '--project=/home/u54671/pyenv/algo/julia_env', '-e', 'import Pkg; Pkg.develop([Pkg.PackageSpec(name="Foo", uuid="fdd1085a-9f79-4a83-9a23-f2e3e5a2036c", path=raw"/home/u54671/repos/tmp/julia/Foo")]); Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")]); Pkg.resolve()']'

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.2 (2022-02-06)
 _/ |\__'_|_|_|\__'_|  |  Fedora 35 build
|__/                   |

julia> import Pkg; Pkg.develop([Pkg.PackageSpec(name="Foo", uuid="fdd1085a-9f79-4a83-9a23-f2e3e5a2036c", path=raw"/home/u54671/repos/tmp/julia/Foo")]);
   Resolving package versions...
    Updating `~/pyenv/algo/julia_env/Project.toml`
  [fdd1085a] ~ Foo v0.0.0 `~/repos/tmp/julia/Foo` ⇒ v0.0.0 `~/repos/tmp/julia/Foo`
    Updating `~/pyenv/algo/julia_env/Manifest.toml`
  [fdd1085a] ~ Foo v0.0.0 `~/repos/tmp/julia/Foo` ⇒ v0.0.0 `~/repos/tmp/julia/Foo`

julia> using Foo
[ Info: Precompiling Foo [fdd1085a-9f79-4a83-9a23-f2e3e5a2036c]
^[[    CondaPkg Found dependencies: /home/u54671/.julia/packages/PythonCall/XgP8G/CondaPkg.toml
    CondaPkg Resolving changes
             + python
    CondaPkg Installing packages
  Package               Version  Build               Channel                    Size
──────────────────────────────────────────────────────────────────────────────────────
  Install:
──────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex           0.1  conda_forge         conda-forge/linux-64     Cached
  + _openmp_mutex           4.5  1_gnu               conda-forge/linux-64     Cached
  + bzip2                 1.0.8  h7f98852_4          conda-forge/linux-64     Cached
  + ca-certificates   2021.10.8  ha878542_0          conda-forge/linux-64     Cached
  + ld_impl_linux-64     2.36.1  hea4e1c9_2          conda-forge/linux-64     Cached
  + libffi                3.4.2  h7f98852_5          conda-forge/linux-64     Cached
  + libgcc-ng            11.2.0  h1d223b6_14         conda-forge/linux-64     Cached
  + libgomp              11.2.0  h1d223b6_14         conda-forge/linux-64     Cached
  + libnsl                2.0.0  h7f98852_0          conda-forge/linux-64     Cached
  + libuuid              2.32.1  h7f98852_1000       conda-forge/linux-64     Cached
  + libzlib              1.2.11  h166bdaf_1014       conda-forge/linux-64     Cached
  + ncurses                 6.3  h9c3ff4c_0          conda-forge/linux-64     Cached
  + openssl               3.0.2  h166bdaf_1          conda-forge/linux-64     Cached
  + pip                  22.0.4  pyhd8ed1ab_0        conda-forge/noarch       Cached
  + python               3.10.4  h2660328_0_cpython  conda-forge/linux-64     Cached
  + python_abi             3.10  2_cp310             conda-forge/linux-64     Cached
  + readline                8.1  h46c0cb4_0          conda-forge/linux-64     Cached
  + setuptools           61.3.1  py310hff52083_0     conda-forge/linux-64     Cached
  + sqlite               3.37.1  h4ff8645_0          conda-forge/linux-64     Cached
  + tk                   8.6.12  h27826a3_0          conda-forge/linux-64     Cached
  + tzdata                2022a  h191b570_0          conda-forge/noarch       Cached
  + wheel                0.37.1  pyhd8ed1ab_0        conda-forge/noarch       Cached
  + xz                    5.2.5  h516909a_1          conda-forge/linux-64     Cached
  + zlib                 1.2.11  h166bdaf_1014       conda-forge/linux-64     Cached

  Summary:

  Install: 24 packages

  Total download: 0 B

──────────────────────────────────────────────────────────────────────────────────────


julia> using Foo

julia> Foo.foo("asd")
String
Vector{String}

julia> 
> python call_using.py
   Resolving package versions...
    Updating `~/pyenv/algo/julia_env/Project.toml`
  [fdd1085a] ~ Foo v0.0.0 `~/repos/tmp/julia/Foo` ⇒ v0.0.0 `~/repos/tmp/julia/Foo`
    Updating `~/pyenv/algo/julia_env/Manifest.toml`
  [fdd1085a] ~ Foo v0.0.0 `~/repos/tmp/julia/Foo` ⇒ v0.0.0 `~/repos/tmp/julia/Foo`
PythonCall.PyList{PythonCall.Py}
Vector{Int64}
None

Suddenly, now I can run call_using.py just fine:

$ python call_using.py
   Resolving package versions...
  No Changes to `~/pyenv/algo/julia_env/Project.toml`
  No Changes to `~/pyenv/algo/julia_env/Manifest.toml`
PythonCall.PyList{PythonCall.Py}
Vector{Int64}
None

Maybe someone can try to reproduce this? Maybe it was just a confusion on my side due to the environments? Still, I think the error message was not quite telling...