maintaining bidirectional transformation equality when automatically converting (semantically) immutable data

[thautwarm]

from julia import Main as jl
jl.seval("1 => 2") #  python tuple (1, 2)
f = jl.seval(r"""
function (p::Pair{Int, Int})
    p.first
end """)

f(jl.seval("1 => 2"))
# TypeError: Julia: MethodError: no method matching (::var"#1#2")(::Tuple{Int64, Int64})
# Closest candidates are:
#  (::var"#1#2")(!Matched::Pair{Int64, Int64}) at none:2

Automatic conversion is handy, but we could avoid possible issues if we hold to_julia(to_python(o)) === o in Julia or to_python(to_julia(o)) == o in Python.

[cjdoris]

Invariants

Going from Julia to Python to Julia, the invariant pyconvert(typeof(x), Py(x)) === x holds for any Julia value x. The typeof(x) is necessary because for example Int32, Int64, BigInt etc are all converted to Python int.

If x is semantically* mutable, then pyconvert(Any, Py(x)) === x also holds, because Py(x) will be a wrapper and converting a wrapper by default just unwraps it. If x is semantically* immutable, then pyconvert(Any, Py(x)) == x should usually hold, but no promises.

*for example BigInt is a mutable type, but is semantically immutable because there is no API to mutate it.

In the other direction (Python to Julia to Python) then pyeq(Py(pyconvert(Any, x)), x) should be true for any x::Py. That is, the default conversion to Julia and back should be equal to the original, but not necessarily be the exact same object. This invariant might not actually always hold (for the same reason above that type information gets lost) but I think there are always fewer Python types than Julia ones so it does hold.

Stricter Invariants?

You want pyconvert(Any, Py(x)) === x to always hold. The issue here is that it means that Py(x) needs to know not only the value of x but also its type. The only way this seems possible would be to e.g. have subtypes of int corresponding to each Julia subtype of Integer (so Int64 is converted to int_Int64, which is then converted back to Int64) but this would complicate things a lot.

Your Example

In your example you have

f = jl.seval("function (p::Pair) ... end")
x = jl.seval("1 => 2")
f(x)

but it doesn't work because x was automatically converted to a Python tuple, which is then converted to a Julia Tuple when calling f.

In practice, when calling f(x), the caller should know enough information about f and x to make this work via one of these two methods:

• Call f(juliacall.convert(T, x)) instead for some appropriate type T (such as jl.Pair or jl.Pair[jl.Int, jl.Int] in the example).
• Define f = jl.seval("PythonCall.pyfunc(function(x::Py) ... end)")) instead. If you wrap a function in pyfunc then when calling f(x) it receives x as an unconverted Py, which the function can convert itself as necessary.

Alternatively you can do:

jlr = jl._jl_raw()
f = jlr.seval("function ... end")
x = jlr.seval("1 => 2")
f(x)

then f, x and f(x) are all wrapped Julia values (instead of being converted to Python values). In particular, x wraps 1 => 2 which is simply unwrapped when being passed to f.

[thautwarm]

Your explanation is pretty neat, and I think jl._jl_raw() is extractly what I need. Actually, I'm always thinking about such a thing when using juliacall.

Checking the functionality of jl._jl_raw, if things works fine I will close this issue.

P.S: I'd appreciate it a lot for this well-designed Julia-Python interop packages 👉🥇.

[github-actions[bot]]

This issue has been marked as stale because it has been open for 30 days with no activity. If the issue is still relevant then please leave a comment, or else it will be closed in 7 days.