Python callback to a Julia function?

[orenbenkiki]

If I have a Julia function which takes a callback (artificial example here):

function caller(callback, arg_vector)
    callback(arg_vector)
    return nothing
end

outer_vector = [0]
caller(outer_vector) do inner_vector 
    inner_vector[1] = 1
end
@assert outer_vector[1] == 1

And I'd like to call it from Python - it seems not possible to do so? Ideally:

outer_vector = np.array([0])
with jl.MyModule.caller(outer_vector) as inner_vector:
    inner_vector[0] = 1
assert outer_vector[0] == 1

I have a Julia package that uses callbacks for various functions (for example, initializing arrays), and I'm trying to wrap it with a Python interface. Being able to zero-copy pass around numpy arrays is a godsend, but it seems that callbacks of the above type are not supported. Looking at the code I see the tests for "callback" are empty...

Is there some manual workaround I could use in my code instead of direct support for the above? Any way at all, as long as I can bury the boilerplate code in my Python wrappers so the end user can use the with statement.

[cjdoris]

I don't think you can do this with the with statement - that just runs the code in the with block immediately, whereas you need to create a function to pass to Julia. It's very different from the do syntax in Julia which does create a function.

You'll have to do something like

outer_vector = np.array([0])
def callback(inner_vector):
    inner_vector[0] = 1
jl.MyModule.caller(callback, outer_vector)
assert outer_vector[0] == 1

[orenbenkiki]

First, with statements do not run the code "immediately", they run it when the yield statement is invoked in the body of the contextmanager.

Second and more importantly, even putting this aside, a plain callback doesn't work. For example:

jl.seval("""
function jl_caller(jl_called::Function)
    return jl_called("foo")
end
""")

def py_called(text: str) -> str:
    return text + "bar"

print(jl.jl_caller(py_called))

Gives the error message:

Traceback (most recent call last):
  File "/Users/obk/projects/Daf.py/callback.py", line 12, in <module>
    print(jl.jl_caller(py_called))
          ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/obk/.julia/packages/PythonCall/wXfah/src/jlwrap/any.jl", line 208, in __call__
    return self._jl_callmethod($(pyjl_methodnum(pyjlany_call)), args, kwargs)
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Julia: MethodError: no method matching jl_caller(::Py)

Closest candidates are:
  jl_caller(!Matched::Function)
   @ Main none:1

It seems that a Python function object is not converted to a Julia function, instead it is wrapped into a generic Py object, so callbacks just aren't supported?

[orenbenkiki]

Julia can call Python and Pythin can call Julia so this is possible. The workaround is somewhat convoluted:

cat callback.py
from contextlib import contextmanager
from juliacall import Main as jl # type: ignore

jl.seval("""
function jl_caller(jl_called::Function)::Any
    return jl_called("foo")
end
""")

jl.seval("""
function py_to_function(py_object::Py)::Function
    return (args...; kwargs...) -> pycall(py_object, args...; kwargs...)
end
""")

# Pass callback as an argument:
def py_called(text: str) -> str:
    return text + "bar"

print(jl.jl_caller(jl.py_to_function(py_called)))

# Use with statement:
@contextmanager
def py_caller() -> None:
    def capture(text):
        yield text
    # yield from capture("foo")
    yield from jl.jl_caller(jl.py_to_function(capture))

with py_caller() as text:
    print(text + "bar")

Running this prints foobar twice as expected. Nice.

So, back to the feature request: Can we have a built-in conversion rule that takes plain-old Python functions and lambdas and wraps them as Julia functions. This would allow passing functions as arguments without having to use the above workaround (that is, remove the need for defining and using py_to_function).

[cjdoris]

It seems that a Python function object is not converted to a Julia function, instead it is wrapped into a generic Py object, so callbacks just aren't supported?

Well no, they are wrapped as a generic Py object, but those are still callable, so can be used as callbacks.

Your issue is simply that you've got a ::Function type annotation on jl_caller when py_called is received as a Py. If you remove it then the simple version works:

>>> jl.seval("""
... function jl_caller(jl_called)
...     return jl_called("foo")
... end
... """)
Julia: jl_caller (generic function with 1 method)
>>> def py_called(text: str) -> str:
...     return text + "bar"
...
>>> print(jl.jl_caller(py_called))
foobar

[cjdoris]

I'm pretty sure we were talking at cross purposes about the with statement. In your original post it looked a lot like you were trying to use with in the same way as Julia's do, but in your later posts it seems that's not the case. Anyway that's all tangential to the main issue.

[orenbenkiki]

Yes, there are two issues - Py vs. Function and with vs. do.

My later post showed a workaround around both issues which requires writing manual wrappers.

So it is possible to do achieve what I want (given writing the manual wrappers), which is great!

That said, ideally one should not have to write such wrappers:

• Python functions "should" be converted to some PyFunction type which is a Julia Function, so they would work even if the Julia function specified ::Function for the callback argument.

• The juliacall Python module should provide a context wrapper function so one could, in Python, say:

with juliacall.context(jl.MyModule.foo)(...args...) as ...:
    ...

Makes sense?

[cjdoris]

I'm happy to consider the PyFunction idea - feel free to make a separate issue about that.

[cjdoris]

I don't understand what you want juliacall.context to do?

[orenbenkiki]

Something along the lines of the following (up to bikeshedding on the names and exact syntax):

from contextlib import contextmanager
from juliacall import Main as jl # type: ignore
from typing import Any
from typing import Callable
from typing import Iterator

#: This would not be needed if/when issue #477 is resolved.
jl.seval("""
function py_function_to_fulia_function(py_object::Py)::Function
    return (args...; kwargs...) -> pycall(py_object, args...; kwargs...)
end
""")

# Example Julia caller function.
jl.seval("""
function jl_caller(callback::Function, positional:: AbstractString; named:: AbstractString)::Any
    extra = 1
    return callback(positional, named, extra)  # All must be positional.
end
""")

# Example Python callback function.
def py_callback(first: str, second: str, third: int) -> Any:
    print(f"first: {first}")
    print(f"second: {second}")
    print(f"third: {third}")
    return 7

# Pass a callback as an explicit Function parameter. Return value is available.
returned = jl.jl_caller(jl.py_function_to_fulia_function(py_callback), "positional", named ="named")
print(f"returned: {returned}")

# Proposed addition to `juliacall`, converts Python `with` to work similarly to Julia's `do`.
@contextmanager
def jl_do(jl_caller: Callable, *args: Any, **kwargs: Any) -> Iterator[Any]:
    def capture(*args: Any) -> Iterator[Any]:
        if len(args) == 1:
            yield args[0]
        else:
            yield args
    yield from jl_caller(jl.py_function_to_fulia_function(capture), *args, **kwargs)

# Use in `with` statement. No return value.
with jl_do(jl.jl_caller, "positional", named = "named") as args:
    print(f"args: {args}")

[cjdoris]

Could you explain some more how this is useful? I don't understand the utility of jl_do - as far as I can tell it has very little similarity to Julia's do syntax.

[orenbenkiki]

Consider Julia do:

jl_caller("positional", named="named") do first, second, third
    println("first: $(first)")
    println("second: $(second)")
    println("third: $(third)")
end

Compared to Python with:

with jl_do(jl.jl_caller, "positional", named="named") as (first, second, third):
    print(f"first: {first}")
    print(f"second: {second}")
    print(f"third: {third}")

Looks mighty similar to me.