Pass function/closure to Python

[tanmayb123]

I'm not sure how difficult this would be, or even if it's possible -- could we pass functions/closures from Swift to Python?

Something like this:

test.py:

def f(x, y):
    return x(y + 1)

if __name__ == "__main__":
    print(f(lambda z: z**2, 5)) # Should print 6^2

test.swift

import Python

let sys = Python.import("sys")
sys.path.append(".")
let test = Python.import("test")

print(test.f({ z in pow(z, 2) }, 5)) // I wish this were possible!

[dan-zheng]

There doesn't exist a "ExpressibleByClosureLiteral" to enable implicit conversion of the { z in pow(z, 2) } expression into a PythonObject.

More critically, translating Swift expressions/statements within a lambda into Python expressions/statements (e.g. turning { z in pow(z, w) } into Python C API calls to build up a similar Python function object) requires non-trivial machinery.

Possible solutions are:

• Language virtualization: turning native language constructs into customizable entry points.
• Adding a new function convention: e.g. adding @convention(python), function conversion triggers translation of Swift AST to Python.

But both are very heavyweight.

[tanmayb123]

Interesting - thanks for sharing!

On a similar note, just like how you can call Python code from Swift, is there a way to call Swift from Python? If so, how difficult would it be to get that running? I see this: https://gist.github.com/jiaaro/e111f0f64d0cdb8aca38 but it's not working for me, for some reason.

Currently, with the Python->Swift interop (which is absolutely amazing btw), there are a few limitations around interacting with an existing Python codebase. I think going both ways could open up a lot more potential.

[dan-zheng]

Calling Swift from Python is a direction we've thought about! That direction is interesting because it allows large Python codebases to incrementally add Swift code, using features like differentiable programming. This seems more useful than calling numpy from Swift, for example.

One approach is to declare and export Swift @_cdecl functions and import them in Python. Python supports importing C-compatible functions. @_cdecl functions are severely constrained (small set of supported parameter/result types, must be top-level), but they can be wrapped in Swift (and in Python, after being imported) to be more friendly.

[rxwei]

I do not think passing Swift closures needs anything heavyweight like virtualization or type system changes. All we need is to get its function pointer and maintain the lifetime of the closure context using a PythonObject.

[cgarciae]

This would be super useful!

[eaplatanios]

I do not think passing Swift closures needs anything heavyweight like virtualization or type system changes. All we need is to get its function pointer and maintain the lifetime of the closure context using a PythonObject.

I do something like that to call Swift from C++ so I'm pretty sure it can be done relatively easily.

[biellls]

Any progress on that front? That would enable some pretty cool things (eg: I'd love to build an ETL tool in swift that wraps over Airflow, but it's not possible unless I can pass closures that can be called from python).

[avnerbarr]

Many python libraries have async function versions with callbacks. Not sure if it is possible to use that from swift. Any idea how to bridge it?

[aepryus]

I'm currently trying to maximize a swift function. It seems that scipy.optimize.minimize is a good way to do so, but I would need to be able to pass the Swift function to the Python library.

As hinted at in this thread, is this currently not possible? Does anyone have any other suggestions?

[graboosky]

Any progress in that matter?

[dabrahams]

It would be easy to make this work:

print(test.f(PythonObject({ z in pow(z, 2) }), 5))

if such cases were common enough to make that syntactically burdensome, one could create an operator to sugar it, but I don't think we're at that point yet:

print(test.f({ z in pow(z, 2) }^, 5))

Amendment: you would of course need to specify types in the closure signature, because there's nothing available to deduce them from:

print(test.f(PythonObject({ (z: Float) in pow(z, 2) }), 5))
print(test.f({ (z: Float) in pow(z, 2) }^, 5))

[avnerbarr]

we are talking about closures back to the swift code, not lambdas in python right?

[graboosky]

@dabrahams but how would you make closure responsive to 'PythonConvertible'?

[dabrahams]

@graboosky without language extensions to make non-nominal types such as functions and tuples able to (implicitly) conform to protocols, you wouldn't. So instead you'd pay the syntactic price of explicit wrapping in PythonObject or the use of an operator.

[graboosky]

I see. At this moment did someone experiment with C? I like the idea of passing C function pointers, I did callback with primitive parameters between 'swift <-> C'. I would like to see how to do that for 'swift <-> C <-> Python'. Unfortunately, Python is magic for me:) Gist welcome:)

[dabrahams]

I'm afraid I don't understand your question. Regardless, I don't know about all the experiments that have been done so I'm probably not the best person to answer it.

Sorry, Dave

On Fri, Mar 6, 2020 at 3:58 AM Patryk Grabowski [email protected] wrote:

I see. At this moment did someone experiment with C? I like the idea of passing C function pointers, I did callback with primitive parameters between 'swift <-> C'. I would like to see how to do that for 'swift <-> C <-> Python'. Unfortunately, Python is magic for me:) Gist welcome:)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tensorflow/swift/issues/211?email_source=notifications&email_token=AAAKYINYQJX24QCVPGATLDTRGDQNVA5CNFSM4H4JWDL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOBDSZY#issuecomment-595736935, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAKYIJDKO5PP2YYCR7HBV3RGDQNVANCNFSM4H4JWDLQ .

-- -Dave