incubator-xtable icon indicating copy to clipboard operation
incubator-xtable copied to clipboard

Add Python wrappers or create interoperability to call the classes/methods using Python

Open sagarlakshmipathy opened this issue 1 year ago • 4 comments

Data engineers quite heavily rely on Python for creating data pipelines. Its important to support the ability to call the Java objects using Python (through Py4J or similar tools/libraries). Right now, users can only use JVM languages out of the box if they were to use OneTable directly i.e. as shown in docker demo.

sagarlakshmipathy avatar Nov 20 '23 19:11 sagarlakshmipathy

Why not also non-jvm and then python bindings to that language.

ion-elgreco avatar Dec 28 '23 12:12 ion-elgreco

@ion-elgreco any language you are looking for in particular?

Right now, the implementation relies on java since that is the only common language for interacting with tables in the major table formats. We can also consider a deployment model where this code is behind some lightweight server and build an API that users can call. Generating clients in various languages is pretty straightforward so I'm curious what people think of this idea.

the-other-tim-brown avatar Dec 28 '23 16:12 the-other-tim-brown

Rust in particular and then using pyo3 bindings. We already do this at Delta-RS

ion-elgreco avatar Dec 28 '23 16:12 ion-elgreco

I think what would be great is some pyspark examples which show how to invoke onetable

soumilshah1995 avatar Mar 10 '24 13:03 soumilshah1995