apache
Rust <> Python integration point
After establishing #518, I want to start the conversation to create the first integration between PyIceberg and iceberg-rust. As discussed in the dev list, we want to create an integration based on pluggable FileIO.
I'm wondering if there's also a way to create an integration for a pluggable catalog, based on the in-memory catalog implementation in #475.
I'm not familiar with the rust ecosystem, so would appreciate any pointers
I'm wondering if there's also a way to create an integration for a pluggable catalog, based on the in-memory catalog implementation in #475.
I believe this should also be possible. So, the pyiceberg community wants to have an in-memory catalog based on iceberg-rust. Does pyiceberg provide an interface that we can integrate with?
The in-memory catalog depends on FileIO, so we might need to build FileIO first. However, it also makes sense to expose a purely in-memory catalog (memory FileIO and memory catalog) to pyiceberg initially.
I think it's definitely possible since PyIceberg is Catalog interface is extensible. I think you need to start with pyo3 first to understand how it works.
Does pyiceberg provide an interface that we can integrate with?
Yes, there is a py-catalog-impl configuration that will try to load a given classpath. (documentation, implementation, test)
The in-memory catalog depends on FileIO, so we might need to build FileIO first. However, it also makes sense to expose a purely in-memory catalog (memory FileIO and memory catalog) to pyiceberg initially.
I'm bringing up this issue because I want the simplest way to integrate iceberg-python and iceberg-rust. If FileIO integration is a prerequisite, we can start there instead.
Hi, @kevinjqliu, I'm sorry for blocking your innovation this way.
I've been a bit busy recently, but I plan to create something that really works next week. For instance, reading data from PyIceberg using pyiceberg-core. This will enable our community to build more cool things based on that.
Looks like @sungwy already started by exposing Transforms in #556
I'll take a stab at exposing the Catalogs, see https://github.com/apache/iceberg-rust/pull/534#issuecomment-2330489500