pathway icon indicating copy to clipboard operation
pathway copied to clipboard

Add Apache Iceberg input connector

Open zxqfd555 opened this issue 1 year ago • 0 comments

Is your feature request related to a problem? Please describe. There is a need for Apache Iceberg input and output connectors. They need to work in both, streaming and static modes. In the case of a streaming mode, updates must be applied incrementally, without fully rereading the table.

Describe the solution you'd like There is a native Rust crate for Iceberg. It can be used for an Iceberg connection.

Describe alternatives you've considered Currently, Iceberg storage can only be accessed by implementing a custom Python input connector. However, this solution is not stable enough for production.

zxqfd555 avatar Dec 13 '24 21:12 zxqfd555