ibis
ibis copied to clipboard
feat: support iceberg read/write
Description of changes
Explorative PR towards addressing https://github.com/ibis-project/ibis/issues/7712.
This draft PR
- Aims to bring more clarity to the concerns raised in https://github.com/ibis-project/ibis/issues/7712#issuecomment-1850331739.
- Implements
ibis.read_iceberg()by- Fetching the files that constitute the Iceberg table one by one.
- Registering the table into the default backend through an
RecordBatchReader.
- Implements
table.to_iceberg()by- Writing into the Iceberg table through
RecordBatch's.
- Writing into the Iceberg table through
pyiceberg dependency
Notice the following change in environment.yml
- pip:
- git+https://github.com/apache/iceberg-python.git@main
This is because write support in pyiceberg is merged into main but not released yet:
- https://github.com/apache/iceberg-python/pull/41
- https://github.com/apache/iceberg-python/issues/23
Looks like the next release is on the way: