ibis icon indicating copy to clipboard operation
ibis copied to clipboard

feat: support iceberg read/write

Open mfatihaktas opened this issue 1 year ago • 0 comments

Description of changes

Explorative PR towards addressing https://github.com/ibis-project/ibis/issues/7712.

This draft PR

  • Aims to bring more clarity to the concerns raised in https://github.com/ibis-project/ibis/issues/7712#issuecomment-1850331739.
  • Implements ibis.read_iceberg() by
    • Fetching the files that constitute the Iceberg table one by one.
    • Registering the table into the default backend through an RecordBatchReader.
  • Implements table.to_iceberg() by
    • Writing into the Iceberg table through RecordBatch's.

pyiceberg dependency

Notice the following change in environment.yml

- pip:
     - git+https://github.com/apache/iceberg-python.git@main

This is because write support in pyiceberg is merged into main but not released yet:

  • https://github.com/apache/iceberg-python/pull/41
  • https://github.com/apache/iceberg-python/issues/23

Looks like the next release is on the way:

Issues closed

mfatihaktas avatar Feb 13 '24 23:02 mfatihaktas