polars icon indicating copy to clipboard operation
polars copied to clipboard

Published 20 hours ago • verified publisher icon pola-rs

Write support for Apache Iceberg

Open randypitcherii opened this issue 1 year ago • 2 comments

Description

I love the lazy reading for Iceberg. It's great.

If Polars support writes back to an Iceberg catalog, that would make it a really powerful too working alongside the sql engines and spark dataframes I'm usually stuck using with Iceberg.

image

Thanks!

randypitcherii avatar Feb 20 '24 16:02 randypitcherii

Good timing on this issue, as I see pyiceberg write support just made its public debut^1; congrats @Fokko! Not sure how it might best integrate via Polars, but could certainly be interesting ;)

alexander-beedie avatar Feb 20 '24 18:02 alexander-beedie

Great news!

I spoke with Fokko today (he rules) and he showed me that polars will let me get an arrow table. With that, pyiceberg allegedly will write to any arbitrary iceberg target : )

So maybe I don't need built in write support in polars if its that easy. I'm going to give it a go and see.

I also took a peek at the write_delta function for inspiration but it was wayyyy over my head.

randypitcherii avatar Feb 20 '24 22:02 randypitcherii