polars icon indicating copy to clipboard operation
polars copied to clipboard

feat(rust): integrate delta-rs

Open winding-lines opened this issue 3 years ago • 2 comments

Integrate the delta-rs library for interacting with the Delta reader.

https://github.com/pola-rs/polars/issues/2858

winding-lines avatar Nov 26 '22 13:11 winding-lines

I would also like this functionality on the Python side; though that would also be possible by using the Delta package and going through pyarrow.

stinodego avatar Dec 02 '22 18:12 stinodego

@stinodego in python you could use something like this

from deltalake import DeltaTable
import pyarrow.parquet as pq

def delta_to_pl(path: str):
    ds = pq.ParquetDataset(
        DeltaTable(path).file_uris()
    ).read()
    return pl.DataFrame(ds)

sa- avatar Dec 03 '22 16:12 sa-

Some parts of the PR merged, the rest not of interest.

winding-lines avatar Dec 26 '22 17:12 winding-lines