polars icon indicating copy to clipboard operation
polars copied to clipboard

feat(rust): streaming parquet from object_stores

Open winding-lines opened this issue 1 year ago • 3 comments

Supports reading parquet files with the object_store crate. Currently the file and s3 protocols are linked in, gcs and azure can be added if this approach makes sense.

The only supported mode is batched since generally remote files are big. Setting a big chunk size will download all the file in one iteration.

This is my first PR in this repo, feedback appreciated.

winding-lines avatar Dec 21 '22 04:12 winding-lines