dagster-polars icon indicating copy to clipboard operation
dagster-polars copied to clipboard

Look into writing Parquet Statistics into Dagster metadata

Open danielgafni opened this issue 1 year ago • 0 comments

https://arrow.apache.org/docs/python/generated/pyarrow.parquet.Statistics.html

Currently dagster-polars is calculating similar statistics manually. The pyarrow statistics might be useful, and they would always be consistent with the actual Parquet file, not the polars DataFrame

danielgafni avatar Aug 10 '23 14:08 danielgafni