dagster-polars
dagster-polars copied to clipboard
Look into writing Parquet Statistics into Dagster metadata
https://arrow.apache.org/docs/python/generated/pyarrow.parquet.Statistics.html
Currently dagster-polars
is calculating similar statistics manually.
The pyarrow
statistics might be useful, and they would always be consistent with the actual Parquet file, not the polars DataFrame