CCF icon indicating copy to clipboard operation
CCF copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

Improve parquet file writer

Open eddyashton opened this issue 2 years ago • 0 comments

Our Piccolo submitter produces slightly weird parquet files. We can improve them in multiple ways:

  • [ ] Compress the data written
  • [x] Write in fewer RowGroups
  • [ ] Don't write encoding statistics
  • [ ] Stream output

eddyashton avatar Aug 01 '23 10:08 eddyashton