parquet4s icon indicating copy to clipboard operation
parquet4s copied to clipboard

Published 20 hours ago • verified publisher icon mjakubowski84

Output path's for files in partition

Open MaksGS09 opened this issue 2 years ago • 1 comments

Hi! I'm write partitioned parquet via fs2 stream. Is there a way to get the full output path (with file name included) after writing records batch? I found several options in the com.github.mjakubowski84.parquet4s.parquet.PostWriteHandler but it doesn't include output path.

MaksGS09 avatar Sep 14 '22 16:09 MaksGS09

Unfortunately, file names are not exposed by PostWriteHandler. You have access only to partition path. Mind that when the batch is written more than a single file can be modified - even per partition.

mjakubowski84 avatar Sep 17 '22 17:09 mjakubowski84