Add counter information to written parquet files
Based on the comment by Serge the parquet file path generation could use some cleanup to track a counter, and rely on the parquet file's writer properties for determining whether to add a snappy or not.
Based on Serge's comment, we also need to use the same uuid for all files from the same batch, otherwise the counter won't provide any value.
https://github.com/apache/spark/blob/0494dc90af48ce7da0625485a4dc6917a244d580/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/ManifestFileCommitProtocol.scala#L115-L120 here's good enough reference on conventional filename formatting.
I think this issue is already done with this PR. The writer function next_data_path takes care of correctly constructing the path for a Parquet file.
You are correct @Jan-Schweizer - will close :)