apache
InternalParquetRecordWriter flushRowGroupToStore not actually writing to file
Describe the bug, including details regarding any error messages, version, and platform.
Description: The flushRowGroupToStore method in InternalParquetRecordWriter purports to "Flush mem columnStore to file," but no data is actually written to the file system until the close method is invoked. This behavior is unexpected, as one would anticipate data being flushed.
Expected Result: Data should be written to the file when flushRowGroupToStore is invoked, effectively flushing the in-memory column store contents to the file.
Actual Result: File contents remain empty until the close method is called, where parquetFileWriter.end is invoked to write the data.
Version : parquet-hadoop 1.13.1
Component(s)
No response
I think this is expected from the layered design. The InternalParquetRecordWriter.flushRowGroupToStore() method simply flushes all column chunks to the ParquetFileWriter which forwards the write to the PositionOutputStream. The PositionOutputStream decides the timing to flush the data to the file.
@wgtmac - During concurrent runs, such as loading 60 million rows across 10 writer jobs, we've noticed significant performance drops and failures. Despite configuring various flush sizes, the data doesn't seem to flush based on the set size, as observed in the created file.
We've checked, and there are no memory issues or garbage collection problems. It doesn't appear that the data is kept in-memory within the client application, nor is it visible in the file during the writing process. Therefore, we want to understand where the data is stored between flushes if it isn't being flushed in chunks as expected.
The data seems to be written to the file only at the end, which is likely causing a performance bottleneck.
Could you please check how many row groups in a single Parquet file? Is it a single row group per file? Usually the entire row group is flushed when flushRowGroupToStore is called unless the PositionOutputStream instance the writer uses has internal buffers to avoid small write.
We've checked, and there are no memory issues or garbage collection problems.
If you use DirectCodecFactory, the content is kept in the direct memory.
I have a similar performance issue in writing, my object size is 20KB each and I am using java recordconsumer api from apache-arrow to write parquet file. I found that it write row-by-row underneath calling writer method of parquet when flushing data to columns. Do you know how to overcome this bottleneck? (My 20KB object has 400 params and 9 images raw blob)
