seafowl icon indicating copy to clipboard operation
seafowl copied to clipboard

Add concurency to the partition writing routine

Open gruuya opened this issue 6 months ago • 0 comments

This PR introduces a new modality that allows for concurrent pulling and flushing into temp files from input plan's streams, while preserving the former sequential/non-concurrent modality as well (achieved by setting min_file_count_hint to None).

The actual speedup from this is quite modest: image

gruuya avatar Aug 23 '24 11:08 gruuya