seafowl
seafowl copied to clipboard
Add concurency to the partition writing routine
This PR introduces a new modality that allows for concurrent pulling and flushing into temp files from input plan's streams, while preserving the former sequential/non-concurrent modality as well (achieved by setting min_file_count_hint
to None
).
The actual speedup from this is quite modest: