datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

Add spilling support for HashJoin

Open comphead opened this issue 1 year ago • 1 comments

Is your feature request related to a problem or challenge?

It would be helpful to have spilling support for the HashJoin. If there is not enough memory on the machine the join can leverage local disk to spill intermittent results

For reference the spilling support for SortMergeJoin https://github.com/apache/datafusion/issues/9359

Some ideas was also covered in https://github.com/apache/datafusion/issues/1599

More reading also https://facebookincubator.github.io/velox/develop/spilling.html

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

comphead avatar Oct 15 '24 21:10 comphead

@andygrove @viirya cc

comphead avatar Oct 15 '24 21:10 comphead

take

dmitriibugakov avatar Oct 27 '24 18:10 dmitriibugakov