datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Published 20 hours ago • verified publisher icon apache

feat: Use fair-spill pool when `spark.memory.offHeap.enabled=false`

Open andygrove opened this issue 4 months ago • 5 comments

Which issue does this PR close?

Closes https://github.com/apache/datafusion-comet/issues/996

Possibly depends on https://github.com/apache/datafusion-comet/pull/988 being resolved first

Rationale for this change

This is an alternative to the approach in https://github.com/apache/datafusion-comet/pull/1002

What changes are included in this PR?

  • Take task concurrency into account when allocating memory per native plan
  • Use fair spool pool instead of greedy pool

How are these changes tested?

andygrove avatar Oct 08 '24 18:10 andygrove