datafusion-comet icon indicating copy to clipboard operation
datafusion-comet copied to clipboard

Implement shared memory pool when native memory management is enabled

Open andygrove opened this issue 1 year ago • 0 comments

What is the problem the feature request solves?

As described in detail in https://github.com/apache/datafusion-comet/issues/949, when using native memory management, we currently create one pool per plan and do not take executor concurrency into account when setting spark.executor.memoryOverhead.

Describe the potential solution

This issue is for implementing a shared pool between native plans within an executor.

Additional context

No response

andygrove avatar Oct 03 '24 17:10 andygrove