incubator-gluten
incubator-gluten copied to clipboard
Memory Isolation mode uses only execution memory
Backend
VL (Velox)
Bug description
Memory Isolation mode uses only execution memory while Spark can utilize storage memory when the storage memory is not used fully. In the description of COLUMNAR_MEMORY_ISOLATION, it says "can be used by each task to X, X = executor memory / max task slots" which doesn't match with its actual behavior. If the behavior is by design, the description needs to be updated to "execution memory / max task slots" to avoid confusion because "executor memory" means "spark.executor.memory".
Gluten version
No response
Spark version
None
Spark configurations
No response
System information
No response
Relevant logs