incubator-gluten icon indicating copy to clipboard operation
incubator-gluten copied to clipboard

Memory Isolation mode uses only execution memory

Open JunhyungSong opened this issue 1 month ago • 0 comments

Backend

VL (Velox)

Bug description

Memory Isolation mode uses only execution memory while Spark can utilize storage memory when the storage memory is not used fully. In the description of COLUMNAR_MEMORY_ISOLATION, it says "can be used by each task to X, X = executor memory / max task slots" which doesn't match with its actual behavior. If the behavior is by design, the description needs to be updated to "execution memory / max task slots" to avoid confusion because "executor memory" means "spark.executor.memory".

Gluten version

No response

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs


JunhyungSong avatar Nov 20 '25 18:11 JunhyungSong