Not enough spark off-heap execution memory

[githhhub570]

We often find that spark overflows to disk when it runs out of memory. When NSE applies for off-heap memory, if it is insufficient, an exception will be thrown, causing the task to fail. We hope that regardless of whether the memory is sufficient or the performance, the task must be completed smoothly, rather than failed. Given this, is there a more efficient way to manage memory? Thanks!

[zhouyuan]

Hi @githhhub570

agreed, all quereis should run successfully first, even with lower performance. I think the issue is mostly due to lacking of Spill support for Sort/HashAgg. We've made some progress on sort spill support and may need some further cleanups. Hashagg spill will be the next feature.

related: #1070 https://github.com/oap-project/gazelle_plugin/issues/1085

thanks, -yuan