Question regarding Dynamic Executor Allocation support for Spark

[dpittner]

What: https://github.com/Intel-bigdata/oap-raydp/search?q=dynamic suggests that dynamic executor allocation may be added later.

Are there plans to add this, are there fundamental challenges in the ray context for supporting this?

[ConeyLiu]

Hi @dpittner. Yeah, we have not supported dynamic executor allocation currently. We will add it support recently. If you use spark on ray only, it should be easy to add the support. If you want to use spark on ray couple with estimator API, we should add the support of ray object ownership transfer.

[kira-lin]

close as stale. This has been supported