Having worker nodes specific for Spark jobs

[vmagdin]

Hi there, I am continuing to enjoy using this project, it's great!

I wish to have a heterogeneous clusters where I have some GPU worker nodes, a few CPU-heavy worker nodes for HPO jobs, and many smaller instances for Spark jobs. I know that it's possible to declare custom resource markers on nodes in Ray. However, I don't think I see a way to get raydp to request those custom resources. Am I missing something, or is this not possible?

Thanks!

[ConeyLiu]

Do you want the spark executor to request those custom resources?

[carsonwang]

Spark supports custom resource scheduling, we can request custom resources for the executor(s) using configuration spark.executor.resource.{resourceName}.amount. What we need to do is pass these resource requirement to Ray.