spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-45278][YARN] Support executor bind address in Yarn executors

Open gedeh opened this issue 5 months ago • 2 comments

What changes were proposed in this pull request?

Uptake --bind-address parameter in YarnCoarseGrainedExecutorBackend when launching new container in Yarn cluster. This PR also ensure YarnAllocator uses default hostname when its not configured.

Why are the changes needed?

We've came across issue with Spark running on Yarn in Istio enabled Kubernetes cluster. Previous PR https://github.com/apache/spark/pull/32633 is not merged because Spark 2.4 was EOL and 3.x branch didnt get enough traction.

Does this PR introduce any user-facing change?

Yes, new config specifically for Yarn cluster mode is added and relevant doc is updated.

How was this patch tested?

Tested in Kubenetes with Istio and added tests to YarnAllocatorSuite

Thanks!

gedeh avatar Aug 27 '24 14:08 gedeh