spark
spark copied to clipboard
[SPARK-45278][YARN] Support executor bind address in Yarn executors
What changes were proposed in this pull request?
Uptake --bind-address
parameter in YarnCoarseGrainedExecutorBackend
when launching new container in Yarn cluster. This PR also ensure YarnAllocator
uses default hostname when its not configured.
Why are the changes needed?
We've came across issue with Spark running on Yarn in Istio enabled Kubernetes cluster. Previous PR https://github.com/apache/spark/pull/32633 is not merged because Spark 2.4 was EOL and 3.x branch didnt get enough traction.
Does this PR introduce any user-facing change?
Yes, new config specifically for Yarn cluster mode is added and relevant doc is updated.
How was this patch tested?
Tested in Kubenetes with Istio and added tests to YarnAllocatorSuite
Thanks!