Wrong driverHostname for Application UI

[atomobianco]

trafficstars

Hi, I am using your wonderful docker images. When I moved up from a cluster with

spark-base:2.2.0-hadoop2.8-hive-java8

to a cluster with the most recent version:

spark-master:2.3.1-hadoop2.7

I notice that the Web UI is no more a valid URL. Before that was giving the http://<worker ip>:4040, while now it's a strange http://a585a25be0f7:4040 which is not reachable (I have to write it by hand).

Do you know what could be the cause?

[earthquakesan]

@atomobianco hi!

what you see there as a hostname a585a25be0f7 is a docker container id (default internal docker hostname representation). You may try to plan with spark.driver.bindAddress configuration option to set it back.

Did you change anything else beside the image version?

[jgoodman8]

Could you solved @atomobianco ?

[atomobianco]

@earthquakesan Thanks for the tip I did not try it yet, but to answer your question no, I only did change the image version. @jgoodman8 not yet solved

[atomobianco]

Hello, coming back on this problem as I am updating the stack to Spark 2.3.2.

As far as I understood @earthquakesan you suggest to submit the job with spark.driver.bindAddress; the problem is that as I am submitting in cluster mode, I don't know which worker will be the driver, therefore I can't set a priori the address, or I am mistaken?

[atomobianco]

For information, here two Spark UI environmental variables taken from the previous version and the current version:

with bde2020/spark-base:2.2.0-hadoop2.8-hive-java8

spark.driver.host	172.24.0.5
spark.master	spark://e0e2e4fea039:7077

with bde2020/spark-base:2.3.2-hadoop2.8

spark.driver.host	afe3b572290b
spark.master	spark://39b10c64d584:7077

[atomobianco]

The workaround I did for the moment is rewrite the /etc/hosts in the worker nodes:

The old one was 172.24.0.5 afe3b572290b

The new one is 172.24.0.5 172.24.0.5

This rewriting is performed inside worker.sh script, which is the Docker entry point for each worker node, so that each worker container gets its IP at run time.

[jgoodman8]

That sounds good. I will try!

[atomobianco]

This line within worker.sh should work:

# Substitute container id with current IP
sed -r "s/$(hostname -i).+/$(hostname -i) $(hostname -i)/" /etc/hosts > /tmp/.intermediate-file-2431; cp tmp/.intermediate-file-2431 /etc/hosts;

[tpsimoes]

This line within worker.sh should work:

# Substitute container id with current IP
sed -r "s/$(hostname -i).+/$(hostname -i) $(hostname -i)/" /etc/hosts > /tmp/.intermediate-file-2431; cp tmp/.intermediate-file-2431 /etc/hosts;

Hi atomobianco,

I'm creating my own image and i have similar issue ... the hostname have the Container ID, and i'm trying (withou success) to change it ... to the IP for example ...