raydp
raydp copied to clipboard
oap-project
when i create initialization twice ray,raydp , An error occurred。
My version: Ray Cluster:1.3.0 Raydp:0.3.0 Python: 3.6.4 when i create initialization twice ray,raydp , An error occurred。 first run is OK. bu start the second is error.
Caused by: java.net.BindException: Address already in use (Bind failed) at java.net.PlainSocketImpl.socketBind(Native Method) at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387) at java.net.ServerSocket.bind(ServerSocket.java:375) at py4j.GatewayServer.startSocket(GatewayServer.java:786)
I set config configs={"spark.ui.port":"xxx"} and configs={"spark.port.maxRetries":"100"}
It's invalid
please help me~~
You only need to init once. If you see this error, it is because the pervious driver is not closed properly. You can find the Java process ID and kill it.
You only need to init once. If you see this error, it is because the pervious driver is not closed properly. You can find the Java process ID and kill it.
What if I want to start two spark driver on the same time? What should I do?
@Bowen0729 , can you please share more about your use case and requirement?
Two users both use raydp in the same conda environment, one of them throw exception address already in use
@Bowen0729 , can you please share more about your use case and requirement?
The Java Gateway port is always 25333, why not a random port?
This sounds like an issue. So two users will run their Spark driver on the same node and connect to the same Ray cluster, right? Are you interested in submitting a PR for this?
This sounds like an issue. So two users will run their Spark driver on the same node and connect to the same Ray cluster, right? Are you interested in submitting a PR for this?
Sure, I will submit a PR recently. Thank you for your reply
@Bowen0729 @carsonwang In this case how do we distinguish the user call init_spark twice by mistake or by intention?
It's a issue about the java gateway always starts with the default port, so when user call init_spark twice in different session, the new java gateway cannot start.
when user call init_spark in the same session, raydp will throw "The spark environment has inited".
+1 on that issue. It's annoying since we are possibly running multiple ray+raydp instances on the same host... Especially when there are multiple users!
