Zhi Lin

No, this PR is for 2.0.0. It's not compatible with 1.3.0, [`setJvmOptions`](https://github.com/ray-project/ray/blob/releases/1.3.0/java/api/src/main/java/io/ray/api/call/ActorCreator.java#L30) is not yet changed in 1.3.0.

If we use a normal task, which calls the actor internally, the actor method still need to be resubmitted because it's a dependency. I'm not sure if using spark to...

Hi @pang-wu , thanks for opening this issue. We'll look into this. It seems like some options are not copied to executors.

You can check the spark UI at port 4040 to see executors' IP. Or you can check ray's dashboard (default 8265) to see which actor is on which node, etc.

Hello, Glad you tried RayDP. I'm not sure, but I guess the exception is not directly caused by RayDP. What version of Spark are you using? The notebook you provide...

I see. My suggestion is that you can try it on a non-raydp spark cluster, like local mode or standalone. If it works on those but not raydp, we'll check...

@SrilekhaIG Glad to see the problem is solved. This is indeed a workaround, we'll see why spark.jars.packages fails in RayDP. But you can use raydp.executor.extraclasspath for now.

Hi @tdeboer-ilmn , Glad you tried raydp. You are right, raydp.init_spark is assumed to be called in the ray cluster. If you need to use ray client, for the current...

@carsonwang the test still hangs sometime. Shall we skip the tests?

yes, it hangs sometime and when I ctrl-c it, it shows popen_fork, even if I should have set start_method to spawn.