joblib-spark icon indicating copy to clipboard operation
joblib-spark copied to clipboard

[Bug] Method cpus does not exist, version 0.5.3 doesn't work with spark 3.4 or 3.5

Open thinkall opened this issue 1 year ago • 1 comments

It seems that PR #52 introduced a bug which made it fails on spark 3.4 and 3.5.

image

Py4JError: An error occurred while calling o8485.cpus. Trace:
py4j.Py4JException: Method cpus([class java.lang.Integer]) does not exist
	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:321)
	at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:329)
	at py4j.Gateway.invoke(Gateway.java:274)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.base/java.lang.Thread.run(Thread.java:829)

@lu-wang-dl , @WeichenXu123 , could you please take a look at this? Thank you very much!

thinkall avatar Aug 15 '24 10:08 thinkall

@thinkall, can you share the example that can repro this issue?

wbo4958 avatar Sep 10 '24 02:09 wbo4958