spark-deep-learning icon indicating copy to clipboard operation
spark-deep-learning copied to clipboard

Published 20 hours ago • verified publisher icon databricks

ERROR When running python unit tests

Open zsellami opened this issue 7 years ago • 6 comments

I run the unit tests command sparkdl$ SPARK_HOME=/usr/local/lib/spark-2.1.1-bin-hadoop2.7 PYSPARK_PYTHON=python2 SCALA_VERSION=2.11.8 SPARK_VERSION=2.1.1 ./python/run-tests.sh but I obtained the next error : py4j.java_gateway: DEBUG: Answer received: !xspy4j.Py4JException: Method listToVectorFunction([class org.apache.spark.sql.Column]) does not exist\n at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)\n at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)\n at py4j.Gateway.invoke(Gateway.java:272)\n at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)\n at py4j.commands.CallCommand.execute(CallCommand.java:79)\n at py4j.GatewayConnection.run(GatewayConnection.java:214)\n at java.lang.Thread.run(Thread.java:748)\n

How can I deal with this ?

zsellami avatar Jun 20 '17 10:06 zsellami

It should be fixed in master now (with https://github.com/databricks/spark-deep-learning/pull/20) !

sueann avatar Jun 20 '17 21:06 sueann

Please do reopen if you're still seeing problems. Thanks!

sueann avatar Jun 21 '17 18:06 sueann

I am not using python3. I use python 2.7

zsellami avatar Jun 22 '17 07:06 zsellami

@zsellami did you make some modifications to the scala part of the code? From the stack trace, a java object could be created but a method could not be found. You should make sure that the java part is recompiled by running build/sbt assembly.

If this does not work, can you please upload the full stack trace using https://gist.github.com ?

thunterdb avatar Jun 22 '17 16:06 thunterdb

I don't make any modification in the code. run build/sbt assembly work without any problem sparkdl$ SPARK_HOME=/usr/local/lib/spark-2.1.1-bin-hadoop2.7 PYSPARK_PYTHON=python2 SCALA_VERSION=2.11.8 SPARK_VERSION=2.1.1 ./python/run-tests.sh failed after some step. I upload the full stack trace of the shell. errorSparkDL.txt

zsellami avatar Jun 23 '17 07:06 zsellami

@zsellami thank you for keeping updated with this issue. Would you like to pull the master branch and try the unit test again? git rev-parse HEAD should return dee8fbff74c807bdbf72edf3588d4d5f64f36b0b Please also run ./build/sbt assembly and pip2 install -r python/requirements.txt from the project root directory. I tested with these locally on a clean branch and the all the tests passed.

phi-dbq avatar Jun 23 '17 17:06 phi-dbq