TransmogrifAI
TransmogrifAI copied to clipboard
Failed to run titanic example, got java.lang.AbstractMethodError
Describe the bug Exception in thread "main" java.lang.AbstractMethodError
To Reproduce
./gradlew -q sparkSubmit -Dmain=com.salesforce.hw.OpTitanicSimple -Dargs="
pwd
/src/main/resources/TitanicDataset/TitanicPassengersTrainData.csv"
Expected behavior Successfully run this example
Logs or screenshots
Exception in thread "main" java.lang.AbstractMethodError
at org.apache.spark.ml.param.Params$class.$init$(params.scala:868)
at com.salesforce.op.stages.impl.tuning.Splitter.
FAILURE: Build failed with an exception.
- What went wrong: Execution failed for task ':sparkSubmit'.
Process 'command '/home/user/spark_workspace/spark-project/bin/spark-submit'' finished with non-zero exit value 1
-
Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
-
Get more help at https://help.gradle.org
BUILD FAILED in 4s
Additional context
I am using version 0.7.0. My spark version is 2.4.5, and my hadoop version is 2.7.7. From what I saw on other issues, the cause of this error seems to be incompatible spark version, however my spark version should be correct.
I have tried using spark 2.4.7, but the results were the same.
I did run ./gradlew compileTestScala installDist
in helloworld
folder and TransmogrifAI
folder before submitting the example, and I am under helloworld
folder when I submitted the command.
I tried submitting it using $SPARK_HOME/bin/spark-submit --class com.salesforce.hw.titanic.OpTitanic build/libs/transmogrifai-helloworld-0.0.1-all.jar --run-type train --model-location /tmp/titanic-model --read-location Passenger=
pwd/src/main/resources/TitanicDataset/TitanicPassengersTrainData.csv
command, but the results were similar.
My java version is 1.8.0_161, and I have set JAVA_HOME correctly.
I did modifiy gradle/wrapper/gradle-wrapper.properties
, so that I could use a local gradle of version 5.2.1. Also I have modified the repositories in build.gradle
to two faster mirrors. I am not sure if this is related to this problem.
I am running this example on a pseudo-cluster on a freshly configured CentOS7 virtual machine. Is this problem caused by the lack of some necessary libraries?
Which Scala version are you using?
I think it was scala 2.11.8, but I can't remember very clearly. I don't have access to the device I was using anymore. I was pretty sure it was a version of scala 2.11 though.