AbstractMethodError in Spark 2.3.0

[Majid-Soheili]

I would like to discretize the Epsilon dataset that it has 2k features and 400k records. To achieve this, I utilized the Spark 2.3.0. Once I execute the code, I deal with below error. It is noteworthy this error had not occurred when I used Spark 2.2.0, but It seems that the Spark 2.2.0 is not suitable for a high dimensional dataset. If is it possible, please fix this problem.

######################################################### Exception in thread "main" java.lang.AbstractMethodError at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99) at org.apache.spark.mllib.feature.MDLPDiscretizer.initializeLogIfNecessary(MDLPDiscretizer.scala:48) at org.apache.spark.internal.Logging$class.log(Logging.scala:46) at org.apache.spark.mllib.feature.MDLPDiscretizer.log(MDLPDiscretizer.scala:48) at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:54) at org.apache.spark.mllib.feature.MDLPDiscretizer.logInfo(MDLPDiscretizer.scala:48) at org.apache.spark.mllib.feature.MDLPDiscretizer.runAll(MDLPDiscretizer.scala:106) at org.apache.spark.mllib.feature.MDLPDiscretizer$.train(MDLPDiscretizer.scala:335) at org.apache.spark.ml.feature.MDLPDiscretizer.fit(MDLPDiscretizer.scala:149) #########################################################

[jfgosselin]

+1 I ran into the same issue today . Is there any plan for an upgrade to Spark 2.3.0 ?

[jfgosselin]

I've recompiled spark-MDLP-discretization with Spark 2.3.2 to and Scala 2.11.2 to solve the issue.

[Majid-Soheili]

I've recompiled spark-MDLP-discretization with Spark 2.3.2 and Scala 2.11.2 to solve the issue.

Hi Jean, If it is possible for you, share your compiled library. Also, there is another issue on the last version on the library that is so important in my impression, The issue and the solution were reported in the address below.

https://github.com/sramirez/spark-MDLP-discretization/issues/36

If it is possible for you, please fix the issue and share it. Thanks a lot.