BigDL-2.x
BigDL-2.x copied to clipboard
bigDL zoo not support spark 3.0 sql dataframe ,please fix up these fault import location,
Hi Sir: bigDL zoo not support spark 3.0 sql dataframe Version "com.intel.analytics.zoo" % "analytics-zoo-bigdl_0.12.1-spark_3.0.0" % "0.9.0" because when I use the
val criterion = MSECriterion() val estimator = NNEstimator(model, criterion) meet error :at NNEstimator class :Caused by: java.lang.ClassNotFoundException: org.apache.spark.ml.adapter.HasFeaturesCol after I check the import old spark ml utils class ,
import org.apache.spark.ml.adapter.{HasFeaturesCol, HasPredictionCol, SchemaUtils} actually in spark 3.0 these class at here
import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol} import org.apache.spark.ml.util.SchemaUtils please fix up these fault import location, then upgrade the big_zoo package version to your maven repo thx
hi mullerhai,
We already use org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}
in Analytics Zoo.
spark.ml.adapter.HasFeaturesCol
actually extends that in Analytics Zoo https://github.com/intel-analytics/analytics-zoo/blob/master/zoo/src/main/scala/com/intel/analytics/zoo/pipeline/nnframes/SharedParamsAdapter.scala
Could you please share your code and detailed error messages?