BigDL-2.x icon indicating copy to clipboard operation
BigDL-2.x copied to clipboard

bigDL zoo not support spark 3.0 sql dataframe ,please fix up these fault import location,

Open mullerhai opened this issue 2 years ago • 1 comments

Hi Sir: bigDL zoo not support spark 3.0 sql dataframe Version "com.intel.analytics.zoo" % "analytics-zoo-bigdl_0.12.1-spark_3.0.0" % "0.9.0" because when I use the

val criterion = MSECriterion() val estimator = NNEstimator(model, criterion) meet error :at NNEstimator class :Caused by: java.lang.ClassNotFoundException: org.apache.spark.ml.adapter.HasFeaturesCol after I check the import old spark ml utils class ,

import org.apache.spark.ml.adapter.{HasFeaturesCol, HasPredictionCol, SchemaUtils} actually in spark 3.0 these class at here

import org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol} import org.apache.spark.ml.util.SchemaUtils please fix up these fault import location, then upgrade the big_zoo package version to your maven repo thx

mullerhai avatar Sep 17 '21 03:09 mullerhai

hi mullerhai, We already use org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol} in Analytics Zoo. spark.ml.adapter.HasFeaturesCol actually extends that in Analytics Zoo https://github.com/intel-analytics/analytics-zoo/blob/master/zoo/src/main/scala/com/intel/analytics/zoo/pipeline/nnframes/SharedParamsAdapter.scala Could you please share your code and detailed error messages?

Le-Zheng avatar Sep 18 '21 01:09 Le-Zheng