Le-Zheng

hi mullerhai, We already use `org.apache.spark.ml.param.shared.{HasFeaturesCol, HasPredictionCol}` in Analytics Zoo. `spark.ml.adapter.HasFeaturesCol ` actually extends that in Analytics Zoo https://github.com/intel-analytics/analytics-zoo/blob/master/zoo/src/main/scala/com/intel/analytics/zoo/pipeline/nnframes/SharedParamsAdapter.scala Could you please share your code and detailed error messages?

> # yolov3.py > ## client command > ``` > ${SPARK_HOME}/bin/spark-submit \ > --master ${RUNTIME_SPARK_MASTER} \ > --deploy-mode client \ > --conf spark.driver.host=172.16.0.200 \ > --conf spark.driver.port=54321 \ > --conf...

New error occurs: http://10.112.231.51:18080/job/BigDL-PRVN-caffe/1250/console Traceback: ../bigdl/caffe/test_caffe_layers.py:17: in from bigdl.dllib.nn.layer import * ../../bigdl/__init__.py:17: in from bigdl.dllib.utils.engine import prepare_env E ImportError: No module named utils.engine

- Caffe UT test failed: error: http://10.112.231.51:18080/job/BigDL-PRVN-caffe/1250/console - Lenet python example failed: Run `${submit} --master local[*] --conf "spark.serializer=org.apache.spark.serializer.JavaSerializer" --driver-memory 10g --executor-cores 14 --total-executor-cores 28 --py-files ${curdir}/dist/lib/bigdl-dllib-*-python-api.zip --jars ${jar_path} --properties-file ${curdir}/dist/conf/spark-bigdl.conf...

change local optimizer to distriOptimizer, runs into new error `Task 3 in stage 3.0 failed 4 times, most recent failure: Lost task 3.3 in stage 3.0 (TID 26) (172.30.27.4 executor...

@yushan111

Sample spark submit : ``` /opt/spark/bin/spark-submit \ --master ${RUNTIME_SPARK_MASTER} \ --deploy-mode cluster \ --name simplequery \ --conf spark.driver.memory=20g \ --conf spark.executor.cores=16 \ --conf spark.executor.memory=20g \ --conf spark.executor.instances=1 \ --conf spark.cores.max=16...

> Still facing the same error after I use Spark 2.4.6 version The spark shell command from bin/spark-shell-with-dllib.sh is: `spark-2.4.6-bin-hadoop2.6/bin/spark-shell --properties-file ~/bigdl/conf/spark-bigdl.conf --jars ~/bigdl/jars/bigdl-dllib-spark_2.4.6-2.1.0-SNAPSHOT.jar --conf spark.driver.extraClassPath=~/bigdl/jars/bigdl-dllib-spark_2.4.6-2.1.0-SNAPSHOT.jar --conf spark.executor.extraClassPath=~/bigdl/jars/bigdl-dllib-spark_2.4.6-2.1.0-SNAPSHOT.jar` > >...

Updated. The Spark 2.3 assembly zip we built is https://oss.sonatype.org/content/repositories/snapshots/com/intel/analytics/bigdl/bigdl-assembly-spark_2.3.1/2.1.0-SNAPSHOT/bigdl-assembly-spark_2.3.1-2.1.0-20220830.083242-1.zip.

http://10.112.231.51:18888/job/BigDL-PRVN-scalatest-zoo-Spark-2.4/577/testReport/com.intel.analytics.bigdl.dllib.keras.layers/LSTMSpec/ Jenkins test passes exactly.