DLlib incompatible version with spark

[OneKay]

Try to use DLlib on Spark I just run bin/spark-shell-with-dllib.sh, and it reports an incompatible version error Spark version:2.3.2.3.1.5.0-152 BigDL version: bigdl-assembly-spark_2.4.6-2.1.0-20220729.043502-156.zip Is it because the low version of Spark?

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.3.2.3.1.5.0-152
      /_/

Using Scala version 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_291)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import com.intel.analytics.bigdl.dllib.NNContext
import com.intel.analytics.bigdl.dllib.NNContext

scala> val sc = NNContext.initNNContext("Run Example")
<console>:22: error: Symbol 'term org.apache.logging' is missing from the classpath.
This symbol is required by 'value com.intel.analytics.bigdl.dllib.NNContext.logger'.
Make sure that term logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'NNContext.class' was compiled against an incompatible version of org.apache.
class $iw extends Serializable {

[jason-dai]

Yes, Spark 2.4.x is needed for this version

[OneKay]

Still facing the same error after I use Spark 2.4.6 version The spark shell command from bin/spark-shell-with-dllib.sh is: spark-2.4.6-bin-hadoop2.6/bin/spark-shell --properties-file ~/bigdl/conf/spark-bigdl.conf --jars ~/bigdl/jars/bigdl-dllib-spark_2.4.6-2.1.0-SNAPSHOT.jar --conf spark.driver.extraClassPath=~/bigdl/jars/bigdl-dllib-spark_2.4.6-2.1.0-SNAPSHOT.jar --conf spark.executor.extraClassPath=~/bigdl/jars/bigdl-dllib-spark_2.4.6-2.1.0-SNAPSHOT.jar

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.6
      /_/

Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import com.intel.analytics.bigdl.dllib.NNContext
import com.intel.analytics.bigdl.dllib.NNContext

scala>

scala> val sc = NNContext.initNNContext("Run Example")
<console>:22: error: Symbol 'term org.apache.logging' is missing from the classpath.
This symbol is required by 'value com.intel.analytics.bigdl.dllib.NNContext.logger'.
Make sure that term logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'NNContext.class' was compiled against an incompatible version of org.apache.
class $iw extends Serializable {

[Le-Zheng]

Still facing the same error after I use Spark 2.4.6 version The spark shell command from bin/spark-shell-with-dllib.sh is: spark-2.4.6-bin-hadoop2.6/bin/spark-shell --properties-file ~/bigdl/conf/spark-bigdl.conf --jars ~/bigdl/jars/bigdl-dllib-spark_2.4.6-2.1.0-SNAPSHOT.jar --conf spark.driver.extraClassPath=~/bigdl/jars/bigdl-dllib-spark_2.4.6-2.1.0-SNAPSHOT.jar --conf spark.executor.extraClassPath=~/bigdl/jars/bigdl-dllib-spark_2.4.6-2.1.0-SNAPSHOT.jar

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.6
      /_/

Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_212)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import com.intel.analytics.bigdl.dllib.NNContext
import com.intel.analytics.bigdl.dllib.NNContext

scala>

scala> val sc = NNContext.initNNContext("Run Example")
<console>:22: error: Symbol 'term org.apache.logging' is missing from the classpath.
This symbol is required by 'value com.intel.analytics.bigdl.dllib.NNContext.logger'.
Make sure that term logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'NNContext.class' was compiled against an incompatible version of org.apache.
class $iw extends Serializable {

Hi @OneKay Thanks for your issue. I guess the assembly zip you downloaded is out of updated. It seems the dependency of org.apache.logging class is missing in the classpath. In the latest BigDL assembly zip https://oss.sonatype.org/content/repositories/snapshots/com/intel/analytics/bigdl/bigdl-assembly-spark_2.4.6/2.1.0-SNAPSHOT/bigdl-assembly-spark_2.4.6-2.1.0-20220821.152819-180.zip, bin/spark-shell-with-dllib.sh specifies ${BIGDL_HOME}/jars/* as default path.

[OneKay]

thx, the latest assembly zip solved the problem.

[Le-Zheng]

Updated. The Spark 2.3 assembly zip we built is https://oss.sonatype.org/content/repositories/snapshots/com/intel/analytics/bigdl/bigdl-assembly-spark_2.3.1/2.1.0-SNAPSHOT/bigdl-assembly-spark_2.3.1-2.1.0-20220830.083242-1.zip.

[OneKay]

Updated. The Spark 2.3 assembly zip we built is https://oss.sonatype.org/content/repositories/snapshots/com/intel/analytics/bigdl/bigdl-assembly-spark_2.3.1/2.1.0-SNAPSHOT/bigdl-assembly-spark_2.3.1-2.1.0-20220830.083242-1.zip.

ok, I'll have a try