deequ
deequ copied to clipboard
Problem when adding deequ dependency in build.sbt on IntelliJ
Spark Version: 3.2.1 Scala Version : 2.13.8
This is what the error looks like :
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession$ at deequ5$.main(Main.scala:11) at deequ5.main(Main.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession$
And this is my build.sbt file
ThisBuild / version := "0.1.0-SNAPSHOT"
ThisBuild / scalaVersion := "2.13.8"
lazy val root = (project in file(".")) .settings( name := "deequ5" ) libraryDependencies += "org.apache.spark" %% "spark-core" % "3.2.1" % "provided" libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.2.1" % "provided" libraryDependencies += "com.amazon.deequ" % "deequ" % "1.1.0_spark-3.0-scala-2.12" % "provided"exclude("org.apache.spark", "spark-core_2.11") exclude("org.apache.spark", "spark-sql_2.11")
Hi, why are you marking deequ
as provided?