deequ icon indicating copy to clipboard operation
deequ copied to clipboard

Problem when adding deequ dependency in build.sbt on IntelliJ

Open shuvamx99 opened this issue 2 years ago • 1 comments

Spark Version: 3.2.1 Scala Version : 2.13.8

This is what the error looks like :

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/spark/sql/SparkSession$ at deequ5$.main(Main.scala:11) at deequ5.main(Main.scala) Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.SparkSession$

And this is my build.sbt file

ThisBuild / version := "0.1.0-SNAPSHOT"

ThisBuild / scalaVersion := "2.13.8"

lazy val root = (project in file(".")) .settings( name := "deequ5" ) libraryDependencies += "org.apache.spark" %% "spark-core" % "3.2.1" % "provided" libraryDependencies += "org.apache.spark" %% "spark-sql" % "3.2.1" % "provided" libraryDependencies += "com.amazon.deequ" % "deequ" % "1.1.0_spark-3.0-scala-2.12" % "provided"exclude("org.apache.spark", "spark-core_2.11") exclude("org.apache.spark", "spark-sql_2.11")

shuvamx99 avatar Jun 09 '22 16:06 shuvamx99

Hi, why are you marking deequ as provided?

qingwei91 avatar Aug 04 '23 06:08 qingwei91