Issue: Anomaly Detection - HoltWinters fail with SBT

[pawelpinkos]

Issue:

There is an issue on Anomaly Detection: HoltWinter feature related to breeze dependency. Deequ depends on breeze in version 0.13.2 and there is a confilct with dependency of spark-mllib which also depends on breeze but in version 1.2.

You can notice the issue depends on build tool you use. In maven seems to works fine because maven use explicite version from deequ dependency (0.13.2):

mvn dependency:tree | grep breeze

[INFO] +- org.scalanlp:breeze_2.12:jar:0.13.2:compile
[INFO] |  +- org.scalanlp:breeze-macros_2.12:jar:0.13.2:compile

But sbt choosing newer version (1.2) and try to use breeze 0.13.2 using api from spark-mllib (version 1.2) and then fail with exception:

sbt dependencyTree | grep breeze

[info] deequbreezedebug:deequbreezedebug_2.12:0.1.0-SNAPSHOT
[info]     | | | +-org.scalanlp:breeze_2.12:1.2 [S]
[info]     | | | | +-org.scalanlp:breeze-macros_2.12:1.2 [S]
[info]     | | +-org.scalanlp:breeze_2.12:1.2 [S]
[info]     | | | +-org.scalanlp:breeze-macros_2.12:1.2 [S]
[info]     | +-org.scalanlp:breeze_2.12:1.2 [S]
[info]     | | +-org.scalanlp:breeze-macros_2.12:1.2 [S]
[info]     +-org.scalanlp:breeze_2.12:0.13.2 (evicted by: 1.2)
[info]     +-org.scalanlp:breeze_2.12:1.2 [S]
[info]       +-org.scalanlp:breeze-macros_2.12:1.2 [S]

Exception:

Exception in thread "main" java.lang.NoClassDefFoundError: breeze/stats/package$
	at com.amazon.deequ.anomalydetection.seasonal.HoltWinters.detect(HoltWinters.scala:242)
	at com.amazon.deequ.anomalydetection.AnomalyDetector.detectAnomaliesInHistory(AnomalyDetector.scala:98)
	at com.amazon.deequ.anomalydetection.AnomalyDetector.isNewPointAnomalous(AnomalyDetector.scala:60)
	at com.amazon.deequ.checks.Check$.isNewestPointNonAnomalous(Check.scala:1157)
	at com.amazon.deequ.checks.Check.$anonfun$isNewestPointNonAnomalous$1(Check.scala:446)
	at scala.runtime.java8.JFunction1$mcZD$sp.apply(JFunction1$mcZD$sp.java:23)
	at com.amazon.deequ.constraints.AnalysisBasedConstraint.runAssertion(AnalysisBasedConstraint.scala:108)
	at com.amazon.deequ.constraints.AnalysisBasedConstraint.pickValueAndAssert(AnalysisBasedConstraint.scala:74)
	at com.amazon.deequ.constraints.AnalysisBasedConstraint.$anonfun$evaluate$2(AnalysisBasedConstraint.scala:60)
	at scala.Option.map(Option.scala:230)
	at com.amazon.deequ.constraints.AnalysisBasedConstraint.evaluate(AnalysisBasedConstraint.scala:60)
	at com.amazon.deequ.constraints.ConstraintDecorator.evaluate(Constraint.scala:56)
	at com.amazon.deequ.checks.Check.$anonfun$evaluate$1(Check.scala:1057)
	at scala.collection.immutable.List.map(List.scala:293)
	at com.amazon.deequ.checks.Check.evaluate(Check.scala:1057)
	at com.amazon.deequ.VerificationSuite.$anonfun$evaluate$1(VerificationSuite.scala:269)
	at scala.collection.immutable.List.map(List.scala:293)
	at com.amazon.deequ.VerificationSuite.evaluate(VerificationSuite.scala:269)
	at com.amazon.deequ.VerificationSuite.doVerificationRun(VerificationSuite.scala:132)
	at com.amazon.deequ.VerificationRunBuilder.run(VerificationRunBuilder.scala:173)
	at Main$.$anonfun$main$1(Main.scala:36)
	at scala.runtime.java8.JFunction1$mcVI$sp.apply(JFunction1$mcVI$sp.java:23)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at Main$.main(Main.scala:31)
	at Main.main(Main.scala)
Caused by: java.lang.ClassNotFoundException: breeze.stats.package$
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 25 more

How to reproduce:

Create SBT project with deequ, example:

Build.sbt:

ThisBuild / scalaVersion := "2.12.14"
lazy val root = (project in file("."))
 .settings(
   name := "DeequBreezeDebug",
   libraryDependencies := {
     Seq(
       "com.amazon.deequ" % "deequ" % "2.0.4-spark-3.3"
     )}
 )

Main.scala

import com.amazon.deequ.VerificationSuite
import com.amazon.deequ.anomalydetection.seasonal.HoltWinters
import com.amazon.deequ.repository.memory.InMemoryMetricsRepository
import org.apache.spark.sql.SparkSession
import com.amazon.deequ.analyzers.Size
import com.amazon.deequ.repository.ResultKey
import java.time.{LocalDate, LocalDateTime, ZoneOffset}

case class Item(id: Long, someVal: String, someDate: LocalDate, someFlag: Boolean)

object Main {
 def main(args: Array[String]): Unit = {
   val spark: SparkSession = SparkSession.builder()
     .appName("HoltWinters-test")
     .master("local")
     .getOrCreate()

   import spark.implicits._

   val sampleData = Seq(
     Item(1, "ABC", LocalDate.of(2023, 5, 1), true),
     Item(2, "ABC", LocalDate.of(2023, 5, 10), false),
     Item(3, "CDE", LocalDate.of(2023, 5, 21), true)
   ).toDF()

   val metricsRepository = new InMemoryMetricsRepository()

   val days = Seq(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15)

   for (day <- days) {
     VerificationSuite()
       .onData(sampleData)
       .useRepository(metricsRepository)
       .saveOrAppendResult(ResultKey(LocalDateTime.of(2023, 8, day, 0, 0).toInstant(ZoneOffset.UTC).toEpochMilli))
       .addAnomalyCheck(new HoltWinters(metricsInterval = HoltWinters.MetricInterval.Daily, seasonality = HoltWinters.SeriesSeasonality.Weekly), Size())
       .run()
   }
 }
}

Description of changes:

Update breeze dependency to version 1.2

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.