iskra icon indicating copy to clipboard operation
iskra copied to clipboard
verified publisher icon VirtusLab

1 test error after migrating to Spark 4.0.0-preview2

Open tribbloid opened this issue 1 year ago • 4 comments

on test "join-right-on":

org.scalatest.exceptions.TestFailedException: List((None, None, 3, "c"), (Some(2), Some(20), 2, "b")) did not equal List((Some(2), Some(20), 2, "b"), (None, None, 3, "c"))
	at app//org.scalatest.matchers.MatchersHelper$.indicateFailure(MatchersHelper.scala:397)
	at app//org.scalatest.matchers.should.Matchers.shouldEqual(Matchers.scala:6893)
	at app//org.scalatest.matchers.should.Matchers.shouldEqual$(Matchers.scala:1808)
	at app//org.scalatest.matchers.should.Matchers$.shouldEqual(Matchers.scala:7725)
	at app//org.virtuslab.iskra.test.SparkUnitTest.shouldEqual(SparkUnitTest.scala:20)
	at app//org.virtuslab.iskra.test.JoinTest.testFun$proxy3$1(JoinTest.scala:87)
	at app//org.virtuslab.iskra.test.JoinTest.$init$$$anonfun$3(JoinTest.scala:67)
	at app//org.scalatest.Transformer.apply$$anonfun$1(Transformer.scala:22)
	at app//org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
	at app//org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:31)
	at app//org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
	at app//org.scalatest.Transformer.apply(Transformer.scala:22)
	at app//org.scalatest.Transformer.apply(Transformer.scala:21)
	at app//org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:206)
	at app//org.scalatest.TestSuite.withFixture(TestSuite.scala:196)
	at app//org.scalatest.TestSuite.withFixture$(TestSuite.scala:138)
	at app//org.scalatest.funsuite.AnyFunSuite.withFixture(AnyFunSuite.scala:1563)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:212)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.runTest$$anonfun$1(AnyFunSuiteLike.scala:216)
	at app//org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:216)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:47)
	at app//org.scalatest.funsuite.AnyFunSuite.runTest(AnyFunSuite.scala:1563)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.runTests$$anonfun$1(AnyFunSuiteLike.scala:249)
	at app//org.scalatest.SuperEngine.traverseSubNodes$1$$anonfun$1(Engine.scala:413)
	at app//scala.runtime.function.JProcedure1.apply(JProcedure1.java:15)
	at app//scala.runtime.function.JProcedure1.apply(JProcedure1.java:10)
	at app//scala.collection.immutable.List.foreach(List.scala:334)
	at app//org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:429)
	at app//org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:396)
	at app//org.scalatest.SuperEngine.runTestsImpl(Engine.scala:475)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.runTests(AnyFunSuiteLike.scala:249)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.runTests$(AnyFunSuiteLike.scala:47)
	at app//org.scalatest.funsuite.AnyFunSuite.runTests(AnyFunSuite.scala:1563)
	at app//org.scalatest.Suite.run(Suite.scala:1114)
	at app//org.scalatest.Suite.run$(Suite.scala:564)
	at app//org.scalatest.funsuite.AnyFunSuite.org$scalatest$funsuite$AnyFunSuiteLike$$super$run(AnyFunSuite.scala:1563)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.run$$anonfun$1(AnyFunSuiteLike.scala:253)
	at app//org.scalatest.SuperEngine.runImpl(Engine.scala:535)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.run(AnyFunSuiteLike.scala:253)
	at app//org.scalatest.funsuite.AnyFunSuiteLike.run$(AnyFunSuiteLike.scala:47)
	at app//org.virtuslab.iskra.test.SparkUnitTest.org$scalatest$BeforeAndAfterAll$$super$run(SparkUnitTest.scala:7)
	at app//org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
	at app//org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:217)
	at app//org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:135)
	at app//org.virtuslab.iskra.test.SparkUnitTest.run(SparkUnitTest.scala:7)
	at app//co.helmethair.scalatest.runtime.Executor.runScalatests(Executor.java:130)
	at app//co.helmethair.scalatest.runtime.Executor.executeSuite(Executor.java:86)
	at app//co.helmethair.scalatest.runtime.Executor.executeTest(Executor.java:53)
	at app//co.helmethair.scalatest.runtime.Executor.lambda$executeSuite$2(Executor.java:82)
	at [email protected]/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at [email protected]/java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:357)
	at [email protected]/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:510)
	at [email protected]/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at [email protected]/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at [email protected]/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at [email protected]/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at [email protected]/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
	at app//co.helmethair.scalatest.runtime.Executor.executeSuite(Executor.java:82)
	at app//co.helmethair.scalatest.runtime.Executor.executeTest(Executor.java:60)
	at app//co.helmethair.scalatest.ScalatestEngine.execute(ScalatestEngine.java:55)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
	at app//org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
	at app//org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at app//org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at app//org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:124)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:99)
	at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:94)
	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:63)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at [email protected]/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at [email protected]/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at [email protected]/java.lang.reflect.Method.invoke(Method.java:569)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:92)
	at jdk.proxy1/jdk.proxy1.$Proxy4.stop(Unknown Source)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:200)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:132)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:103)
	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:63)
	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:121)
	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

haven't figure out the cause yet

tribbloid avatar Dec 19 '24 20:12 tribbloid

Thanks for reporting. Actually an attempt to bump spark to 4.0.0-preview2 caused a few more issues for me:

  • I had to add //> using jvm 17 to project.scala because that's the minimal java version required by spark 4 (it was 11 by default on my machine)
  • join-right-on failed because the rows of the data frame were reordered; that's quite surprising but I guess spark doesn't give any guarantees on ordering unless one explicitly uses .orderBy on a data frame (which is not yet supported in iskra, waiting for a major refactor that I'm slowly working on right now). The fix might be to change tests to compare sets of values rather than lists
  • The test in Workers.scala failed to because of the same reason, combined with some differences in how data frames are printed. I would need to rethink how self-standing examples could be reused as end-to-end tests (that might require parsing the console output or changing the approach to such tests entirely)

prolativ avatar Dec 20 '24 08:12 prolativ

I see, I used a copy-pasted gradle build (unfortunately as a submodule), should I publish it as an independent fork?

BTW, the following test JVM argument will be required for Spark 4 itself and all dependents:

"-XX:+IgnoreUnrecognizedVMOptions",
        "--add-opens=java.base/java.lang=ALL-UNNAMED",
        "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED",
        "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED",
        "--add-opens=java.base/java.io=ALL-UNNAMED",
        "--add-opens=java.base/java.net=ALL-UNNAMED",
        "--add-opens=java.base/java.nio=ALL-UNNAMED",
        "--add-opens=java.base/java.util=ALL-UNNAMED",
        "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED",
        "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED",
        "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED",
        "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
        "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED",
        "--add-opens=java.base/sun.security.action=ALL-UNNAMED",
        "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED",
        "-Djdk.reflect.useDirectMethodHandle=false",
        "-Dio.netty.tryReflectionSetAccessible=true"

tribbloid avatar Dec 20 '24 20:12 tribbloid

sorry @prolativ do you encounter some difficulty in reproducing this testing environment in scala-cli? would you like me to publish this project under sbt or gradle?

tribbloid avatar Jan 10 '25 20:01 tribbloid

I did reproduce the issue directly in this repo, which is built using scala-cli. I guess you meant spark 4, not scala 4 😆 Currently I'm struggling to find time to implement the redesign of iskra'a API that's been boiling in my head for some time already, but that's not that easy 😅 So making it run on spark 4, which doesn't have a stable release yet, is in no measure my focus at the moment, even though this doesn't seem to require that much work. The build too used for publishing shouldn't matter at all from the library's users

prolativ avatar Jan 10 '25 21:01 prolativ