VariantSpark
VariantSpark copied to clipboard
Error running mvn clean install
Hello, I am trying to install VariantSpark on a Centos 7 box, jdk 1.8, scala 2.3.1, spark 2.1.1. When I do a mvn clean install, the following test is failing. Could you please help me resolve this?
-Thanks! Vishal.
[INFO] Running au.csiro.variantspark.hail.HailIntegrationTest 2020-02-05 13:23:48 Hail: INFO: Hail temporary directory: file:/tmp/hail.kR2vbvV713ou 2020-02-05 13:23:48 Hail: INFO: SparkUI: http://tabbyhp:4040 2020-02-05 13:23:48 Hail: INFO: Running Hail version 0.2.16-6da0d3571629 2020-02-05 13:23:48 Hail: INFO: Reading table with no type imputation Loading column 'sample' as type 'str' (type not specified) Loading column 'x22_16050408' as type 'str' (type not specified) Loading column 'x22_16050612' as type 'str' (type not specified) Loading column 'x22_16050678' as type 'str' (type not specified) Loading column 'x22_16050984' as type 'str' (type not specified) Loading column 'x22_16051107' as type 'str' (type not specified) Loading column 'x22_16051249' as type 'str' (type not specified) Loading column 'x22_16051347' as type 'str' (type not specified) Loading column 'x22_16051453' as type 'str' (type not specified) Loading column 'x22_16051477' as type 'str' (type not specified) Loading column 'x22_16051480' as type 'float64' (user-specified)
FATAL: caught exception java.lang.UnsatisfiedLinkError: /tmp/libhail9047678369918805725.so: /usr/local/lib64/libstdc++.so.6: version CXXABI_1.3.9' not found (required by /tmp/libhail9047678369918805725.so) java.lang.UnsatisfiedLinkError: /tmp/libhail9047678369918805725.so: /usr/local/lib64/libstdc++.so.6: version
CXXABI_1.3.9' not found (required by /tmp/libhail9047678369918805725.so)
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1934)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1817)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at is.hail.nativecode.NativeCode.
[INFO] Running au.csiro.variantspark.hail.adapter.HailAdapterIntegrationTest [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in au.csiro.variantspark.hail.adapter.HailAdapterIntegrationTest [INFO] Running au.csiro.variantspark.input.CsvStdFeatureSourceTest [ERROR] Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.004 s <<< FAILURE! - in au.csiro.variantspark.input.CsvStdFeatureSourceTest [ERROR] testConvertsLargeFileCorectly(au.csiro.variantspark.input.CsvStdFeatureSourceTest) Time elapsed: 0.003 s <<< ERROR! java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stopped SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
The currently active SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at au.csiro.variantspark.input.CsvStdFeatureSourceTest.testConvertsLargeFileCorectly(CsvStdFeatureSourceTest.scala:39)
[ERROR] testConvetsCSVCorectly(au.csiro.variantspark.input.CsvStdFeatureSourceTest) Time elapsed: 0.001 s <<< ERROR! java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stopped SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
The currently active SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at au.csiro.variantspark.input.CsvStdFeatureSourceTest.testConvetsCSVCorectly(CsvStdFeatureSourceTest.scala:19)
[INFO] Running au.csiro.variantspark.input.DefVariantToFeatureConverterTest [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in au.csiro.variantspark.input.DefVariantToFeatureConverterTest [INFO] Running au.csiro.variantspark.input.generate.NoisyEffectLabelGeneratorTest [ERROR] Tests run: 5, Failures: 0, Errors: 5, Skipped: 0, Time elapsed: 0.029 s <<< FAILURE! - in au.csiro.variantspark.input.generate.NoisyEffectLabelGeneratorTest [ERROR] testAdditiveEffectCorrectness(au.csiro.variantspark.input.generate.NoisyEffectLabelGeneratorTest) Time elapsed: 0.006 s <<< ERROR! java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stopped SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
The currently active SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at au.csiro.variantspark.input.generate.NoisyEffectLabelGeneratorTest.testAdditiveEffectCorrectness(NoisyEffectLabelGeneratorTest.scala:62)
[ERROR] testResponseGeneration_var_0_5_prec_0_50(au.csiro.variantspark.input.generate.NoisyEffectLabelGeneratorTest) Time elapsed: 0.004 s <<< ERROR! java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stopped SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
The currently active SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at au.csiro.variantspark.input.generate.NoisyEffectLabelGeneratorTest.testResponseGeneration_var_0_5_prec_0_50(NoisyEffectLabelGeneratorTest.scala:86)
[ERROR] testMultiplicativeEffectCorrectness(au.csiro.variantspark.input.generate.NoisyEffectLabelGeneratorTest) Time elapsed: 0.008 s <<< ERROR! java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stopped SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
The currently active SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at au.csiro.variantspark.input.generate.NoisyEffectLabelGeneratorTest.testMultiplicativeEffectCorrectness(NoisyEffectLabelGeneratorTest.scala:77)
[ERROR] testResponseGeneration_var_0_2_prec_0_75(au.csiro.variantspark.input.generate.NoisyEffectLabelGeneratorTest) Time elapsed: 0.009 s <<< ERROR! java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stopped SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
The currently active SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at au.csiro.variantspark.input.generate.NoisyEffectLabelGeneratorTest.testResponseGeneration_var_0_2_prec_0_75(NoisyEffectLabelGeneratorTest.scala:28)
[ERROR] testMultiplicativeResponseGeneration_var_0_2_prec_0_75(au.csiro.variantspark.input.generate.NoisyEffectLabelGeneratorTest) Time elapsed: 0.002 s <<< ERROR! java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext. This stopped SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
The currently active SparkContext was created at:
org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) au.csiro.variantspark.test.TestSparkContext$.spark$lzycompute(TestSparkContext.scala:12) au.csiro.variantspark.test.TestSparkContext$.spark(TestSparkContext.scala:7) au.csiro.variantspark.test.SparkTest$class.spark(SparkTest.scala:4) au.csiro.variantspark.algo.PairWiseDistanceTest.spark$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.spark(PairWiseDistanceTest.scala:10) au.csiro.variantspark.test.SparkTest$class.sc(SparkTest.scala:5) au.csiro.variantspark.algo.PairWiseDistanceTest.sc$lzycompute(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.sc(PairWiseDistanceTest.scala:10) au.csiro.variantspark.algo.PairWiseDistanceTest.testCorrectlyCalculatesPairWiseDistance2D(PairWiseDistanceTest.scala:33) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:498) org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at au.csiro.variantspark.input.generate.NoisyEffectLabelGeneratorTest.testMultiplicativeResponseGeneration_var_0_2_prec_0_75(NoisyEffectLabelGeneratorTest.scala:43)
[INFO] Running au.csiro.variantspark.metrics.GiniTest [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in au.csiro.variantspark.metrics.GiniTest [INFO] Running au.csiro.variantspark.metrics.MetricsTest [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 s - in au.csiro.variantspark.metrics.MetricsTest [INFO] Running au.csiro.variantspark.misc.ReproducibilityTest [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.001 s - in au.csiro.variantspark.misc.ReproducibilityTest [INFO] Running au.csiro.variantspark.output.ParquetFeatureSinkTest [WARNING] Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0 s - in au.csiro.variantspark.output.ParquetFeatureSinkTest [INFO] Running au.csiro.variantspark.utils.SampleTest [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in au.csiro.variantspark.utils.SampleTest [INFO] [INFO] Results: [INFO] [ERROR] Errors: [ERROR] HailIntegrationTest.testRunImportanceAnalysis:97 » Spark Job 875 cancelled bec... [ERROR] CsvStdFeatureSourceTest.testConvertsLargeFileCorectly:39 » IllegalState Cannot... [ERROR] CsvStdFeatureSourceTest.testConvetsCSVCorectly:19 » IllegalState Cannot call m... [ERROR] NoisyEffectLabelGeneratorTest.testAdditiveEffectCorrectness:62 » IllegalState ... [ERROR] NoisyEffectLabelGeneratorTest.testMultiplicativeEffectCorrectness:77 » IllegalState [ERROR] NoisyEffectLabelGeneratorTest.testMultiplicativeResponseGeneration_var_0_2_prec_0_75:43 » IllegalState [ERROR] NoisyEffectLabelGeneratorTest.testResponseGeneration_var_0_2_prec_0_75:28 » IllegalState [ERROR] NoisyEffectLabelGeneratorTest.testResponseGeneration_var_0_5_prec_0_50:86 » IllegalState [INFO] [ERROR] Tests run: 88, Failures: 0, Errors: 8, Skipped: 2 [INFO] [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 1:11.330s [INFO] Finished at: Wed Feb 05 13:23:50 EST 2020 [INFO] Final Memory: 73M/1684M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.20:test (default-test) on project variant-spark_2.11: There are test failures. [ERROR] [ERROR] Please refer to /home/vncuser/VariantSpark/target/surefire-reports for the individual test results. [ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump, [date].dumpstream and [date]-jvmRun[N].dumpstream. [ERROR] Error occurred in starting fork, check output in log [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException: Error occurred in starting fork, check output in log [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:634) [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:533) [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:279) [ERROR] at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:243) [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1077) [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:907) [ERROR] at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:785) [ERROR] at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) [ERROR] at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84) [ERROR] at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59) [ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183) [ERROR] at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161) [ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320) [ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156) [ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537) [ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196) [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:141) [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [ERROR] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [ERROR] at java.lang.reflect.Method.invoke(Method.java:498) [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:414) [ERROR] at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:357) [ERROR] -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
What version of Java is currently your default?
I.e. what is the output of java -version
in a terminal? It needs to 1.8.x if it isn't already.
HI @vishaln79 ,
the problem is caused by the version of libstd++ packaged with Centos 7.
Apparently ships with libstdc++-4.8.5 which supports CXXABI_1.3.7 while VariantSpark (an in particula Hail library) needs at least CXXABI_1.3.9 which come with libstd++-5.1
(Here a bit more info on the issue: https://forums.centos.org/viewtopic.php?f=47&t=68062&sid=a75125fe4cd82c485422b73b0527aa24#p285777)
They way to check which version is supported on your system is to run
strings /usr/local/lib64/libstdc++.so.6 | grep CXXABI
To fix this issue you either need to install a never (>-5.1) of version of stdlib++ (or gcc) or switch another distribution - e.g. Centos 8 which comes with the newer version by default.
Assuming the answers helped to fix the problem