mango OOM error

Using Mango as of 9f518e0 I created test input data based on the ChrM mouse files using ADAM.

I launch Mango with:

./bin/mango-submit data/test_files/mm10_chrM.fa -read_files data/test_files/mouse_chrM_adam.adam -var_files data/test_files/small_adam_vcf.adam -feat_file data/test_files/smalltest.bed

The web UI comes up, but whne I try to search for "chrM:0-5000" and click submit on the web UI, I see the following OOM error on the terminal

./bin/mango-submit data/test_files/mm10_chrM.fa -read_files data/test_files/mouse_chrM_adam.adam -var_files data/test_files/small_adam_vcf.adam -feat_file data/test_files/smalltest.bed 
Using SPARK_SUBMIT=/home/jp/Apps/Spark/spark-1.5.2-bin-hadoop2.6/bin/spark-submit
2016-05-19 10:26:54 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2016-05-19 10:26:55 WARN  MetricsSystem:71 - Using default name DAGScheduler for source because spark.app.id is not set.
View the visualization at: 8080
Variant visualization at: /variants
Overall visualization at: /overall
Quit at: /quit
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2016-05-19 10:27:28 ERROR Executor:96 - Exception in task 2.0 in stage 9.0 (TID 170)
java.lang.OutOfMemoryError: Java heap space
    at java.util.IdentityHashMap.resize(IdentityHashMap.java:471)
    at java.util.IdentityHashMap.put(IdentityHashMap.java:440)
    at org.apache.spark.util.SizeEstimator$SearchState.enqueue(SizeEstimator.scala:159)
    at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:203)
    at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:202)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:202)
    at org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:186)
    at org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:54)
    at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78)
    at org.apache.spark.util.collection.SizeTracker$class.resetSamples(SizeTracker.scala:61)
    at org.apache.spark.util.collection.PartitionedSerializedPairBuffer.resetSamples(PartitionedSerializedPairBuffer.scala:56)
    at org.apache.spark.util.collection.SizeTracker$class.$init$(SizeTracker.scala:51)
    at org.apache.spark.util.collection.PartitionedSerializedPairBuffer.<init>(PartitionedSerializedPairBuffer.scala:56)
    at org.apache.spark.util.collection.ExternalSorter.newBuffer(ExternalSorter.scala:140)
    at org.apache.spark.util.collection.ExternalSorter.<init>(ExternalSorter.scala:149)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:55)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-05-19 10:27:28 ERROR Executor:96 - Exception in task 21.0 in stage 9.0 (TID 189)
java.lang.OutOfMemoryError: Java heap space
    at java.util.IdentityHashMap.resize(IdentityHashMap.java:471)
    at java.util.IdentityHashMap.put(IdentityHashMap.java:440)
    at org.apache.spark.util.SizeEstimator$SearchState.enqueue(SizeEstimator.scala:159)
    at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:203)
    at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:202)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:202)
    at org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:186)
    at org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:54)
    at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78)
    at org.apache.spark.util.collection.SizeTracker$class.resetSamples(SizeTracker.scala:61)
    at org.apache.spark.util.collection.PartitionedSerializedPairBuffer.resetSamples(PartitionedSerializedPairBuffer.scala:56)
    at org.apache.spark.util.collection.SizeTracker$class.$init$(SizeTracker.scala:51)
    at org.apache.spark.util.collection.PartitionedSerializedPairBuffer.<init>(PartitionedSerializedPairBuffer.scala:56)
    at org.apache.spark.util.collection.ExternalSorter.newBuffer(ExternalSorter.scala:140)
    at org.apache.spark.util.collection.ExternalSorter.<init>(ExternalSorter.scala:149)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:55)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-05-19 10:27:28 ERROR Executor:96 - Exception in task 1.0 in stage 9.0 (TID 169)
java.lang.OutOfMemoryError: Java heap space
    at java.util.IdentityHashMap.resize(IdentityHashMap.java:471)
    at java.util.IdentityHashMap.put(IdentityHashMap.java:440)
    at org.apache.spark.util.SizeEstimator$SearchState.enqueue(SizeEstimator.scala:159)
    at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:203)
    at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:202)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:202)
    at org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:186)
    at org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:54)
    at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78)
    at org.apache.spark.util.collection.SizeTracker$class.resetSamples(SizeTracker.scala:61)
    at org.apache.spark.util.collection.PartitionedSerializedPairBuffer.resetSamples(PartitionedSerializedPairBuffer.scala:56)
    at org.apache.spark.util.collection.SizeTracker$class.$init$(SizeTracker.scala:51)
    at org.apache.spark.util.collection.PartitionedSerializedPairBuffer.<init>(PartitionedSerializedPairBuffer.scala:56)
    at org.apache.spark.util.collection.ExternalSorter.newBuffer(ExternalSorter.scala:140)
    at org.apache.spark.util.collection.ExternalSorter.<init>(ExternalSorter.scala:149)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:55)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-05-19 10:27:30 ERROR Executor:96 - Exception in task 8.0 in stage 9.0 (TID 176)
java.lang.OutOfMemoryError: Java heap space
    at java.util.IdentityHashMap.resize(IdentityHashMap.java:471)
    at java.util.IdentityHashMap.put(IdentityHashMap.java:440)
    at org.apache.spark.util.SizeEstimator$SearchState.enqueue(SizeEstimator.scala:159)
    at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:203)
    at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:202)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:202)
    at org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:186)
    at org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:54)
    at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78)
    at org.apache.spark.util.collection.SizeTracker$class.resetSamples(SizeTracker.scala:61)
    at org.apache.spark.util.collection.PartitionedSerializedPairBuffer.resetSamples(PartitionedSerializedPairBuffer.scala:56)
    at org.apache.spark.util.collection.SizeTracker$class.$init$(SizeTracker.scala:51)
    at org.apache.spark.util.collection.PartitionedSerializedPairBuffer.<init>(PartitionedSerializedPairBuffer.scala:56)
    at org.apache.spark.util.collection.ExternalSorter.newBuffer(ExternalSorter.scala:140)
    at org.apache.spark.util.collection.ExternalSorter.<init>(ExternalSorter.scala:149)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:55)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-05-19 10:27:33 ERROR Executor:96 - Exception in task 5.0 in stage 9.0 (TID 173)
java.lang.OutOfMemoryError: Java heap space
2016-05-19 10:27:31 ERROR Executor:96 - Exception in task 13.0 in stage 9.0 (TID 181)
java.lang.OutOfMemoryError: Java heap space
2016-05-19 10:27:33 ERROR SparkUncaughtExceptionHandler:96 - Uncaught exception in thread Thread[Executor task launch worker-3,5,main]
java.lang.OutOfMemoryError: Java heap space
2016-05-19 10:27:33 ERROR SparkUncaughtExceptionHandler:96 - Uncaught exception in thread Thread[Executor task launch worker-2,5,main]
java.lang.OutOfMemoryError: Java heap space
2016-05-19 10:27:35 ERROR SparkUncaughtExceptionHandler:96 - Uncaught exception in thread Thread[Executor task launch worker-1,5,main]
java.lang.OutOfMemoryError: Java heap space
    at java.util.IdentityHashMap.resize(IdentityHashMap.java:471)
    at java.util.IdentityHashMap.put(IdentityHashMap.java:440)
    at org.apache.spark.util.SizeEstimator$SearchState.enqueue(SizeEstimator.scala:159)
    at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:203)
    at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:202)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:202)
    at org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:186)
    at org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:54)
    at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78)
    at org.apache.spark.util.collection.SizeTracker$class.resetSamples(SizeTracker.scala:61)
    at org.apache.spark.util.collection.PartitionedSerializedPairBuffer.resetSamples(PartitionedSerializedPairBuffer.scala:56)
    at org.apache.spark.util.collection.SizeTracker$class.$init$(SizeTracker.scala:51)
    at org.apache.spark.util.collection.PartitionedSerializedPairBuffer.<init>(PartitionedSerializedPairBuffer.scala:56)
    at org.apache.spark.util.collection.ExternalSorter.newBuffer(ExternalSorter.scala:140)
    at org.apache.spark.util.collection.ExternalSorter.<init>(ExternalSorter.scala:149)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:55)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
2016-05-19 10:27:37 ERROR Executor:96 - Exception in task 17.0 in stage 9.0 (TID 185)
java.lang.OutOfMemoryError: Java heap space
2016-05-19 10:27:35 ERROR Executor:96 - Exception in task 9.0 in stage 9.0 (TID 177)
java.lang.OutOfMemoryError: Java heap space
2016-05-19 10:27:35 ERROR SparkUncaughtExceptionHandler:96 - Uncaught exception in thread Thread[Executor task launch worker-26,5,main]
java.lang.OutOfMemoryError: Java heap space
    at java.util.IdentityHashMap.resize(IdentityHashMap.java:471)
    at java.util.IdentityHashMap.put(IdentityHashMap.java:440)
    at org.apache.spark.util.SizeEstimator$SearchState.enqueue(SizeEstimator.scala:159)
    at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:203)
    at org.apache.spark.util.SizeEstimator$$anonfun$visitSingleObject$1.apply(SizeEstimator.scala:202)
    at scala.collection.immutable.List.foreach(List.scala:318)
    at org.apache.spark.util.SizeEstimator$.visitSingleObject(SizeEstimator.scala:202)
    at org.apache.spark.util.SizeEstimator$.org$apache$spark$util$SizeEstimator$$estimate(SizeEstimator.scala:186)
    at org.apache.spark.util.SizeEstimator$.estimate(SizeEstimator.scala:54)
    at org.apache.spark.util.collection.SizeTracker$class.takeSample(SizeTracker.scala:78)
    at org.apache.spark.util.collection.SizeTracker$class.resetSamples(SizeTracker.scala:61)
    at org.apache.spark.util.collection.PartitionedSerializedPairBuffer.resetSamples(PartitionedSerializedPairBuffer.scala:56)
    at org.apache.spark.util.collection.SizeTracker$class.$init$(SizeTracker.scala:51)
    at org.apache.spark.util.collection.PartitionedSerializedPairBuffer.<init>(PartitionedSerializedPairBuffer.scala:56)
    at org.apache.spark.util.collection.ExternalSorter.newBuffer(ExternalSorter.scala:140)
    at org.apache.spark.util.collection.ExternalSorter.<init>(ExternalSorter.scala:149)
    at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:55)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
    at org.apache.spark.scheduler.Task.run(Task.scala:88)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Also, my default java heap size is set to 32 GB

May 19 '16 14:05 jpdna

I believe this is an issue when trying to calculate and render all point mutations. This should be fixed once convolution is in. If not, we will have to look at other options.

Best,

Alyssa Morrow [email protected] mailto:[email protected] 414-254-6645

On May 19, 2016, at 7:31 AM, Justin Paschall [email protected] wrote:

./bin/mango-submit data/test_files/mm10_chrM.fa -read_files data/test_files/mouse_chrM_adam.adam -var_files data/test_files/small_adam_vcf.adam -feat_file data/test_files/smalltest.bed

May 19 '16 16:05 akmorrow13

Is there another Mango branch on of your your personal repos you suggest I look at over the next day or so?

May 19 '16 18:05 jpdna

mango mango copied to clipboard

OOM error

mango
mango copied to clipboard