bazam icon indicating copy to clipboard operation
bazam copied to clipboard

Published 20 hours ago • verified publisher icon ssadedin

OutOfMemoryError

Open brentp opened this issue 6 years ago • 2 comments

hi, I was able to use bazam on a BAM file to realign and it was chugging along smoothly but it eventually errored with the message below. I ran as:

java -Xmx4G -jar bazam.jar -L regions.bed -bam $bam -n 3 \
   | bwa mem ...

where regions.bed is:

1	0	249250621
20	0	63025520
21	0	48129895
22	0	51304566

the final status message was:

gngs.pair.PairScanner   [1]     INFO    |2:41:56 Wed Oct 10 14:41:56 MDT 2018 :  Processed 650989000 @ 18663.56/s 21:10917077, loc: 651m,71.1m,240.5k chimeric: 803.2k formatted: 71.1m, written: 68.3m

traceback:

An exception occurred in the Actor thread Actor Thread 132
An exception occurred in the Actor thread Actor Thread 128

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Actor Thread 132"

java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.HashSet.<init>(HashSet.java:144)
        at org.codehaus.groovy.reflection.CachedClass$8.initValue(CachedClass.java:216)
        at org.codehaus.groovy.reflection.CachedClass$8.initValue(CachedClass.java:214)
        at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:49)
        at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:36)
        at org.codehaus.groovy.reflection.CachedClass.getInterfaces(CachedClass.java:266)
        at org.codehaus.groovy.reflection.CachedClass.<init>(CachedClass.java:252)
        at org.codehaus.groovy.reflection.ClassInfo.createCachedClass(ClassInfo.java:369)
        at org.codehaus.groovy.reflection.ClassInfo.access$300(ClassInfo.java:46)
        at org.codehaus.groovy.reflection.ClassInfo$LazyCachedClassRef.initValue(ClassInfo.java:423)
        at org.codehaus.groovy.reflection.ClassInfo$LazyCachedClassRef.initValue(ClassInfo.java:414)
        at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:49)
        at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:40)
        at org.codehaus.groovy.reflection.ClassInfo.getCachedClass(ClassInfo.java:135)
        at org.codehaus.groovy.reflection.ReflectionCache.getCachedClass(ReflectionCache.java:110)
        at org.codehaus.groovy.reflection.CachedClass$8.initValue(CachedClass.java:223)
        at org.codehaus.groovy.reflection.CachedClass$8.initValue(CachedClass.java:214)
        at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:49)
        at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:36)
        at org.codehaus.groovy.reflection.CachedClass.getInterfaces(CachedClass.java:266)
        at org.codehaus.groovy.reflection.CachedClass.<init>(CachedClass.java:252)
        at org.codehaus.groovy.reflection.stdclasses.CachedClosureClass.<init>(CachedClosureClass.java:30)
        at org.codehaus.groovy.reflection.ClassInfo.createCachedClass(ClassInfo.java:365)
        at org.codehaus.groovy.reflection.ClassInfo.access$300(ClassInfo.java:46)
        at org.codehaus.groovy.reflection.ClassInfo$LazyCachedClassRef.initValue(ClassInfo.java:423)
        at org.codehaus.groovy.reflection.ClassInfo$LazyCachedClassRef.initValue(ClassInfo.java:414)
        at org.codehaus.groovy.util.LazyReference.getLocked(LazyReference.java:49)
        at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:40)
   at org.codehaus.groovy.util.LazyReference.get(LazyReference.java:40)
        at org.codehaus.groovy.reflection.ClassInfo.getCachedClass(ClassInfo.java:135)
        at org.codehaus.groovy.reflection.ReflectionCache.getCachedClass(ReflectionCache.java:110)
        at groovy.lang.MetaClassImpl.<init>(MetaClassImpl.java:182)
        at groovy.lang.MetaClassImpl.<init>(MetaClassImpl.java:218)
An exception occurred in the Actor thread Actor Thread 129
java.lang.OutOfMemoryError: GC overhead limit exceeded
An exception occurred in the Actor thread Actor Thread 126
java.lang.OutOfMemoryError: GC overhead limit exceeded

java.lang.NoClassDefFoundError: Could not initialize class org.codehaus.groovy.runtime.NullObject
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:891)
        at groovyx.gpars.actor.Actor.sweepQueue(Actor.java:393)
        at groovyx.gpars.actor.Actor.handleTermination(Actor.java:316)
        at groovyx.gpars.actor.AbstractLoopingActor.terminate(AbstractLoopingActor.java:204)
        at groovyx.gpars.actor.AbstractLoopingActor$1.registerError(AbstractLoopingActor.java:65)
        at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:140)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)


java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.NoClassDefFoundError: Could not initialize class org.codehaus.groovy.runtime.NullObject
        at org.codehaus.groovy.runtime.InvokerHelper.invokeMethod(InvokerHelper.java:891)
        at groovyx.gpars.actor.Actor.sweepQueue(Actor.java:393)
        at groovyx.gpars.actor.Actor.handleTermination(Actor.java:316)
        at groovyx.gpars.actor.AbstractLoopingActor.terminate(AbstractLoopingActor.java:204)
        at groovyx.gpars.actor.AbstractLoopingActor$1.registerError(AbstractLoopingActor.java:65)
        at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:140)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
An exception occurred in the Actor thread Actor Thread 125
java.io.IOException: Stream closed
        at sun.nio.cs.StreamEncoder.ensureOpen(StreamEncoder.java:45)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:118)
        at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:135)
        at java.io.OutputStreamWriter.write(OutputStreamWriter.java:220)
        at java.io.Writer.write(Writer.java:157)
        at java.io.Writer.append(Writer.java:227)
        at gngs.pair.PairWriter$_act_closure1$_closure2.doCall(PairWriter.groovy:60)
        at gngs.pair.PairWriter$_act_closure1$_closure2.call(PairWriter.groovy)
        at groovyx.gpars.actor.DefaultActor.onMessage(DefaultActor.java:94)
        at groovyx.gpars.actor.DefaultActorClosure.call(DefaultActorClosure.java:38)
        at groovyx.gpars.actor.AbstractLoopingActor.runEnhancedWithoutRepliesOnMessages(AbstractLoopingActor.java:293)
        at groovyx.gpars.actor.AbstractLoopingActor.access$400(AbstractLoopingActor.java:30)
        at groovyx.gpars.actor.AbstractLoopingActor$1.handleMessage(AbstractLoopingActor.java:93)
        at groovyx.gpars.util.AsyncMessagingCore.run(AsyncMessagingCore.java:132)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

brentp avatar Oct 10 '18 20:10 brentp

I think that 4G is probably not enough RAM (depending on the structure of your BAM file Bazam will cache a lot of reads in memory). So I'd suggest trying with

java -Xmx24g -jar bazam.jar ...

You might not need all of that 24g but it's a reasonably safe value to figure out what's right.

The alternative to use less memory per-instance is to apply sharding, but then you need to run multiple Bazams (either in series or parallel if you can do that).

ssadedin avatar Oct 10 '18 23:10 ssadedin

I think this was a problem with using the regions. I was able to do the full bam with 4G but it errored on regions. Doesn't bazam write reads to disk to avoid high memory use?

brentp avatar Oct 11 '18 15:10 brentp