Memory error

[ZohrehFattahi]

Hi,

I am trying to run the exome pipeline. When I run it for 2 or 3 samples it runs correctly. But, when I run a command for about 20 samples I get the following error.

This is the command: /data/NGS/Reanalysis-Package/STRetch-master/tools/bin/bpipe run -p ./Twist_Exome_Core_Covered_Targets_hg19_liftover.bed *.fastq.gz

Here is the error: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:109) at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131) Caused by: java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3332) at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124) at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596) at java.lang.StringBuilder.append(StringBuilder.java:190) at org.codehaus.groovy.runtime.IOGroovyMethods.getText(IOGroovyMethods.java:886) at org.codehaus.groovy.runtime.ResourceGroovyMethods.getText(ResourceGroovyMethods.java:600) at org.codehaus.groovy.runtime.dgm$962.doMethodInvoke(Unknown Source) at org.codehaus.groovy.reflection.GeneratedMetaMethod$Proxy.doMethodInvoke(GeneratedMetaMethod.java:73) at org.codehaus.groovy.runtime.metaclass.MethodMetaProperty$GetBeanMethodMetaProperty.getProperty(MethodMetaProperty.java:76) at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:64) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:296) at bpipe.Runner.loadPipelineSrc(Runner.groovy:662) at bpipe.Runner$loadPipelineSrc.callStatic(Unknown Source) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:56) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:194) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callStatic(AbstractCallSite.java:222) at bpipe.Runner.main(Runner.groovy:377) ... 6 more

It seems that the error is related to memory. I am running on a server with 264g memory. Can you please guide me on how I can solve it?

Zohreh

[hdashnow]

Hi Zohreh,

bpipe (the workflow software running STRetch) should not use substantial memory on 20 samples. I've successfully run it on thousands of samples without issue.

Here are a few things you could try:

• Specify the amount of memory for Java, like this: MAX_JAVA_MEM=24g /data/NGS/Reanalysis-Package/STRetch-master/tools/bin/bpipe run...
• Check that you are running the command in a high-memory environment. For example, many clusters are set up to have a limited memory "head node" where you submit jobs, that are then sent to a higher memory environment. On Linux, check memory with grep MemTotal /proc/meminfo
• Check the latest version of bpipe is being used, and if not, update it. Check the version by running just bpipe without any thing else. In your system: /data/NGS/Reanalysis-Package/STRetch-master/tools/bin/bpipe. You can check/download the latest version here: https://github.com/ssadedin/bpipe/releases/. Simply replace the copy in /data/NGS/Reanalysis-Package/STRetch-master/tools/bin/bpipe.

You can read more about memory usage in bpipe here: https://github.com/ssadedin/bpipe/issues/97

Let me know if these suggestions worked or not. It could be useful for other users.

Warm regards, Harriet