spades
spades copied to clipboard
ablab
Out of Memory - spades-hammer
Description of bug
Hi,
I know what it means, mmap can not allocate sufficient memory for the next step. But how much memory does it need? and why?
22:18:38.986 82G / 128G ERROR General (mmapped_reader.hpp : 52) mmap(2) failed. Reason: Cannot allocate memory. Error code: 12
== Error == system call for: "['/spades/bin/spades-hammer', '/tmp/corrected/configs/config.info']" finished abnormally, OS return value: 12
With the attached memory profile we can see that SPAdes consumes about 140 GB of memory before it fails. So it looks like spades-hammer requests an enormous amount of memory at once (750 - 140 = 610 GB).
As you can see in the log file, SPAdes on this machine has access to 48 cores and 750 GB of memory give or take.
Can I change the configuration to make this work? e.g. tell SPAdes to use less cores?
Thank you for your help.
PD: I don't understand how to proceed without marking "There are no errors in my spades.log". Of course, there is, that's the point.
spades.log
params.txt
SPAdes version
SPAdes 3.15.4
Operating System
Linux-4.14.262-135.489.amzn1.x86_64-x86_64-with-Ubuntu-20.04-focal
Python Version
Python 2.7.18
Method of SPAdes installation
Binaries
No errors reported in spades.log
- [X] Yes
Well, your input contains almost 30 billion distinct k-mers. I think this is one of the largest datasets I saw to-date. BayesHammer would require up to 40-50 bytes per k-mer, so, this is well beyond the amount of RAM you're having.
Also, keep in mind that very often these high-memory systems are not configured properly – you cannot allocate really this amount of RAM for the single application. See https://github.com/ablab/spades/issues/871 as an example how the solution could look like.
