std::bad_alloc on small-size reference and VCFs

[nicolaprezza]

hap.py crashes after 10 hours of computation, presumably because it tries to allocate too much memory in RAM: peak RAM usage at termination is 117 GB on a system with 128 GB (the manual says that the tool never exceeds 64 GB: https://github.com/Illumina/hap.py#hardware) and the error is a std::bad_alloc. The reference is Human Chr1 (about 250 MB), and the two input VCFs take just 14 and 8 MB. No input format errors are reported, nor any informative error that might hint what the source of the problem is. This is the traceback (including /usr/bin/time's output):

[W] overlapping records at 1:1869284 for sample 0 [W] Variants that overlap on the reference allele: 73 [I] Total VCF records: 192326 [I] Non-reference VCF records: 192326 [W] overlapping records at 1:1165320 for sample 0 [W] Variants that overlap on the reference allele: 283 [I] Total VCF records: 341672 [I] Non-reference VCF records: 341672 Hap.py v0.3.12 2020-01-31 02:27:40,810 WARNING terminate called after throwing an instance of 'std::bad_alloc' 2020-01-31 02:27:40,814 WARNING what(): std::bad_alloc 2020-01-31 02:27:40,814 WARNING Aborted (core dumped) 2020-01-31 02:27:40,814 ERROR Exception when running <function xcmpWrapper at 0x7fcb42329a28>: 2020-01-31 02:27:40,814 ERROR ------------------------------------------------------------ 2020-01-31 02:27:40,814 ERROR Traceback (most recent call last): 2020-01-31 02:27:40,814 ERROR File "/biodata/Nicola/workspace/hap.py/build/lib/python27/Tools/parallel.py", line 71, in parMapper 2020-01-31 02:27:40,823 ERROR return arg[1]['fun'](arg[0], *arg[1]['args'], **arg[1]['kwargs']) 2020-01-31 02:27:40,823 ERROR File "/biodata/Nicola/workspace/hap.py/build/lib/python27/Haplo/xcmp.py", line 69, in xcmpWrapper 2020-01-31 02:27:40,825 ERROR subprocess.check_call(to_run, shell=True, stdout=tfo, stderr=tfe) 2020-01-31 02:27:40,825 ERROR File "/usr/lib/python2.7/subprocess.py", line 190, in check_call 2020-01-31 02:27:40,826 ERROR raise CalledProcessError(retcode, cmd) 2020-01-31 02:27:40,827 ERROR CalledProcessError: Command 'xcmp /tmp/truth.ppnMtGZk.vcf.gz /tmp/query.ppSFz72W.vcf.gz -l 1:115118971-142561441 -o /tmp/result.1:115118971-142561441iBrR7q.bcf -r ../REF/chr1.fasta -f 0 -n 16768 --expand-hapblocks 30 --window 50 --no-hapcmp 0 --qq QUAL' returned non-zero exit status 134 2020-01-31 02:27:40,827 ERROR ------------------------------------------------------------ 2020-01-31 02:27:40,834 ERROR One of the xcmp jobs failed. 2020-01-31 02:27:40,836 ERROR Traceback (most recent call last): 2020-01-31 02:27:40,837 ERROR File "/biodata/Nicola/workspace/hap.py/build/bin/hap.py", line 529, in 2020-01-31 02:27:40,840 ERROR main() 2020-01-31 02:27:40,840 ERROR File "/biodata/Nicola/workspace/hap.py/build/bin/hap.py", line 452, in main 2020-01-31 02:27:40,840 ERROR raise Exception("One of the xcmp jobs failed.") 2020-01-31 02:27:40,840 ERROR Exception: One of the xcmp jobs failed. Command exited with non-zero status 1 Command being timed: "/biodata/Nicola/workspace/hap.py/build/bin/hap.py ../VCF/HG00096.chr1.heterozygous.reformatted.vcf HG00096.chr1.48x.FW+RC.m6.snp.vcf.sorted.vcf -o HG00096.chr1.48x.FW+RC.m6.snp.report -r ../REF/chr1.fasta" User time (seconds): 35050.65 System time (seconds): 396.40 Percent of CPU this job got: 100% Elapsed (wall clock) time (h:mm:ss or m:ss): 9:48:46 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 123559796 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 41063 Minor (reclaiming a frame) page faults: 91436887 Voluntary context switches: 399661 Involuntary context switches: 344160 Swaps: 0 File system inputs: 7524056 File system outputs: 32288 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 1