Error 137?

[YingChen94]

Hello, I encountered an error of Error 137 (see log output below). Do you have any insights what might cause the issue? Thanks!

...
Processed 70000000 kmers.
Processed 70100000 kmers.
Processed 70200000 kmers.
Processed 70300000 kmers.
Processed 70400000 kmers.
Bloom filter stats:
        #counters               = 80000000000
        #size (B)               = 10000000000
        popcount                = 70426792
        FPR                     = 0.088%
Writing a 10000000000 byte filter to CF8.k40.bf on disk.
pigz -p88 -cd CF8.fq.gz | command time -v -o CF8.k40-w32.physlr.tsv.gz.time /home/ying_local/miniconda3/envs/physlr_test/bin/share/physlr-1.0.4-6/src/physlr-indexlr -t88 -k40 -w32 -r CF8.k40.bf - | pigz -p88 >CF8.k40-w32.physlr.tsv.gz
Loading repeat Bloom filter from CF8.k40.bf
Finished loading repeat Bloom filter
physlr-indexlr: Using more than 48 threads with Bloom filter does not scale, reverting to 48.
pigz -p88 -cd CF8.k40-w32.physlr.tsv.gz | command time -v -o CF8.k40-w32.n100-5000.physlr.tsv.gz.time /home/ying_local/miniconda3/envs/physlr_test/bin/share/physlr-1.0.4-6/src/physlr-filter-barcodes -n100 -N5000 - | pigz -p88 >CF8.k40-w32.n100-5000.physlr.tsv.gz
Time at readMxs (ms): 5147764
make: *** [/home/ying_local/miniconda3/envs/physlr_test/bin/share/physlr-1.0.4-6/bin/physlr-make:903: CF8.k40-w32.n100-5000.physlr.tsv.gz] Error 137
make: *** Deleting file 'CF8.k40-w32.n100-5000.physlr.tsv.gz'

[lcoombe]

Hi @YingChen94,

How much memory do you have available on your machine? And, what is the memory usage listed in the CF8.k40-w32.n100-5000.physlr.tsv.gz.time file from the failed step? Error 137 generally means that you have run out of memory, so the job was killed. So, you could try running the job on a larger machine if the memory was the issue.

Thank you for your interest in physlr! Lauren

[YingChen94]

Indeed it looks like it used up all the memory. Do you know how much memory physlr needs for a 4Gb genome? Thanks!

Command terminated by signal 9
        Command being timed: "/home/ying_local/miniconda3/envs/physlr_test/bin/share/physlr-1.0.4-6/src/physlr-filter-barcodes -n100 -N5000 -"
        User time (seconds): 4249.81
        System time (seconds): 1636.78
        Percent of CPU this job got: 95%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 1:42:43
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 717240644
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 14
        Minor (reclaiming a frame) page faults: 215135707
        Voluntary context switches: 500404
        Involuntary context switches: 30717350
        Swaps: 0
        File system inputs: 1864
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[lcoombe]

@aafshinfard - Do you have any guidelines for the expected memory usage, and any suggestions for reducing the memory of this step?