Diamond Blastp - Failed to allocate sufficient memory. Please refer to the manual for instructions on memory usage.

[Oceazh]

Hi. Thank you for developing such a highly efficient and useful software, but I'm running into a bit of an issue. I'm trying to run diamond blastp on a file with 2.76 million sequences and got this error. I have allocated 120GB memory for this task through a Slurm script, and there is 2TB available on the local disk. The database.dmnd database I used is 1.3 GB. diamond blastp -p 12 -q query_genes.faa -d query_genes.dmnd -o out.tsv --sensitive -c1 -b 6 --tmpdir ../../tmp

diamond v2.1.12.166 (C) Max Planck Society for the Advancement of Science, Benjamin Buchfink, University of Tuebingen Documentation, support and updates available at http://www.diamondsearch.org Please cite: http://dx.doi.org/10.1038/s41592-021-01101-x Nature Methods (2021)

#CPU threads: 12 Scoring parameters: (Matrix=BLOSUM62 Lambda=0.267 K=0.041 Penalties=11/1) Temporary directory: ../../tmp #Target sequences to report alignments for: 25 Opening the database... [0.234s] Database: query_genes.dmnd (type: Diamond database, sequences: 2768812, letters: 909142967) Block size = 6000000000 Opening the input file... [0.143s] Opening the output file... [0s] Loading query sequences... [12.553s] Masking queries... [6.022s] Algorithm: Double-indexed Building query histograms... [24.539s] Seeking in database... [0s] Loading reference sequences... [3.674s] Masking reference... [5.953s] Initializing temporary storage... [0.046s] Building reference histograms... [24.256s] Allocating buffers... Failed to allocate sufficient memory. Please refer to the manual for instructions on memory usage.

[Oceazh]

I ran this order again and encountered a new error. diamond v2.1.12.166 (C) Max Planck Society for the Advancement of Science, Benjamin Buchfink, University of Tuebingen Documentation, support and updates available at http://www.diamondsearch.org Please cite: http://dx.doi.org/10.1038/s41592-021-01101-x Nature Methods (2021)

#CPU threads: 12 Scoring parameters: (Matrix=BLOSUM62 Lambda=0.267 K=0.041 Penalties=11/1) Temporary directory: ./tmp3 #Target sequences to report alignments for: 6400 Opening the database... [0.057s] Database: output_aai3/similarity/query_genes (type: Diamond database, sequences: 2768812, letters: 909142967) Block size = 6000000000 Opening the input file... [0.213s] Opening the output file... [0s] Loading query sequences... [11.013s] Masking queries... [4.724s] Algorithm: Double-indexed Building query histograms... [6.2s] Seeking in database... [0s] Loading reference sequences... [1.408s] Masking reference... [5.089s] Initializing temporary storage... [0.01s] Building reference histograms... [7.065s] Allocating buffers... [0s] Processing query block 1, reference block 1/1, shape 1/2. Building reference seed array... [3.908s] Building query seed array... [3.926s] Computing hash join... [3.857s] Masking low complexity seeds... [9.204s] Searching alignments... [287.406s] Deallocating memory... [0s] Processing query block 1, reference block 1/1, shape 2/2. Building reference seed array... [3.236s] Building query seed array... [4.549s] Computing hash join... [3.415s] Masking low complexity seeds... [9.571s] Searching alignments... [316.351s] Deallocating memory... [0s] Deallocating buffers... [0.03s] Clearing query masking... [0.114s] Computing alignments... terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc

[bbuchfink]

You can try to set a lower block size.