diamond
diamond copied to clipboard
bbuchfink
diamond hangs at Deallocating buffers...
hey,
i have a diamond blastp job of an faa file of 24,195 entries against the uniref90 database. The job has been running for days but it seems like 0 cpus are being used. I checked $TMPDIR which is empty and the log has been stuck at Deallocating buffers for 4 days.
Deallocating memory... [0s]
Deallocating buffers... [5.488s]
Clearing query masking... [0.231s]
Opening temporary output file... [0.001s]
Computing alignments... Loading trace points... [6.147s]
Sorting trace points... [1.049s]
Computing alignments... [538.678s]
Deallocating buffers...
Here is the commmand being run
name=uniref90
cpus=64
query=gomc_all.faa
db=./uniprot_db/$name/$name.dmnd
tmpdir=$TMPDIR
outdir=./db/diamond_out/$name
mkdir -p $outdir
/usr/bin/time -v diamond blastp \
--index-chunks 1 \
--id 90 \
--subject-cover 80 \
--query-cover 80 \
--max-target-seqs 1 \
--threads $cpus \
--db $db \
--query $query \
--out $outdir/$name.m8 \
--tmpdir $tmpdir
strace showed that the job is doing FUTEX_WAIT.
Some additional info
diamond version: 2.1.8
installation: via bioconda
OS: CentOS 7.9 x86_64
$TMPDIR is not full
Rerunning this process will take a long time so I am looking for more understanding in terms of whether this is normal/if not how should I fix it?
Thanks!
There is a known issue that is potentially causing this. It will be fixed in the next release. To access this patch now, you can clone the repository and compile from source: https://github.com/bbuchfink/diamond/commit/711f910906990e66070075445ecba3448fce3632
Thank you for your prompt reply! I tried compiling from source by doing the following
git clone https://github.com/bbuchfink/diamond.git
git checkout 711f910
cd diamond
mkdir bin
cd bin
cmake ..
make -j $(nproc --all)
but I am having the following error:
Scanning dependencies of target arch_sse4_1
Scanning dependencies of target arch_avx2
Scanning dependencies of target arch_generic
[ 0%] [ 0%] [ 0%] [ 1%] Building CXX object CMakeFiles/arch_sse4_1.dir/src/dp/swipe/banded_3frame_swipe.cpp.o
Building CXX object CMakeFiles/arch_avx2.dir/src/dp/swipe/banded_3frame_swipe.cpp.o
Building CXX object CMakeFiles/arch_generic.dir/src/dp/swipe/banded_3frame_swipe.cpp.o
Building CXX object CMakeFiles/arch_avx2.dir/src/search/stage1_2.cpp.o
[ 2%] Building CXX object CMakeFiles/arch_sse4_1.dir/src/search/stage1_2.cpp.o
[ 3%] Building CXX object CMakeFiles/arch_generic.dir/src/search/stage1_2.cpp.o
[ 3%] Building CXX object CMakeFiles/arch_generic.dir/src/tools/benchmark.cpp.o
[ 3%] Building CXX object CMakeFiles/arch_sse4_1.dir/src/tools/benchmark.cpp.o
[ 3%] Building CXX object CMakeFiles/arch_avx2.dir/src/tools/benchmark.cpp.o
[ 4%] Building CXX object CMakeFiles/arch_avx2.dir/src/dp/swipe/swipe_wrapper.cpp.o
[ 5%] Building CXX object CMakeFiles/arch_sse4_1.dir/src/dp/swipe/swipe_wrapper.cpp.o
[ 5%] Building CXX object CMakeFiles/arch_sse4_1.dir/src/masking/tantan.cpp.o
[ 6%] Building CXX object CMakeFiles/arch_generic.dir/src/dp/swipe/swipe_wrapper.cpp.o
[ 7%] Building CXX object CMakeFiles/arch_sse4_1.dir/src/dp/scan_diags.cpp.o
[ 7%] Building CXX object CMakeFiles/arch_generic.dir/src/masking/tantan.cpp.o
[ 7%] Building CXX object CMakeFiles/arch_sse4_1.dir/src/dp/ungapped_simd.cpp.o
[ 8%] Building CXX object CMakeFiles/arch_generic.dir/src/dp/scan_diags.cpp.o
[ 9%] Building CXX object CMakeFiles/arch_sse4_1.dir/src/dp/swipe/anchored_wrapper.cpp.o
[ 9%] Building CXX object CMakeFiles/arch_generic.dir/src/dp/ungapped_simd.cpp.o
[ 10%] Building CXX object CMakeFiles/arch_sse4_1.dir/src/dp/score_profile.cpp.o
[ 10%] Building CXX object CMakeFiles/arch_avx2.dir/src/masking/tantan.cpp.o
[ 11%] Building CXX object CMakeFiles/arch_generic.dir/src/dp/swipe/anchored_wrapper.cpp.o
[ 12%] Building CXX object CMakeFiles/arch_generic.dir/src/dp/score_profile.cpp.o
[ 13%] Building CXX object CMakeFiles/arch_avx2.dir/src/dp/scan_diags.cpp.o
[ 13%] Building CXX object CMakeFiles/arch_avx2.dir/src/dp/ungapped_simd.cpp.o
[ 14%] Building CXX object CMakeFiles/arch_avx2.dir/src/dp/swipe/anchored_wrapper.cpp.o
[ 15%] Building CXX object CMakeFiles/arch_avx2.dir/src/dp/score_profile.cpp.o
[ 15%] Built target arch_generic
[ 15%] Built target arch_avx2
[ 15%] Built target arch_sse4_1
Scanning dependencies of target diamond
[ 15%] [ 15%] [ 16%] [ 17%] Building CXX object CMakeFiles/diamond.dir/src/stats/score_matrix.cpp.o
Building CXX object CMakeFiles/diamond.dir/src/run/main.cpp.o
Building CXX object CMakeFiles/diamond.dir/src/basic/config.cpp.o
Building CXX object CMakeFiles/diamond.dir/src/data/queries.cpp.o
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:114:25: warning: multi-character character constant [-Wmultichar]
} while ((int64_t)b * 1'000'000'000 < db_letters && b < max_b && c < max_c);
^
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:114:33: warning: missing terminating ' character [enabled by default]
} while ((int64_t)b * 1'000'000'000 < db_letters && b < max_b && c < max_c);
^
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:114:2: error: missing terminating ' character
} while ((int64_t)b * 1'000'000'000 < db_letters && b < max_b && c < max_c);
^
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:413:111: warning: multi-character character constant [-Wmultichar]
("file-buffer-size", 0, "file buffer size in bytes (default=67108864)", file_buffer_size, INT64_C(67'108'864))
^
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:443:106: warning: multi-character character constant [-Wmultichar]
("swipe-task-size", 0, "task size for DP parallelism (100000000)", swipe_task_size, INT64_C(100'000'000))
^
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:613:58: warning: multi-character character constant [-Wmultichar]
("max-swipe-dp", 0, "", max_swipe_dp, INT64_C(1'000'000))
^
[ 17%] Building CXX object CMakeFiles/diamond.dir/src/data/seed_histogram.cpp.o
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp: In function ‘std::pair<double, int> block_size(int64_t, int64_t, Sensitivity, bool, int)’:
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:114:25: error: expected ‘)’ before '\x303030'
} while ((int64_t)b * 1'000'000'000 < db_letters && b < max_b && c < max_c);
^
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:114:25: error: expected ‘;’ before '\x303030'
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:114:30: error: expected ‘;’ before numeric constant
} while ((int64_t)b * 1'000'000'000 < db_letters && b < max_b && c < max_c);
^
In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/stdint.h:9:0,
from /usr/include/c++/4.8.2/cstdint:41,
from /usr/include/c++/4.8.2/bits/char_traits.h:376,
from /usr/include/c++/4.8.2/string:40,
from /usr/include/c++/4.8.2/stdexcept:39,
from /usr/include/c++/4.8.2/array:38,
from /usr/include/c++/4.8.2/tuple:39,
from /usr/include/c++/4.8.2/functional:55,
from /usr/include/c++/4.8.2/memory:79,
from /home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:23:
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp: In constructor ‘Config::Config(int, const char**, bool, CommandLineParser&)’:
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:413:103: error: expected ‘)’ before '\x313038'
("file-buffer-size", 0, "file buffer size in bytes (default=67108864)", file_buffer_size, INT64_C(67'108'864))
^
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:413:112: error: no match for call to ‘(OptionsGroup::AddFunc) (const char [17], int, const char [45], int64_t&, int)’
("file-buffer-size", 0, "file buffer size in bytes (default=67108864)", file_buffer_size, INT64_C(67'108'864))
^
In file included from /home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:26:0:
/home/y1weng/packages-code/DIAMOND/diamond/src/util/command_line_parser.h:207:9: note: candidate is:
struct AddFunc
^
/home/y1weng/packages-code/DIAMOND/diamond/src/util/command_line_parser.h:213:12: note: template<class T> OptionsGroup::AddFunc& OptionsGroup::AddFunc::operator()(const char*, char, const char*, T&, T, int)
AddFunc& operator()(const char* id, char short_id, const char* desc, T& store, T def = T(), const int min_count = 1)
^
/home/y1weng/packages-code/DIAMOND/diamond/src/util/command_line_parser.h:213:12: note: template argument deduction/substitution failed:
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:413:112: note: deduced conflicting types for parameter ‘T’ (‘long int’ and ‘int’)
("file-buffer-size", 0, "file buffer size in bytes (default=67108864)", file_buffer_size, INT64_C(67'108'864))
^
In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/stdint.h:9:0,
from /usr/include/c++/4.8.2/cstdint:41,
from /usr/include/c++/4.8.2/bits/char_traits.h:376,
from /usr/include/c++/4.8.2/string:40,
from /usr/include/c++/4.8.2/stdexcept:39,
from /usr/include/c++/4.8.2/array:38,
from /usr/include/c++/4.8.2/tuple:39,
from /usr/include/c++/4.8.2/functional:55,
from /usr/include/c++/4.8.2/memory:79,
from /home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:23:
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:443:98: error: expected ‘)’ before '\x303030'
("swipe-task-size", 0, "task size for DP parallelism (100000000)", swipe_task_size, INT64_C(100'000'000))
^
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:443:107: error: no match for call to ‘(OptionsGroup::AddFunc) (const char [16], int, const char [41], int64_t&, int)’
("swipe-task-size", 0, "task size for DP parallelism (100000000)", swipe_task_size, INT64_C(100'000'000))
^
In file included from /home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:26:0:
/home/y1weng/packages-code/DIAMOND/diamond/src/util/command_line_parser.h:207:9: note: candidate is:
struct AddFunc
^
/home/y1weng/packages-code/DIAMOND/diamond/src/util/command_line_parser.h:213:12: note: template<class T> OptionsGroup::AddFunc& OptionsGroup::AddFunc::operator()(const char*, char, const char*, T&, T, int)
AddFunc& operator()(const char* id, char short_id, const char* desc, T& store, T def = T(), const int min_count = 1)
^
/home/y1weng/packages-code/DIAMOND/diamond/src/util/command_line_parser.h:213:12: note: template argument deduction/substitution failed:
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:443:107: note: deduced conflicting types for parameter ‘T’ (‘long int’ and ‘int’)
("swipe-task-size", 0, "task size for DP parallelism (100000000)", swipe_task_size, INT64_C(100'000'000))
^
In file included from /usr/lib/gcc/x86_64-redhat-linux/4.8.5/include/stdint.h:9:0,
from /usr/include/c++/4.8.2/cstdint:41,
from /usr/include/c++/4.8.2/bits/char_traits.h:376,
from /usr/include/c++/4.8.2/string:40,
from /usr/include/c++/4.8.2/stdexcept:39,
from /usr/include/c++/4.8.2/array:38,
from /usr/include/c++/4.8.2/tuple:39,
from /usr/include/c++/4.8.2/functional:55,
from /usr/include/c++/4.8.2/memory:79,
from /home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:23:
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:613:50: error: expected ‘)’ before '\x303030'
("max-swipe-dp", 0, "", max_swipe_dp, INT64_C(1'000'000))
^
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:613:59: error: no match for call to ‘(OptionsGroup::AddFunc) (const char [13], int, const char [1], int64_t&, int)’
("max-swipe-dp", 0, "", max_swipe_dp, INT64_C(1'000'000))
^
In file included from /home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:26:0:
/home/y1weng/packages-code/DIAMOND/diamond/src/util/command_line_parser.h:207:9: note: candidate is:
struct AddFunc
^
/home/y1weng/packages-code/DIAMOND/diamond/src/util/command_line_parser.h:213:12: note: template<class T> OptionsGroup::AddFunc& OptionsGroup::AddFunc::operator()(const char*, char, const char*, T&, T, int)
AddFunc& operator()(const char* id, char short_id, const char* desc, T& store, T def = T(), const int min_count = 1)
^
/home/y1weng/packages-code/DIAMOND/diamond/src/util/command_line_parser.h:213:12: note: template argument deduction/substitution failed:
/home/y1weng/packages-code/DIAMOND/diamond/src/basic/config.cpp:613:59: note: deduced conflicting types for parameter ‘T’ (‘long int’ and ‘int’)
("max-swipe-dp", 0, "", max_swipe_dp, INT64_C(1'000'000))
^
[ 18%] Building CXX object CMakeFiles/diamond.dir/src/output/daa/daa_record.cpp.o
At global scope:
cc1plus: warning: unrecognized command line option "-Wno-unknown-warning-option" [enabled by default]
cc1plus: warning: unrecognized command line option "-Wno-deprecated-copy" [enabled by default]
cc1plus: warning: unrecognized command line option "-Wno-implicit-fallthrough" [enabled by default]
make[2]: *** [CMakeFiles/diamond.dir/src/basic/config.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/diamond.dir/all] Error 2
make: *** [all] Error 2
Addition info:
> cmake --version
cmake version 2.8.12.2
> gcc --version
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44)
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Any idea on why and how to fix this?
Thank you!
Appreciate your prompt reponse!
Confirm successful installation. Rerunning the job right now. Will keep you posted on if this solves the hanging issue.