agamemnon
agamemnon copied to clipboard
Published
20 hours ago •
ivlachos
ivlachos
failure compiling Pufferfish, a dependency of Agamemnon
I've compiled TBB in a separate folder, I've, also, installed TBB via conda but cmake doesn't see it and defaults to /path/agamemnon/agamemnon/pufferfish/external/install. Compilation fails due to multiple htslib errors. I've compiled htslib as well, but agamemnon wants to install it from scratch every time I run the install script. Your help would be greatly appreciated. We are using RHEL 7.9 in HPC environment.
(agamemnon-0.1.0) [user@node](master|✔)» pwd
/path/agamemnon/agamemnon/pufferfish/build
(agamemnon-0.1.0) [user@node](master|✔)» cmake -DCMAKE_PREFIX_PATH=../../tbb ..
CC: /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/bin/x86_64-conda-linux-gnu-cc
CC version:
version: 1.0.0
Top-level source directory variable not set externally; setting it to /path/agamemnon/agamemnon/pufferfish
-- Could NOT find Jemalloc (missing: JEMALLOC_LIBRARY) -- Could NOT find TBB (missing: TBB_INCLUDE_DIRS TBB_LIBRARIES tbb tbbmalloc tbbmalloc_proxy) (Required is at least version "2018.0")
Build system will fetch and build Intel Threading Building Blocks
==================================================================
TBB_INCLUDE_DIRS = /path/agamemnon/agamemnon/pufferfish/external/install/include
TBB_LIBRARY_DIRS = /path/agamemnon/agamemnon/pufferfish/external/install/lib
TBB_LIBRARIES = /path/agamemnon/agamemnon/pufferfish/external/install/lib/libtbb.so;/path/agamemnon/agamemnon/pufferfish/external/install/lib/libtbbmalloc.so
Building extra components
-- Configuring done
-- Generating done
-- Build files have been written to: /path/agamemnon/agamemnon/pufferfish/build
(agamemnon-0.1.0) [user@node](master|✔)» conda list tbb
# packages in environment at /path/Anaconda3/2021.05/envs/agamemnon-0.1.0:
#
# Name Version Build Channel
tbb 2021.5.0 h4bd325d_0 conda-forge
(agamemnon-0.1.0) [user@node](master|✔)» conda list jemalloc
# packages in environment at /path/Anaconda3/2021.05/envs/agamemnon-0.1.0:
#
# Name Version Build Channel
jemalloc 5.2.1 h9c3ff4c_6 conda-forge
libjemalloc 5.2.1 h9c3ff4c_6 conda-forge
(agamemnon-0.1.0) [user@node](:d8805c4|✔)» gcc --version
gcc (crosstool-NG 1.23.0.449-a04d0) 7.3.0
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
(agamemnon-0.1.0) [user@node](:d8805c4|✔)» make -j 10
Consolidate compiler generated dependencies of target ntcard
Consolidate compiler generated dependencies of target Async++
Consolidate compiler generated dependencies of target graphdump
Consolidate compiler generated dependencies of target ksw2pp_sse4
Consolidate compiler generated dependencies of target ksw2pp_basic
Consolidate compiler generated dependencies of target ksw2pp_sse2
Consolidate compiler generated dependencies of target twopaco
[ 6%] Performing build step for 'libseqlib'
[ 17%] Built target libSetCover
[ 17%] Built target libtbb
[ 23%] Built target ksw2pp_sse2
[ 23%] Built target ksw2pp_sse4
Making all in src
[ 33%] Built target ntcard
[ 37%] Built target ksw2pp_basic
[ 37%] Built target Async++
Consolidate compiler generated dependencies of target bcalm_pufferize
[ 41%] Built target graphdump
Consolidate compiler generated dependencies of target getLineage
Consolidate compiler generated dependencies of target puffer
[ 51%] Built target twopaco
[ 54%] Linking CXX static library libksw2pp.a
[ 57%] Built target getLineage
[ 57%] Built target bcalm_pufferize
[ 57%] Built target ksw2pp
[ 82%] Built target puffer
[ 85%] Building CXX object src/CMakeFiles/edgedensity2.dir/EdgeDensityCustom.cpp.o
[ 85%] Building CXX object src/CMakeFiles/pufferfish.dir/Pufferfish.cpp.o
[ 85%] Building CXX object src/CMakeFiles/edgedensity.dir/EdgeDensity.cpp.o
In file included from ../SeqLib/BamWalker.h:16,
from BamWriter.cpp:1:
../htslib/htslib/cram_io.h:176:34: error: use of enum 'cram_block_method_int' without previous declaration
176 | char *cram_block_method2str(enum cram_block_method_int m);
| ^~~~~~~~~~~~~~~~~~~~~
../htslib/htslib/cram_io.h: In function 'cram_block* cram_get_block_by_id(cram_slice*, int)':
../htslib/htslib/cram_io.h:186:14: error: invalid use of incomplete type 'cram_slice' {aka 'struct cram_slice'}
186 | if (slice->block_by_id && v < 256) {
| ^~
In file included from ../SeqLib/BamWalker.h:15,
from BamWriter.cpp:1:
../htslib/htslib/cram.h:96:16: note: forward declaration of 'cram_slice' {aka 'struct cram_slice'}
96 | typedef struct cram_slice cram_slice;
| ^~~~~~~~~~
In file included from ../SeqLib/BamWalker.h:16,
from BamWriter.cpp:1:
../htslib/htslib/cram_io.h:187:21: error: invalid use of incomplete type 'cram_slice' {aka 'struct cram_slice'}
187 | return slice->block_by_id[v];
| ^~
In file included from ../SeqLib/BamWalker.h:15,
from BamWriter.cpp:1:
../htslib/htslib/cram.h:96:16: note: forward declaration of 'cram_slice' {aka 'struct cram_slice'}
96 | typedef struct cram_slice cram_slice;
| ^~~~~~~~~~
In file included from ../SeqLib/BamWalker.h:16,
from BamWriter.cpp:1:
../htslib/htslib/cram_io.h:190:18: error: invalid use of incomplete type 'cram_slice' {aka 'struct cram_slice'}
190 | if (slice->block_by_id &&
| ^~
In file included from ../SeqLib/BamWalker.h:15,
from BamWriter.cpp:1:
../htslib/htslib/cram.h:96:16: note: forward declaration of 'cram_slice' {aka 'struct cram_slice'}
96 | typedef struct cram_slice cram_slice;
| ^~~~~~~~~~
In file included from ../SeqLib/BamWalker.h:16,
from BamWriter.cpp:1:
../htslib/htslib/cram_io.h:191:18: error: invalid use of incomplete type 'cram_slice' {aka 'struct cram_slice'}
191 | slice->block_by_id[v] &&
| ^~
In file included from ../SeqLib/BamWalker.h:15,
from BamWriter.cpp:1:
../htslib/htslib/cram.h:96:16: note: forward declaration of 'cram_slice' {aka 'struct cram_slice'}
96 | typedef struct cram_slice cram_slice;
| ^~~~~~~~~~
In file included from ../SeqLib/BamWalker.h:16,
from BamWriter.cpp:1:
../htslib/htslib/cram_io.h:192:18: error: invalid use of incomplete type 'cram_slice' {aka 'struct cram_slice'}
192 | slice->block_by_id[v]->content_id == id)
| ^~
In file included from ../SeqLib/BamWalker.h:15,
from BamWriter.cpp:1:
../htslib/htslib/cram.h:96:16: note: forward declaration of 'cram_slice' {aka 'struct cram_slice'}
96 | typedef struct cram_slice cram_slice;
| ^~~~~~~~~~
In file included from ../SeqLib/BamWalker.h:16,
from BamWriter.cpp:1:
../htslib/htslib/cram_io.h:193:25: error: invalid use of incomplete type 'cram_slice' {aka 'struct cram_slice'}
193 | return slice->block_by_id[v];
| ^~
In file included from ../SeqLib/BamWalker.h:15,
from BamWriter.cpp:1:
../htslib/htslib/cram.h:96:16: note: forward declaration of 'cram_slice' {aka 'struct cram_slice'}
96 | typedef struct cram_slice cram_slice;
| ^~~~~~~~~~
In file included from ../SeqLib/BamWalker.h:16,
from BamWriter.cpp:1:
../htslib/htslib/cram_io.h:197:30: error: invalid use of incomplete type 'cram_slice' {aka 'struct cram_slice'}
197 | for (i = 0; i < slice->hdr->num_blocks; i++) {
| ^~
In file included from ../SeqLib/BamWalker.h:15,
from BamWriter.cpp:1:
../htslib/htslib/cram.h:96:16: note: forward declaration of 'cram_slice' {aka 'struct cram_slice'}
96 | typedef struct cram_slice cram_slice;
| ^~~~~~~~~~
In file included from ../SeqLib/BamWalker.h:16,
from BamWriter.cpp:1:
../htslib/htslib/cram_io.h:198:34: error: invalid use of incomplete type 'cram_slice' {aka 'struct cram_slice'}
198 | cram_block *b = slice->block[i];
| ^~
…
In file included from ../SeqLib/BamWalker.h:16,
from ../SeqLib/BamReader.h:6,
from BamReader.cpp:1:
../htslib/htslib/cram_io.h: In function 'hFILE* cram_hfile(cram_fd*)':
../htslib/htslib/cram_io.h:643:14: error: invalid use of incomplete type 'cram_fd' {aka 'struct cram_fd'}
643 | return fd->fp;
| ^~
In file included from ../SeqLib/BamRecord.h:13,
from ../SeqLib/GenomicRegionCollection.h:11,
from ../SeqLib/ReadFilter.h:12,
from ../SeqLib/BamReader.h:5,
from BamReader.cpp:1:
../htslib/htslib/hts.h:57:8: note: forward declaration of 'cram_fd' {aka 'struct cram_fd'}
57 | struct cram_fd;
| ^~~~~~~
make[5]: *** [Makefile:492: libseqlib_a-BamReader.o] Error 1
make[4]: *** [Makefile:358: all-recursive] Error 1
make[3]: *** [Makefile:299: all] Error 2
make[2]: *** [CMakeFiles/libseqlib.dir/build.make:85: libseqlib-prefix/src/libseqlib-stamp/libseqlib-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:239: CMakeFiles/libseqlib.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
...
[ 86%] Linking CXX executable edgedensity
[ 87%] Linking CXX executable edgedensity2
[ 87%] Built target edgedensity
[ 87%] Built target edgedensity2
[ 88%] Linking CXX executable pufferfish
[ 88%] Built target pufferfish
make: *** [Makefile:156: all] Error 2
Hi @ponomarevsy ,
So if I'm correct, you still are able to install the package, but it doesn't go through your TBB binary. right?
Hi @fataltes. I've copied and pasted all warnings and errors I see during the installation (please let me know if you need more information):
» ./install.sh
Building Pufferfish, Puffaligner and Cedar ...
...
-- Could NOT find Jemalloc (missing: JEMALLOC_LIBRARY)
-- Could NOT find TBB (missing: TBB_INCLUDE_DIRS TBB_LIBRARIES tbb tbbmalloc tbbmalloc_proxy) (Required is at least version "2018.0")
...
/path/agamemnon/pufferfish/external/seqlib/missing: Unknown `--is-lightweight' option
Try `/path/agamemnon/pufferfish/external/seqlib/missing --help' for more information
...
Making all in htslib
/path/Anaconda3/2021.05/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: cannot find -lz
/path/Anaconda3/2021.05/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: cannot find -lbz2
/path/Anaconda3/2021.05/bin/../lib/gcc/x86_64-conda_cos6-linux-gnu/7.3.0/../../../../x86_64-conda_cos6-linux-gnu/bin/ld: cannot find -llzma
collect2: error: ld returned 1 exit status
make[5]: *** [Makefile:260: libhts.so] Error 1
make[4]: *** [Makefile:358: all-recursive] Error 1
make[3]: *** [Makefile:299: all] Error 2
make[2]: *** [CMakeFiles/libseqlib.dir/build.make:85: libseqlib-prefix/src/libseqlib-stamp/libseqlib-build] Error 2
make[1]: *** [CMakeFiles/Makefile2:239: CMakeFiles/libseqlib.dir/all] Error 2
make: *** [Makefile:156: all] Error 2
...
mkdir: cannot create directory ‘binaries’: File exists
mv: cannot move ‘pufferfish’ to ‘binaries/pufferfish’: File exists
mkdir: cannot create directory ‘cedar’: File exists
cp: cannot stat ‘pufferfish’: No such file or directory
mv: cannot stat ‘cedar’: No such file or directory
mv: cannot move ‘hisat2’ to ‘binaries/hisat2’: File exists
chmod: cannot access ‘pufferfish/pufferfish’: No such file or directory
chmod: cannot access ‘cedar/cedar’: No such file or directory
Thank you @ponomarevsy ,
I see that it has not been able to install bz2 and lzma as well. Just a few things to check with you:
- you are using install.sh script. right?
- Make sure to clean everything and run the script from scratch on an empty directory (there are a few warnings about the main directories already existing)
- Make sure the script has the write and execute access through chmod and able to download external libraries from internet.
At that point, we should at least not see the mkdir and mv warnings.
Also, what is your system and the spec of the OS?
Good morning, @fataltes. Please see my answers below...
The following libraries are installed:
(agamemnon-0.1.0) [user@node](master|✚6…)» conda list xz
# packages in environment at /path/Anaconda3/2021.05/envs/agamemnon-0.1.0:
#
# Name Version Build Channel
xz 5.2.5 h516909a_1 conda-forge
(agamemnon-0.1.0) [user@node](master|✚6…)» conda list zlib
# packages in environment at /path/Anaconda3/2021.05/envs/agamemnon-0.1.0:
#
# Name Version Build Channel
libzlib 1.2.11 h36c2ea0_1013 conda-forge
zlib 1.2.11 h36c2ea0_1013 conda-forge
(agamemnon-0.1.0) [user@node](master|✚6…)» conda list bzip
# packages in environment at /path/Anaconda3/2021.05/envs/agamemnon-0.1.0:
#
# Name Version Build Channel
bzip2 1.0.8 h7f98852_4 conda-forge
- you are using install.sh script. right?
That is correct.
- Make sure to clean everything and run the script from scratch on an empty directory (there are a few warnings about the main directories already existing)
Started from scratch and got rid of some errors at the end (see my previous email). Htslib still fails.
(agamemnon-0.1.0) [user@node](master|✚1)» ./install.sh
Building Pufferfish, Puffaligner and Cedar ...
Cloning into 'pufferfish'...
remote: Enumerating objects: 9794, done.
remote: Counting objects: 100% (874/874), done.
remote: Compressing objects: 100% (586/586), done.
remote: Total 9794 (delta 547), reused 526 (delta 285), pack-reused 8920
Receiving objects: 100% (9794/9794), 6.88 MiB | 0 bytes/s, done.
Resolving deltas: 100% (7086/7086), done.
Note: checking out 'tags/v1.1.0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b new_branch_name
HEAD is now at d8805c4... Moved Cedar-related codes to a subdirectory named cedar under src and include
CC: /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/bin/x86_64-conda-linux-gnu-cc
CC version:
Detected non-ARM host. Setting USE_ARM to false.
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/bin/x86_64-conda-linux-gnu-cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/bin/x86_64-conda-linux-gnu-c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
version: 1.0.0
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
Top-level source directory variable not set externally; setting it to /path/agamemnon/agamemnon/pufferfish
not setting -DHAVE_NUMERIC_LIMITS128
-- Found PkgConfig: /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/bin/pkg-config (found version "0.29.2")
-- Found Jemalloc: /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/lib/libjemalloc.so (found version "5.2.1-0-gea6b3e973b477b8061e0076bb257dbd7f3faa756")
-- Could NOT find TBB (missing: TBB_INCLUDE_DIRS TBB_LIBRARIES tbb tbbmalloc tbbmalloc_proxy) (Required is at least version "2018.0")
Build system will fetch and build Intel Threading Building Blocks
==================================================================
-- Looking for lzma_auto_decoder in /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/lib/liblzma.so
-- Looking for lzma_auto_decoder in /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/lib/liblzma.so - found
-- Looking for lzma_easy_encoder in /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/lib/liblzma.so
-- Looking for lzma_easy_encoder in /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/lib/liblzma.so - found
-- Looking for lzma_lzma_preset in /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/lib/liblzma.so
-- Looking for lzma_lzma_preset in /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/lib/liblzma.so - found
-- Found LibLZMA: /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/lib/liblzma.so (found version "5.2.5")
Found liblzma library: /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/lib/liblzma.so
===========================================
-- Found BZip2: /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/lib/libbz2.so (found version "1.0.8")
-- Looking for BZ2_bzCompressInit
-- Looking for BZ2_bzCompressInit - found
Found libbz2 library: /path/Anaconda3/2021.05/envs/agamemnon-0.1.0/lib/libbz2.so
===========================================
TBB_INCLUDE_DIRS = /path/agamemnon/agamemnon/pufferfish/external/install/include
TBB_LIBRARY_DIRS = /path/agamemnon/agamemnon/pufferfish/external/install/lib
TBB_LIBRARIES = /path/agamemnon/agamemnon/pufferfish/external/install/lib/libtbb.so;/path/agamemnon/agamemnon/pufferfish/external/install/lib/libtbbmalloc.so
Building extra components
-- Configuring done
-- Generating done
-- Build files have been written to: /path/agamemnon/agamemnon/pufferfish/build
...
cp: cannot stat ‘pufferfish’: No such file or directory
mv: cannot stat ‘cedar’: No such file or directory
chmod: cannot access ‘pufferfish/pufferfish’: No such file or directory
chmod: cannot access ‘cedar/cedar’: No such file or directory
- Make sure the script has the write and execute access through chmod and able to download external libraries from internet.
That should not be a problem as I am using a sysadmin account (RHEL 7.9).