gapless
gapless copied to clipboard
schmeing
pipeline crashed : scaffold (key error)
Hi @schmeing
I've tried troubleshooting with different conda installations and I can't get Gapless to run on my assembly.
The command that fails and yields pipeline crashed : scaffold is
gapless.sh -j 10 -i /hpcdata/bcbb/homc/Tcas_filtcontigs.fa -t pb_hifi /hpcdata/bcbb/homc/Tcas_PacBio/PACBIO_DATA/XECAF_20221215_S64018_PL100274063-1_A01.ccs.fastq
From gapless_scaffold.log:
0:00:04.577490 Reading in original assembly
0:00:04.777931 Loading repeats
0:00:05.460704 Filtering mappings
Traceback (most recent call last):
File "/hpcdata/bcbb/homc/conda_envs/envs/gapless/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3621, in get_loc
return self._engine.get_loc(casted_key)
File "pandas/_libs/index.pyx", line 136, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/index.pyx", line 163, in pandas._libs.index.IndexEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 5198, in pandas._libs.hashtable.PyObjectHashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 5206, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'y'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/hpcdata/bcbb/homc/conda_envs/envs/gapless/bin/gapless.py", line 13327, in <module>
main(sys.argv[1:])
File "/hpcdata/bcbb/homc/conda_envs/envs/gapless/bin/gapless.py", line 13156, in main
GaplessScaffold(args[0], args[1], args[2], min_mapq, min_mapping_length, min_length_contig_break, prefix, stats)
File "/hpcdata/bcbb/homc/conda_envs/envs/gapless/bin/gapless.py", line 9069, in GaplessScaffold
mappings, cov_counts, cov_probs, read_names, read_len = ReadMappings(mapping_file, contig_ids, min_mapq, min_mapping_length, keep_all_subreads, alignment_precision, num_read_len_groups, pdf)
File "/hpcdata/bcbb/homc/conda_envs/envs/gapless/bin/gapless.py", line 427, in ReadMappings
cov_probs = GetCoverageProbabilities(cov_counts, pdf)
File "/hpcdata/bcbb/homc/conda_envs/envs/gapless/bin/gapless.py", line 386, in GetCoverageProbabilities
PlotXY(pdf, "Coverage", "% Bins (Size: {})".format(bsize), probs.loc[selection2, 'count'], probs.loc[selection2, 'nbins']/probs.loc[selection2, 'nbin_sum']*100, linex=x_values, liney=nbinom.pmf(x_values,opt_par[0],opt_par[1])*100)
File "/hpcdata/bcbb/homc/conda_envs/envs/gapless/bin/gapless.py", line 285, in PlotXY
sns.lineplot(x=linex, y=liney, color='red', linewidth=2.5)
File "/hpcdata/bcbb/homc/conda_envs/envs/gapless/lib/python3.10/site-packages/seaborn/relational.py", line 645, in lineplot
p.plot(ax, kwargs)
File "/hpcdata/bcbb/homc/conda_envs/envs/gapless/lib/python3.10/site-packages/seaborn/relational.py", line 459, in plot
lines = ax.plot(sub_data["x"], sub_data["y"], **kws)
File "/hpcdata/bcbb/homc/conda_envs/envs/gapless/lib/python3.10/site-packages/pandas/core/frame.py", line 3505, in __getitem__
indexer = self.columns.get_loc(key)
File "/hpcdata/bcbb/homc/conda_envs/envs/gapless/lib/python3.10/site-packages/pandas/core/indexes/base.py", line 3623, in get_loc
raise KeyError(key) from err
Output from conda list:
# packages in environment at /hpcdata/bcbb/homc/conda_envs/envs/gapless:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
alsa-lib 1.2.8 h166bdaf_0 conda-forge
attr 2.5.1 h166bdaf_1 conda-forge
biopython 1.81 py310h1fa729e_0 conda-forge
brotli 1.0.9 h166bdaf_8 conda-forge
brotli-bin 1.0.9 h166bdaf_8 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2023.5.7 hbcca054_0 conda-forge
cairo 1.16.0 hbbf8b49_1016 conda-forge
certifi 2023.5.7 pyhd8ed1ab_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
dbus 1.13.6 h5008d03_3 conda-forge
expat 2.5.0 hcb278e6_1 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.39.4 py310h2372a71_0 conda-forge
freetype 2.12.1 hca18f0e_1 conda-forge
gapless 0.4 hdfd78af_0 bioconda
gettext 0.21.1 h27087fc_0 conda-forge
glib 2.76.3 hfc55251_0 conda-forge
glib-tools 2.76.3 hfc55251_0 conda-forge
graphite2 1.3.13 h58526e2_1001 conda-forge
gst-plugins-base 1.22.3 h938bd60_1 conda-forge
gstreamer 1.22.3 h977cf35_1 conda-forge
harfbuzz 7.3.0 hdb3a94d_0 conda-forge
icu 72.1 hcb278e6_0 conda-forge
k8 0.2.5 hdcf5f25_4 bioconda
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.4.4 py310hbf28c38_1 conda-forge
krb5 1.20.1 h81ceb04_0 conda-forge
lame 3.100 h166bdaf_1003 conda-forge
lcms2 2.15 haa2dc70_1 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libblas 3.9.0 16_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h166bdaf_8 conda-forge
libbrotlidec 1.0.9 h166bdaf_8 conda-forge
libbrotlienc 1.0.9 h166bdaf_8 conda-forge
libcap 2.67 he9d0100_0 conda-forge
libcblas 3.9.0 16_linux64_openblas conda-forge
libclang 16.0.4 default_h1cdf331_0 conda-forge
libclang13 16.0.4 default_h4d60ac6_0 conda-forge
libcups 2.3.3 h36d4200_3 conda-forge
libdeflate 1.18 h0b41bf4_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libevent 2.1.12 h3358134_0 conda-forge
libexpat 2.5.0 hcb278e6_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libflac 1.4.2 h27087fc_0 conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgcrypt 1.10.1 h166bdaf_0 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libglib 2.76.3 hebfc3b9_0 conda-forge
libgomp 12.2.0 h65d4601_19 conda-forge
libgpg-error 1.46 h620e276_0 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
libjpeg-turbo 2.1.5.1 h0b41bf4_0 conda-forge
liblapack 3.9.0 16_linux64_openblas conda-forge
libllvm16 16.0.4 h5cf9203_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libogg 1.3.4 h7f98852_1 conda-forge
libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge
libopus 1.3.1 h7f98852_1 conda-forge
libpng 1.6.39 h753d276_0 conda-forge
libpq 15.3 hbcd7760_1 conda-forge
libsndfile 1.2.0 hb75c966_0 conda-forge
libsqlite 3.42.0 h2797004_0 conda-forge
libstdcxx-ng 12.2.0 h46fd767_19 conda-forge
libsystemd0 253 h8c4010b_1 conda-forge
libtiff 4.5.0 ha587672_6 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libwebp-base 1.3.0 h0b41bf4_0 conda-forge
libxcb 1.15 h0b41bf4_0 conda-forge
libxkbcommon 1.5.0 h5d7e998_3 conda-forge
libxml2 2.11.4 h0d562d8_0 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
matplotlib 3.5.2 py310hff52083_1 conda-forge
matplotlib-base 3.5.2 py310h5701ce4_1 conda-forge
minimap2 2.26 he4a0461_1 bioconda
mpg123 1.31.3 hcb278e6_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
mysql-common 8.0.32 hf1915f5_2 conda-forge
mysql-libs 8.0.32 hca2cd23_2 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
nspr 4.35 h27087fc_0 conda-forge
nss 3.89 he45b914_0 conda-forge
numpy 1.22.3 py310h4ef5377_2 conda-forge
openjpeg 2.5.0 hfec8fc6_2 conda-forge
openssl 3.1.0 hd590300_3 conda-forge
packaging 23.1 pyhd8ed1ab_0 conda-forge
pandas 1.4.2 py310h769672d_2 conda-forge
patsy 0.5.3 pyhd8ed1ab_0 conda-forge
pcre2 10.40 hc3806b6_0 conda-forge
pillow 9.5.0 py310h582fbeb_1 conda-forge
pip 23.1.2 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
ply 3.11 py_1 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
pulseaudio-client 16.1 h5195f5e_3 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyqt 5.15.7 py310hab646b1_3 conda-forge
pyqt5-sip 12.11.0 py310heca2aa9_3 conda-forge
python 3.10.2 hc74c709_4_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python_abi 3.10 3_cp310 conda-forge
pytz 2023.3 pyhd8ed1ab_0 conda-forge
qt-main 5.15.8 h01ceb2d_12 conda-forge
racon 1.5.0 h21ec9f0_2 bioconda
readline 8.2 h8228510_1 conda-forge
scipy 1.8.0 py310hea5193d_1 conda-forge
seaborn 0.12.2 hd8ed1ab_0 conda-forge
seaborn-base 0.12.2 pyhd8ed1ab_0 conda-forge
seqtk 1.4 h7132678_0 bioconda
setuptools 67.7.2 pyhd8ed1ab_0 conda-forge
sip 6.7.9 py310hc6cd4ac_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sqlite 3.42.0 h2c6b66d_0 conda-forge
statsmodels 0.14.0 py310h278f3c1_1 conda-forge
tk 8.6.12 h27826a3_0 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
tomli 2.0.1 pyhd8ed1ab_0 conda-forge
tornado 6.3.2 py310h2372a71_0 conda-forge
typing_extensions 4.6.2 pyha770c72_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
unicodedata2 15.0.0 py310h5764c6d_0 conda-forge
wheel 0.40.0 pyhd8ed1ab_0 conda-forge
xcb-util 0.4.0 hd590300_1 conda-forge
xcb-util-image 0.4.0 h8ee46fc_1 conda-forge
xcb-util-keysyms 0.4.0 h8ee46fc_1 conda-forge
xcb-util-renderutil 0.3.9 hd590300_1 conda-forge
xcb-util-wm 0.4.1 h8ee46fc_1 conda-forge
xkeyboard-config 2.38 h0b41bf4_0 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.8.4 h8ee46fc_1 conda-forge
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h0b41bf4_2 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h0b41bf4_1003 conda-forge
xorg-xf86vidmodeproto 2.3.1 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
zlib 1.2.13 h166bdaf_4 conda-forge
zstd 1.5.2 h3eb15da_6 conda-forge
Install was done via the following:
conda create -c conda-forge --name gapless python=3.10.2 pandas=1.4.2 numpy=1.22.3 scipy=1.8.0 seaborn matplotlib=3.5.2 pillow biopython
conda activate gapless
conda install -c bioconda gapless
conda install -c bioconda minimap2 racon seqtk
Do you have any recommendations? The failure seems to be at matplotlib but I have tried both the default latest as well as one that was recommended in someone else's issue (different error https://github.com/schmeing/gapless/issues/1).
Please let me know. This is a very important tool for us to get working. Thanks
M
Sorry I should have provide the error message in the gapless_scaffold.log:
`0:00:04.096412 Reading in original assembly 0:00:04.141333 Loading repeats 0:00:04.152210 Filtering mappings Traceback (most recent call last): File "/fslgroup/fslg_pws_module/compute/.conda-pws/envs/gapless/lib/python3.11/site-packages/pandas/core/indexes/base.py", line 3653, in get_loc return self._engine.get_loc(casted_key) File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item KeyError: 'y'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/fslgroup/fslg_pws_module/compute/.conda-pws/envs/gapless/bin/gapless.py", line 13327, in
Thanks
Jeff