VirSorter2
VirSorter2 copied to clipboard
Published
20 hours ago •
jiarong
jiarong
Error in rule classify_by_group
Hi jiarong, I have met this error "Error in rule classify_by_group" when I run the vs. After scanning the pre-answers, I tried all three options to install the vs2. But they all reported the same error, even including the Apptainer method, and the command was "virsorter2.sif run -w test.out -i test.fa --min-length 5000 -j 40 all".
The log file is as follows:
Hi, the error does not look like from the apptainer version. It shows a path "/home/dcy/". Is it your home directory? which should be NOT in the apptainer version.
Hi, the error does not look like from the apptainer version. It shows a path "/home/dcy/". Is it your home directory? which should be in the apptainer version.
Yes, '/home/dcy/' is my user's path. My apptainer version is 1.1.6.
I have met the same error.
[2023-03-29 23:18 ERROR] See error details in /beegfs/home/syl/software/test_forvir/test.out/log/iter-0/step3-classify/all-score-ssDNA.log
[Wed Mar 29 23:18:01 2023]
Error in rule classify_by_group:
jobid: 57
output: iter-0/ssDNA/all.pdg.clf
shell:
Log=/beegfs/home/syl/software/test_forvir/test.out/log/iter-0/step3-classify/all-score-ssDNA.log
python /beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/classify.py iter-0/ssDNA/all.pdg.ftr /beegfs/home/syl/database_db/vir/group/ssDNA/model ssDNA iter-0/ssDNA/all.pdg.clf 2> $Log || { echo "See error details in $Log" | python /beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/echo.py --level error; exit 1; }
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
[2023-03-29 23:18 ERROR] See error details in /beegfs/home/syl/software/test_forvir/test.out/log/iter-0/step3-classify/all-score-dsDNAphage.log
[Wed Mar 29 23:18:01 2023]
Error in rule classify_by_group:
jobid: 56
output: iter-0/dsDNAphage/all.pdg.clf
shell:
Log=/beegfs/home/syl/software/test_forvir/test.out/log/iter-0/step3-classify/all-score-dsDNAphage.log
python /beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/classify.py iter-0/dsDNAphage/all.pdg.ftr /beegfs/home/syl/database_db/vir/group/dsDNAphage/model dsDNAphage iter-0/dsDNAphage/all.pdg.clf 2> $Log || { echo "See error details in $Log" | python /beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/echo.py --level error; exit 1; }
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Exiting because a job execution failed. Look above for error message
*** An error occurred. Detailed errors may not be printed for certain rules. Refer to the log file of the failed command for troubleshooting
Issues can be raised at: https://github.com/jiarong/VirSorter2/issues
Or send an email to virsorter2 near gmail.com if you do not use GitHub
all-score-ssDNA.log as follows
/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator MinMaxScaler from version 0.22.1 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations
UserWarning,
/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator DecisionTreeClassifier from version 0.22.1 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations
UserWarning,
/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator RandomForestClassifier from version 0.22.1 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations
UserWarning,
/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator GridSearchCV from version 0.22.1 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations
UserWarning,
/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator Pipeline from version 0.22.1 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations
UserWarning,
/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/base.py:444: UserWarning: X has feature names, but MinMaxScaler was fitted without feature names
f"X has feature names, but {self.__class__.__name__} was fitted without"
Traceback (most recent call last):
File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/classify.py", line 77, in <module>
main()
File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/classify.py", line 68, in main
pred_prob = model.predict_proba(X)
File "/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/utils/metaestimators.py", line 113, in <lambda>
out = lambda *args, **kwargs: self.fn(obj, *args, **kwargs) # noqa
File "/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/pipeline.py", line 535, in predict_proba
Xt = transform.transform(Xt)
File "/beegfs/home/syl/anaconda3/envs/kofamscan/lib/python3.7/site-packages/sklearn/preprocessing/_data.py", line 506, in transform
if self.clip:
AttributeError: 'MinMaxScaler' object has no attribute 'clip'
Hi, the error does not look like from the apptainer version. It shows a path "/home/dcy/". Is it your home directory? which should be in the apptainer version.
Yes, '/home/dcy/' is my user's path. My apptainer version is 1.1.6.
@dangchenyuan Sorry I had typo in my last reply. I meant to say the error output does not look like from apptainer version, which should NOT have your user's path. Please double check your run with apptainer version.
@jiarong
I have changed the environment path of python, but I still got the same problem.
my commend line:
virsorter run -w test.out -i test.fa --min-length 1500 -j 4 all -d /beegfs/home/syl/database_db/vir --use-conda-off
the log:
/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator MinMaxScaler from version 0.22.1 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator DecisionTreeClassifier from version 0.22.1 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator RandomForestClassifier from version 0.22.1 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator GridSearchCV from version 0.22.1 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/base.py:318: UserWarning: Trying to unpickle estimator Pipeline from version 0.22.1 when using version 1.2.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
warnings.warn(
/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/base.py:432: UserWarning: X has feature names, but MinMaxScaler was fitted without feature names
warnings.warn(
Traceback (most recent call last):
File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/classify.py", line 77, in <module>
main()
File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/virsorter/./scripts/classify.py", line 68, in main
pred_prob = model.predict_proba(X)
File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/pipeline.py", line 546, in predict_proba
Xt = transform.transform(Xt)
File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/utils/_set_output.py", line 140, in wrapped
data_to_wrap = f(self, X, *args, **kwargs)
File "/beegfs/home/syl/anaconda3/envs/vs2/lib/python3.10/site-packages/sklearn/preprocessing/_data.py", line 518, in transform
if self.clip:
AttributeError: 'MinMaxScaler' object has no attribute 'clip'
@jiarong and this are the version of the packages,
(vs2) [syl@login test_forvir]$ conda list
# packages in environment at /beegfs/home/syl/anaconda3/envs/vs2:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
amply 0.1.5 pyhd8ed1ab_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
apptainer 1.1.7 h58603c9_0 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
arrow 1.2.3 pyhd8ed1ab_0 conda-forge
attrs 22.2.0 pyh71513ae_0 conda-forge
binaryornot 0.4.4 py_1 conda-forge
boltons 23.0.0 pyhd8ed1ab_0 conda-forge
brotlipy 0.7.0 py310h5764c6d_1005 conda-forge
bz2file 0.98 py_0 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
ca-certificates 2022.12.7 ha878542_0 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
certifi 2022.12.7 pyhd8ed1ab_0 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
cffi 1.15.1 py310h255011f_3 conda-forge
chardet 5.1.0 py310hff52083_0 conda-forge
charset-normalizer 2.1.1 pyhd8ed1ab_0 conda-forge
click 8.1.3 unix_pyhd8ed1ab_2 conda-forge
cni 1.0.1 ha975731_1 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
cni-plugins 1.0.1 ha8f183a_0 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
coin-or-cbc 2.10.8 h3786ebc_0 conda-forge
coin-or-cgl 0.60.6 h6f57e76_2 conda-forge
coin-or-clp 1.17.7 hc56784d_2 conda-forge
coin-or-osi 0.108.7 h2720bb7_2 conda-forge
coin-or-utils 2.11.6 h202d8b1_2 conda-forge
coincbc 2.10.8 0_metapackage conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
conda 23.3.0 py310hff52083_0 conda-forge
conda-package-handling 2.0.2 pyh38be061_0 conda-forge
conda-package-streaming 0.7.0 pyhd8ed1ab_1 conda-forge
configargparse 1.5.3 pyhd8ed1ab_0 conda-forge
cookiecutter 2.1.1 pyh6c4a22f_0 conda-forge
cryptography 40.0.1 py310h34c0648_0 conda-forge
curl 7.88.1 hdc1c0ab_1 conda-forge
datrie 0.8.2 py310h5764c6d_6 conda-forge
docutils 0.19 py310hff52083_1 conda-forge
expat 2.5.0 h27087fc_0 conda-forge
fmt 9.1.0 h924138e_0 conda-forge
gettext 0.21.1 h27087fc_0 conda-forge
git 2.40.0 pl5321h3ca6407_1 conda-forge
gitdb 4.0.10 pyhd8ed1ab_0 conda-forge
gitpython 3.1.31 pyhd8ed1ab_0 conda-forge
icu 72.1 hcb278e6_0 conda-forge
idna 3.4 pyhd8ed1ab_0 conda-forge
importlib-metadata 6.1.0 pyha770c72_0 conda-forge
importlib_resources 5.12.0 pyhd8ed1ab_0 conda-forge
jinja2 3.1.2 pyhd8ed1ab_1 conda-forge
jinja2-time 0.2.0 pyhd8ed1ab_3 conda-forge
joblib 1.2.0 pyhd8ed1ab_0 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
jq 1.5 0 bioconda
jsonschema 4.17.3 pyhd8ed1ab_0 conda-forge
jupyter_core 5.3.0 py310hff52083_0 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
krb5 1.20.1 h81ceb04_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
libarchive 3.6.2 h3d51595_0 conda-forge
libblas 3.9.0 16_linux64_openblas conda-forge
libcblas 3.9.0 16_linux64_openblas conda-forge
libcurl 7.88.1 hdc1c0ab_1 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc 7.2.0 h69d50b8_2 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
libgcc-ng 12.2.0 h65d4601_19 conda-forge
libgfortran-ng 12.2.0 h69a702a_19 conda-forge
libgfortran5 12.2.0 h337968e_19 conda-forge
libgomp 12.2.0 h65d4601_19 conda-forge
libiconv 1.17 h166bdaf_0 conda-forge
liblapack 3.9.0 16_linux64_openblas conda-forge
liblapacke 3.9.0 16_linux64_openblas conda-forge
libmamba 1.4.1 hcea66bb_0 conda-forge
libmambapy 1.4.1 py310h1428755_0 conda-forge
libnghttp2 1.52.0 h61bc06f_0 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libopenblas 0.3.21 pthreads_h78a6416_3 conda-forge
libseccomp 2.4.4 h166bdaf_3 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
libsolv 0.7.23 h3eb15da_0 conda-forge
libsqlite 3.40.0 h753d276_0 conda-forge
libssh2 1.10.0 hf14f497_3 conda-forge
libstdcxx-ng 12.2.0 h46fd767_19 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libxml2 2.10.3 hfdac1af_6 conda-forge
libzlib 1.2.13 h166bdaf_4 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
lzo 2.10 h516909a_1000 conda-forge
mamba 1.4.1 py310h51d5547_0 conda-forge
markupsafe 2.1.2 py310h1fa729e_0 conda-forge
nbformat 5.8.0 pyhd8ed1ab_0 conda-forge
ncurses 6.3 h27087fc_1 conda-forge
numpy 1.24.2 py310h8deb116_0 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
openssl 3.1.0 h0b41bf4_0 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
packaging 23.0 pyhd8ed1ab_0 conda-forge
pandas 1.5.3 py310h9b08913_1 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
pcre2 10.40 hc3806b6_0 conda-forge
perl 5.32.1 2_h7f98852_perl5 conda-forge
pip 23.0.1 pyhd8ed1ab_0 conda-forge
pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge
platformdirs 3.2.0 pyhd8ed1ab_0 conda-forge
pluggy 1.0.0 pyhd8ed1ab_5 conda-forge
pooch 1.7.0 pyha770c72_3 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
psutil 5.9.4 py310h5764c6d_0 conda-forge
pulp 2.7.0 py310hff52083_0 conda-forge
pybind11-abi 4 hd8ed1ab_3 conda-forge
pycosat 0.6.4 py310h5764c6d_1 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyopenssl 23.1.1 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.9 pyhd8ed1ab_0 conda-forge
pyrsistent 0.19.3 py310h1fa729e_0 conda-forge
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.10.10 he550d4f_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-fastjsonschema 2.16.3 pyhd8ed1ab_0 conda-forge
python-slugify 8.0.1 pyhd8ed1ab_0 conda-forge
python_abi 3.10 3_cp310 conda-forge
pytz 2023.3 pyhd8ed1ab_0 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
pyyaml 6.0 py310h5764c6d_5 conda-forge
ratelimiter 1.2.0 pyhd8ed1ab_1003 conda-forge
readline 8.2 h8228510_1 conda-forge
reproc 14.2.4 h0b41bf4_0 conda-forge
reproc-cpp 14.2.4 hcb278e6_0 conda-forge
requests 2.28.2 pyhd8ed1ab_0 conda-forge
ruamel.yaml 0.16.12 py310h5764c6d_3 conda-forge
ruamel.yaml.clib 0.2.7 py310h1fa729e_1 conda-forge
scikit-learn 1.2.2 py310h41b6a48_1 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
scipy 1.10.1 py310h8deb116_0 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
screed 1.0.4 py_0 bioconda
setuptools 67.6.1 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
smmap 3.0.5 pyh44b312d_0 conda-forge
snakemake-minimal 5.26.0 py_0 bioconda
squashfs-tools 4.4 hd0129a2_3 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
text-unidecode 1.3 py_0 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 https://mirrors.tuna.tsingh ua.edu.cn/anaconda/cloud/conda-forge
tk 8.6.12 h27826a3_0 conda-forge
toolz 0.12.0 pyhd8ed1ab_0 conda-forge
toposort 1.10 pyhd8ed1ab_0 conda-forge
tqdm 4.65.0 pyhd8ed1ab_1 conda-forge
traitlets 5.9.0 pyhd8ed1ab_0 conda-forge
typing-extensions 4.5.0 hd8ed1ab_0 conda-forge
typing_extensions 4.5.0 pyha770c72_0 conda-forge
tzdata 2023c h71feb2d_0 conda-forge
unidecode 1.3.6 pyhd8ed1ab_0 conda-forge
urllib3 1.26.15 pyhd8ed1ab_0 conda-forge
virsorter 2.2.4 pyhdfd78af_0 bioconda
wheel 0.40.0 pyhd8ed1ab_0 conda-forge
wrapt 1.15.0 py310h1fa729e_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
yaml-cpp 0.7.0 h27087fc_2 conda-forge
zipp 3.15.0 pyhd8ed1ab_0 conda-forge
zstandard 0.19.0 py310hdeb6495_1 conda-forge
zstd 1.5.2 h3eb15da_6 conda-forge
@TKsh6 If you reinstall your OS, conda version should work. If you do not want to reinstall OS, it's too complicated for me to find out the specific problem that is different for different users. That's why I recommend the apptainer/docker version.
it's not convenient to perform batch work through docker version, every time I need to ssh the specific node for running.
Apptainer is not available on all nodes? It's typically installed to be available server wide. You can ask you server admin to help.
Yes, you are right. I have solved this both in conda. Only need to install scikit-learn=0.22.1. Thanks a lot.
Hi jiarong, I have solved it now. Because of the conflict with different sklearn versions.