snakemake-wrappers
snakemake-wrappers copied to clipboard
Published
20 hours ago •
snakemake
snakemake
Error occurred during initialization of VM java/lang/NoClassDefFoundError: java/lang/Object
Snakemake version
5.8.1
NOTE: I will go over why I'm not using the latest version in the additional context section
Wrapper version
0.64.0/bio/fastqc
Describe the bug
I'm getting the following error trying to use the fastqc wrapper:
Error occurred during initialization of VM
Logs
Building DAG of jobs...
Using shell: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job counts:
count jobs
1 qc_before_trim_r2
1
[Wed Aug 26 09:34:45 2020]
Job 0: --- Quality check of raw data with FastQC before trimming.
python /scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/.snakemake/scripts/tmpb0_czsv2.wrapper.py
Activating conda environment: /scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/.snakemake/conda/91998b6c
fastqc --quiet -t 1 --outdir /tmp/tmpjnd9fio7 /home/moldach/projects/def-mtarailo/common/WGS_6/MTG324/MTG324_R2.fastq.gz ' 2> logs/fastqc/MTG324_R2.log'
Error occurred during initialization of VM
java/lang/NoClassDefFoundError: java/lang/Object
Traceback (most recent call last):
File "/scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/.snakemake/scripts/tmpb0_czsv2.wrapper.py", line 35, in <module>
shell(
File "/home/moldach/bin/snakemake/lib/python3.8/site-packages/snakemake/shell.py", line 156, in __new__
raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'set -euo pipefail; fastqc --quiet -t 1 --outdir /tmp/tmpjnd9fio7 /home/moldach/projects/def-mtarailo/common/WGS_6/MTG324/MTG324_R2.fastq.gz ' 2> logs$
[Wed Aug 26 09:35:09 2020]
Error in rule qc_before_trim_r2:
jobid: 0
output: qc/fastQC/before_trim/MTG324_R2_fastqc.html, qc/fastQC/before_trim/MTG324_R2_fastqc.zip
log: logs/fastqc/MTG324_R2.log (check log file(s) for error message)
conda-env: /scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/.snakemake/conda/91998b6c
RuleException:
CalledProcessError in line 140 of /scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/Snakefile:
Command 'source /home/moldach/miniconda3/bin/activate '/scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/.snakemake/conda/91998b6c'; set -euo pipefail; python /scratch/moldach/MADDOG/V$
File "/scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/Snakefile", line 140, in __rule_qc_before_trim_r2
File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.8.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Minimal example
The config.yaml
# Files
REF_GENOME: "c_elegans.PRJNA13758.WS265.genomic.fa"
GENOME_ANNOTATION: "c_elegans.PRJNA13758.WS265.annotations.gff3"
# Tools
QC_TOOL: "fastQC"
TRIM_TOOL: "trimmomatic"
ALIGN_TOOL: "bwa"
MARKDUP_TOOL: "picard"
CALLING_TOOL: "varscan"
ANNOT_TOOL: "vep"
The Snakefile
# Directories------------------------------------------------------------------
configfile: "config.yaml"
# Setting the names of all directories
dir_list = ["REF_DIR", "LOG_DIR", "BENCHMARK_DIR", "QC_DIR", "TRIM_DIR", "ALIGN_DIR", "MARKDUP_DIR", "CALLING_DIR", "ANNOT_DIR"]
dir_names = ["refs", "logs", "benchmarks", "qc", "trimming", "alignment", "mark_duplicates", "variant_calling", "annotation"]
dirs_dict = dict(zip(dir_list, dir_names))
import os
import pandas as pd
# getting the samples information (names, path to r1 & r2) from samples.txt
samples_information = pd.read_csv("samples.txt", sep='\t', index_col=False)
# get a list of the sample names
sample_names = list(samples_information['sample'])
sample_locations = list(samples_information['location'])
samples_dict = dict(zip(sample_names, sample_locations))
# get number of samples
len_samples = len(sample_names)
# Rules -----------------------------------------------------------------------
rule all:
input:
expand('{QC_DIR}/{QC_TOOL}/before_trim/{sample}_{pair}_fastqc.{ext}', QC_DIR=dirs_dict["QC_DIR"], QC_TOOL=config["QC_TOOL"], sample=sample_names, pair=['R1', 'R2'], ext=['html', 'zip'])
def getHome(sample):
return(list(os.path.join(samples_dict[sample],"{0}_{1}.fastq.gz".format(sample,pair)) for pair in ['R1','R2']))
rule qc_before_trim_r1:
input:
r1 = lambda wildcards: getHome(wildcards.sample)[0]
output:
html=os.path.join(dirs_dict["QC_DIR"],config["QC_TOOL"],"before_trim","{sample}_R1_fastqc.html"),
zip=os.path.join(dirs_dict["QC_DIR"],config["QC_TOOL"],"before_trim","{sample}_R1_fastqc.zip"),
params: ""
log:
"logs/fastqc/{sample}_R1.log"
resources:
mem = 1000,
time = 30
threads: 1
message: """--- Quality check of raw data with FastQC before trimming."""
wrapper:
"0.64.0/bio/fastqc"
rule qc_before_trim_r2:
input:
r1 = lambda wildcards: getHome(wildcards.sample)[1]
output:
html=os.path.join(dirs_dict["QC_DIR"],config["QC_TOOL"],"before_trim","{sample}_R2_fastqc.html"),
zip=os.path.join(dirs_dict["QC_DIR"],config["QC_TOOL"],"before_trim","{sample}_R2_fastqc.zip"),
params: ""
log:
"logs/fastqc/{sample}_R2.log"
resources:
mem = 1000,
time = 30
threads: 1
message: """--- Quality check of raw data with FastQC before trimming."""
wrapper:
"0.64.0/bio/fastqc"
Additional context
It was mentioned in the preamble to this issue that I should I try the newest version of Snakemake. I downloaded the newest version via:
$ mamba create -c conda-forge -c bioconda -n snakemake snakemake
$ conda activate snakemake
But now when I try a dry-run I get a Segmentation fault
Using the older version of Snakemake (for comparison)
$ source ~/bin/snakemake/bin/activate
(snakemake) $ snakemake -n -r
Building DAG of jobs...
Job counts:
count jobs
1 all
1 qc_before_trim_r1
1 qc_before_trim_r2
3
[Wed Aug 26 10:28:41 2020]
Job 2: --- Quality check of raw data with FastQC before trimming.
Reason: Missing output files: qc/fastQC/before_trim/MTG324_R2_fastqc.html, qc/fastQC/before_trim/MTG324_R2_fastqc.zip
[Wed Aug 26 10:28:41 2020]
Job 1: --- Quality check of raw data with FastQC before trimming.
Reason: Missing output files: qc/fastQC/before_trim/MTG324_R1_fastqc.zip, qc/fastQC/before_trim/MTG324_R1_fastqc.html
[Wed Aug 26 10:28:41 2020]
localrule all:
input: qc/fastQC/before_trim/MTG324_R1_fastqc.html, qc/fastQC/before_trim/MTG324_R1_fastqc.zip, qc/fastQC/before_trim/MTG324_R2_fastqc.html, qc/fastQC/before_trim/MTG324_R2_fastqc.zip
jobid: 0
reason: Input files updated by another job: qc/fastQC/before_trim/MTG324_R2_fastqc.html, qc/fastQC/before_trim/MTG324_R2_fastqc.zip, qc/fastQC/before_trim/MTG324_R1_fastqc.zip, qc/fastQC/before_trim/MTG324_R1_fastqc.html
Job counts:
count jobs
1 all
1 qc_before_trim_r1
1 qc_before_trim_r2
3
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
$ deactivate
Newest version of Snakemake
$ conda activate snakemake
$ conda --version
5.23.0
$ snakemake -n -r
Segmentation fault
I guess other pertinent information is that I'm on an academic HPC with a SLURM scheduler.
$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
The issue I see with me using the newest Snakemake version is the following:
If I were to run an interactive job (to get more memory) (via salloc --time=1:0:0 --mem=1000) and then try to submit a job (which is a full pipeline consisting of many wrappers) (via bash -c "nohup snakemake --profile slurm --use-conda --jobs 500 &") it would only run jobs as long as the interactive job was set for.
As I understand it, Snakemake needs to be run from the head node - it submit jobs to the SLURM scheduler.
Is it possible that Snakemake version 5.23.0 is more memory intensive than 5.8.1? And if so does this preclude me from using it?
I've used the verbose flag -r to get a more comprehensive log:
Building DAG of jobs...
Creating conda environment https:/github.com/snakemake/snakemake-wrappers/raw/0.64.0/bio/fastqc/environment.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /tmp/tmp1y8ge388.yaml:
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... failed
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in __call__
return func(*args, **kwargs)
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda_env/cli/main.py", line 80, in do_call
exit_code = getattr(module, func_name)(args, parser)
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda_env/cli/main_create.py", line 117, in execute
result[installer_type] = installer.install(prefix, pkg_specs, args, env)
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda_env/installers/conda.py", line 40, in install
unlink_link_transaction.execute()
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/core/link.py", line 244, in execute
self.verify()
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
return f(*args, **kwds)
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/core/link.py", line 231, in verify
exceptions = self._verify(self.prefix_setups, self.prefix_action_groups)
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/core/link.py", line 590, in _verify
for exc in self.verify_executor.map(UnlinkLinkTransaction._verify_individual_level,
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/common/io.py", line 525, in map
yield func(thing)
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/core/link.py", line 411, in _verify_individual_level
error_result = axn.verify()
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/core/path_actions.py", line 418, in verify
create_link(self.source_full_path, self.intermediate_path, LinkType.copy)
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/gateways/disk/create.py", line 373, in create_link
copy(src, dst)
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/gateways/disk/create.py", line 310, in copy
_do_copy(src, dst)
File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/gateways/disk/create.py", line 324, in _do_copy
copyfileobj(fsrc, fdst, buffer_size)
OSError: [Errno 5] Input/output error
`$ /home/moldach/miniconda3/bin/conda-env create --file /scratch/moldach/MADDOG/VCF-FILES/biostars439754/.snakemake/conda/f288ba8b.yaml --prefix /scratch/moldach/MADDOG/VCF-FILES/biostars439754/.snakemake/conda/f288ba8b`
environment variables:
ACLOCAL_PATH=/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/share/aclocal
CIO_TEST=<not set>
CMAKE_LIBRARY_PATH=/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-5.4.0/lib64
CMAKE_PREFIX_PATH=/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-5.4.0
CONDA_AUTO_UPDATE_CONDA=false
CONDA_EXE=/home/moldach/miniconda3/bin/conda
CONDA_PYTHON_EXE=/home/moldach/miniconda3/bin/python
CONDA_ROOT=/home/moldach/miniconda3
CONDA_SHLVL=0
CPATH=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/int
el2016.4/openmpi/2.1.1/include:/cvmfs/restricted.computecanada.ca/easy
build/software/2017/Core/ifort/2016.4.258/include:/cvmfs/restricted.co
mputecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/includ
e/fftw:/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core
/imkl/11.3.4.258/mkl/include
CPLUS_INCLUDE_PATH=/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/include
CURL_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
C_INCLUDE_PATH=/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/include
EASYBUILD_BUILDPATH=/tmp/moldach
LD_LIBRARY_PATH=/opt/software/slurm/lib
LIBRARY_PATH=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/int
el2016.4/openmpi/2.1.1/lib:/cvmfs/soft.computecanada.ca/easybuild/soft
ware/2017/Core/ifort/2016.4.258/compilers_and_libraries_2016.4.258/lin
ux/compiler/lib/intel64:/cvmfs/restricted.computecanada.ca/easybuild/s
oftware/2017/Core/ifort/2016.4.258/compilers_and_libraries_2016.4.258/
linux/compiler/lib/intel64:/cvmfs/soft.computecanada.ca/easybuild/soft
ware/2017/Core/icc/2016.4.258/compilers_and_libraries_2016.4.258/linux
/compiler/lib/intel64:/cvmfs/restricted.computecanada.ca/easybuild/sof
tware/2017/Core/icc/2016.4.258/compilers_and_libraries_2016.4.258/linu
x/compiler/lib/intel64:/cvmfs/soft.computecanada.ca/nix/var/nix/profil
es/gcc-5.4.0/lib64:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/g
cc-5.4.0/lib:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core
/imkl/11.3.4.258/mkl/lib/intel64:/cvmfs/restricted.computecanada.ca/ea
sybuild/software/2017/Core/imkl/11.3.4.258/mkl/lib/intel64:/cvmfs/rest
ricted.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/l
ib/intel64:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib
LMOD_PACKAGE_PATH=/cvmfs/soft.computecanada.ca/config/lmod/
MANPATH=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/int
el2016.4/openmpi/2.1.1/share/man:/cvmfs/restricted.computecanada.ca/ea
sybuild/software/2017/Core/ifort/2016.4.258/compilers_and_libraries_20
16.4.258/linux/man/common:/cvmfs/restricted.computecanada.ca/easybuild
/software/2017/Core/icc/2016.4.258/compilers_and_libraries_2016.4.258/
linux/man/common:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc
-5.4.0/share/man:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.
09/share/man:/cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/sh
are/man::/opt/puppetlabs/puppet/share/man
MIC_LD_LIBRARY_PATH=/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/imkl/1
1.3.4.258/mkl/lib/mic:/cvmfs/restricted.computecanada.ca/easybuild/sof
tware/2017/Core/imkl/11.3.4.258/lib/intel64_lin_mic
MODULEPATH=/opt/software/modulefiles:/cvmfs/soft.computecanada.ca/easybuild/modul
es/2017/avx2/MPI/intel2016.4/openmpi2.1:/cvmfs/soft.computecanada.ca/e
asybuild/modules/2017/avx2/Compiler/intel2016.4:/home/moldach/.local/e
asybuild/modules/2017/Core:/cvmfs/soft.computecanada.ca/easybuild/modu
les/2017/Core:/cvmfs/soft.computecanada.ca/custom/modules
PATH=/home/moldach/bin/snakemake/bin:/home/moldach/nb_distribution/:/home/m
oldach/miniconda3/condabin:/home/moldach/projects/def-mtgraovac/moldac
h/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compil
er/intel2016.4/openmpi/2.1.1/bin:/cvmfs/restricted.computecanada.ca/ea
sybuild/software/2017/Core/ifort/2016.4.258/compilers_and_libraries_20
16.4.258/linux/bin/intel64:/cvmfs/restricted.computecanada.ca/easybuil
d/software/2017/Core/icc/2016.4.258/compilers_and_libraries_2016.4.258
/linux/bin/intel64:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/g
cc-5.4.0/bin:/cvmfs/restricted.computecanada.ca/easybuild/software/201
7/Core/imkl/11.3.4.258/mkl/bin:/cvmfs/restricted.computecanada.ca/easy
build/software/2017/Core/imkl/11.3.4.258/bin:/cvmfs/soft.computecanada
.ca/custom/bin/computecanada:/cvmfs/soft.computecanada.ca/easybuild/bi
n:/cvmfs/soft.computecanada.ca/custom/bin:/cvmfs/soft.computecanada.ca
/nix/var/nix/profiles/16.09/bin:/cvmfs/soft.computecanada.ca/nix/var/n
ix/profiles/16.09/sbin:/opt/software/bin:/opt/software/slurm/bin:/usr/
local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/opt/
dell/srvadmin/bin
PKG_CONFIG_PATH=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/int
el2016.4/openmpi/2.1.1/lib/pkgconfig:/cvmfs/soft.computecanada.ca/nix/
var/nix/profiles/16.09/lib/pkgconfig:/cvmfs/soft.computecanada.ca/nix/
var/nix/profiles/16.09/share/pkgconfig
PYTHONPATH=/cvmfs/soft.computecanada.ca/custom/python/site-packages
REQUESTS_CA_BUNDLE=<not set>
RSNT_LD_LIBRARY_PATH=/opt/software/slurm/lib
SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt
__LMOD_Priority_MODULEPATH=/opt/software/modulefiles:-10
__LMOD_REF_COUNT_MODULEPATH=/opt/software/modulefiles:1;/cvmfs/soft.computecanada.ca/easybuild/mod
ules/2017/avx2/MPI/intel2016.4/openmpi2.1:1;/cvmfs/soft.computecanada.
ca/easybuild/modules/2017/avx2/Compiler/intel2016.4:1;/home/moldach/.l
ocal/easybuild/modules/2017/Core:1;/cvmfs/soft.computecanada.ca/easybu
ild/modules/2017/Core:1;/cvmfs/soft.computecanada.ca/custom/modules:1
active environment : None
shell level : 0
user config file : /home/moldach/.condarc
populated config files : /home/moldach/.condarc
conda version : 4.8.4
conda-build version : not installed
python version : 3.8.3.final.0
virtual packages : __glibc=2.17
base environment : /home/moldach/miniconda3 (writable)
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/moldach/miniconda3/pkgs
/home/moldach/.conda/pkgs
envs directories : /home/moldach/miniconda3/envs
/home/moldach/.conda/envs
platform : linux-64
user-agent : conda/4.8.4 requests/2.23.0 CPython/3.8.3 Linux/3.10.0-1062.12.1.el7.x86_64 centos/7.7.1908 glibc/2.17
UID:GID : 3019658:3019658
netrc file : None
offline mode : False
An unexpected error has occurred. Conda has prepared the above report.
Relevant issue: #191 Relevant stackoverflow question: https://stackoverflow.com/q/64000711/3998252
