snakemake-wrappers icon indicating copy to clipboard operation
snakemake-wrappers copied to clipboard

Published 20 hours ago • verified publisher icon snakemake

Error occurred during initialization of VM java/lang/NoClassDefFoundError: java/lang/Object

Open moldach opened this issue 5 years ago • 2 comments

Snakemake version

5.8.1

NOTE: I will go over why I'm not using the latest version in the additional context section

Wrapper version

0.64.0/bio/fastqc

Describe the bug

I'm getting the following error trying to use the fastqc wrapper:

Error occurred during initialization of VM

Logs

Building DAG of jobs...
Using shell: /cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/bin/bash
Provided cores: 1 (use --cores to define parallelism)
Rules claiming more threads will be scaled down.
Job counts:
        count   jobs
        1       qc_before_trim_r2
        1

[Wed Aug 26 09:34:45 2020]
Job 0: --- Quality check of raw data with FastQC before trimming.

python /scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/.snakemake/scripts/tmpb0_czsv2.wrapper.py
Activating conda environment: /scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/.snakemake/conda/91998b6c
fastqc  --quiet -t 1 --outdir /tmp/tmpjnd9fio7 /home/moldach/projects/def-mtarailo/common/WGS_6/MTG324/MTG324_R2.fastq.gz ' 2> logs/fastqc/MTG324_R2.log'
Error occurred during initialization of VM
java/lang/NoClassDefFoundError: java/lang/Object
Traceback (most recent call last):
  File "/scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/.snakemake/scripts/tmpb0_czsv2.wrapper.py", line 35, in <module>
    shell(
  File "/home/moldach/bin/snakemake/lib/python3.8/site-packages/snakemake/shell.py", line 156, in __new__
    raise sp.CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'set -euo pipefail;  fastqc  --quiet -t 1 --outdir /tmp/tmpjnd9fio7 /home/moldach/projects/def-mtarailo/common/WGS_6/MTG324/MTG324_R2.fastq.gz ' 2> logs$
[Wed Aug 26 09:35:09 2020]
Error in rule qc_before_trim_r2:
    jobid: 0
    output: qc/fastQC/before_trim/MTG324_R2_fastqc.html, qc/fastQC/before_trim/MTG324_R2_fastqc.zip
    log: logs/fastqc/MTG324_R2.log (check log file(s) for error message)
    conda-env: /scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/.snakemake/conda/91998b6c

RuleException:
CalledProcessError in line 140 of /scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/Snakefile:
Command 'source /home/moldach/miniconda3/bin/activate '/scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/.snakemake/conda/91998b6c'; set -euo pipefail;  python /scratch/moldach/MADDOG/V$
  File "/scratch/moldach/MADDOG/VCF-FILES/biostars439754/MTG353/Snakefile", line 140, in __rule_qc_before_trim_r2
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/python/3.8.0/lib/python3.8/concurrent/futures/thread.py", line 57, in run
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

Minimal example

The config.yaml

# Files
REF_GENOME: "c_elegans.PRJNA13758.WS265.genomic.fa"
GENOME_ANNOTATION: "c_elegans.PRJNA13758.WS265.annotations.gff3"

# Tools
QC_TOOL: "fastQC"
TRIM_TOOL: "trimmomatic"
ALIGN_TOOL: "bwa"
MARKDUP_TOOL: "picard"
CALLING_TOOL: "varscan"
ANNOT_TOOL: "vep"

The Snakefile

# Directories------------------------------------------------------------------
configfile: "config.yaml"

# Setting the names of all directories
dir_list = ["REF_DIR", "LOG_DIR", "BENCHMARK_DIR", "QC_DIR", "TRIM_DIR", "ALIGN_DIR", "MARKDUP_DIR", "CALLING_DIR", "ANNOT_DIR"]
dir_names = ["refs", "logs", "benchmarks", "qc", "trimming", "alignment", "mark_duplicates", "variant_calling", "annotation"]
dirs_dict = dict(zip(dir_list, dir_names))

import os
import pandas as pd
# getting the samples information (names, path to r1 & r2) from samples.txt
samples_information = pd.read_csv("samples.txt", sep='\t', index_col=False)
# get a list of the sample names
sample_names = list(samples_information['sample'])
sample_locations = list(samples_information['location'])
samples_dict = dict(zip(sample_names, sample_locations))
# get number of samples
len_samples = len(sample_names)


# Rules -----------------------------------------------------------------------

rule all:
    input:
        expand('{QC_DIR}/{QC_TOOL}/before_trim/{sample}_{pair}_fastqc.{ext}', QC_DIR=dirs_dict["QC_DIR"], QC_TOOL=config["QC_TOOL"], sample=sample_names, pair=['R1', 'R2'], ext=['html', 'zip'])
		
		
def getHome(sample):
  return(list(os.path.join(samples_dict[sample],"{0}_{1}.fastq.gz".format(sample,pair)) for pair in ['R1','R2']))

rule qc_before_trim_r1:
    input:
        r1 = lambda wildcards: getHome(wildcards.sample)[0]
    output:
        html=os.path.join(dirs_dict["QC_DIR"],config["QC_TOOL"],"before_trim","{sample}_R1_fastqc.html"),
        zip=os.path.join(dirs_dict["QC_DIR"],config["QC_TOOL"],"before_trim","{sample}_R1_fastqc.zip"),
    params: ""
    log:
        "logs/fastqc/{sample}_R1.log"
    resources:
        mem = 1000,
        time = 30
    threads: 1
    message: """--- Quality check of raw data with FastQC before trimming."""
    wrapper:
        "0.64.0/bio/fastqc"

rule qc_before_trim_r2:
    input:
        r1 = lambda wildcards: getHome(wildcards.sample)[1]
    output:
        html=os.path.join(dirs_dict["QC_DIR"],config["QC_TOOL"],"before_trim","{sample}_R2_fastqc.html"),
        zip=os.path.join(dirs_dict["QC_DIR"],config["QC_TOOL"],"before_trim","{sample}_R2_fastqc.zip"),
    params: ""
    log:
        "logs/fastqc/{sample}_R2.log"
    resources:
        mem = 1000,
        time = 30
    threads: 1
    message: """--- Quality check of raw data with FastQC before trimming."""
    wrapper:
        "0.64.0/bio/fastqc"

Additional context

It was mentioned in the preamble to this issue that I should I try the newest version of Snakemake. I downloaded the newest version via:

$ mamba create -c conda-forge -c bioconda -n snakemake snakemake
$ conda activate snakemake

But now when I try a dry-run I get a Segmentation fault

Using the older version of Snakemake (for comparison)

$ source ~/bin/snakemake/bin/activate
(snakemake) $ snakemake -n -r
Building DAG of jobs...
Job counts:
        count   jobs
        1       all
        1       qc_before_trim_r1
        1       qc_before_trim_r2
        3

[Wed Aug 26 10:28:41 2020]
Job 2: --- Quality check of raw data with FastQC before trimming.
Reason: Missing output files: qc/fastQC/before_trim/MTG324_R2_fastqc.html, qc/fastQC/before_trim/MTG324_R2_fastqc.zip

[Wed Aug 26 10:28:41 2020]
Job 1: --- Quality check of raw data with FastQC before trimming.
Reason: Missing output files: qc/fastQC/before_trim/MTG324_R1_fastqc.zip, qc/fastQC/before_trim/MTG324_R1_fastqc.html

[Wed Aug 26 10:28:41 2020]
localrule all:
    input: qc/fastQC/before_trim/MTG324_R1_fastqc.html, qc/fastQC/before_trim/MTG324_R1_fastqc.zip, qc/fastQC/before_trim/MTG324_R2_fastqc.html, qc/fastQC/before_trim/MTG324_R2_fastqc.zip
    jobid: 0
    reason: Input files updated by another job: qc/fastQC/before_trim/MTG324_R2_fastqc.html, qc/fastQC/before_trim/MTG324_R2_fastqc.zip, qc/fastQC/before_trim/MTG324_R1_fastqc.zip, qc/fastQC/before_trim/MTG324_R1_fastqc.html

Job counts:
        count   jobs
        1       all
        1       qc_before_trim_r1
        1       qc_before_trim_r2
        3
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

$ deactivate

Newest version of Snakemake

$ conda activate snakemake
$ conda --version
5.23.0
$ snakemake -n -r
Segmentation fault

I guess other pertinent information is that I'm on an academic HPC with a SLURM scheduler.

$ cat /etc/os-release

NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

The issue I see with me using the newest Snakemake version is the following:

If I were to run an interactive job (to get more memory) (via salloc --time=1:0:0 --mem=1000) and then try to submit a job (which is a full pipeline consisting of many wrappers) (via bash -c "nohup snakemake --profile slurm --use-conda --jobs 500 &") it would only run jobs as long as the interactive job was set for.

As I understand it, Snakemake needs to be run from the head node - it submit jobs to the SLURM scheduler.

Is it possible that Snakemake version 5.23.0 is more memory intensive than 5.8.1? And if so does this preclude me from using it?

moldach avatar Aug 26 '20 17:08 moldach

I've used the verbose flag -r to get a more comprehensive log:

Building DAG of jobs...
Creating conda environment https:/github.com/snakemake/snakemake-wrappers/raw/0.64.0/bio/fastqc/environment.yaml...
Downloading and installing remote packages.
CreateCondaEnvironmentException:
Could not create conda environment from /tmp/tmp1y8ge388.yaml:
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... done
Preparing transaction: ...working... done
Verifying transaction: ...working... failed

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in __call__
        return func(*args, **kwargs)
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda_env/cli/main.py", line 80, in do_call
        exit_code = getattr(module, func_name)(args, parser)
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda_env/cli/main_create.py", line 117, in execute
        result[installer_type] = installer.install(prefix, pkg_specs, args, env)
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda_env/installers/conda.py", line 40, in install
        unlink_link_transaction.execute()
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/core/link.py", line 244, in execute
        self.verify()
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
        return f(*args, **kwds)
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/core/link.py", line 231, in verify
        exceptions = self._verify(self.prefix_setups, self.prefix_action_groups)
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/core/link.py", line 590, in _verify
        for exc in self.verify_executor.map(UnlinkLinkTransaction._verify_individual_level,
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/common/io.py", line 525, in map
        yield func(thing)
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/core/link.py", line 411, in _verify_individual_level
        error_result = axn.verify()
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/core/path_actions.py", line 418, in verify
        create_link(self.source_full_path, self.intermediate_path, LinkType.copy)
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/gateways/disk/create.py", line 373, in create_link
        copy(src, dst)
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/gateways/disk/create.py", line 310, in copy
        _do_copy(src, dst)
      File "/home/moldach/miniconda3/lib/python3.8/site-packages/conda/gateways/disk/create.py", line 324, in _do_copy
        copyfileobj(fsrc, fdst, buffer_size)
    OSError: [Errno 5] Input/output error

`$ /home/moldach/miniconda3/bin/conda-env create --file /scratch/moldach/MADDOG/VCF-FILES/biostars439754/.snakemake/conda/f288ba8b.yaml --prefix /scratch/moldach/MADDOG/VCF-FILES/biostars439754/.snakemake/conda/f288ba8b`

  environment variables:
             ACLOCAL_PATH=/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/share/aclocal
                 CIO_TEST=<not set>
       CMAKE_LIBRARY_PATH=/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-5.4.0/lib64
        CMAKE_PREFIX_PATH=/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc-5.4.0
  CONDA_AUTO_UPDATE_CONDA=false
                CONDA_EXE=/home/moldach/miniconda3/bin/conda
         CONDA_PYTHON_EXE=/home/moldach/miniconda3/bin/python
               CONDA_ROOT=/home/moldach/miniconda3
              CONDA_SHLVL=0
                    CPATH=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/int
                          el2016.4/openmpi/2.1.1/include:/cvmfs/restricted.computecanada.ca/easy
                          build/software/2017/Core/ifort/2016.4.258/include:/cvmfs/restricted.co
                          mputecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/mkl/includ
                          e/fftw:/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core
                          /imkl/11.3.4.258/mkl/include
       CPLUS_INCLUDE_PATH=/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/include
           CURL_CA_BUNDLE=/etc/pki/tls/certs/ca-bundle.crt
           C_INCLUDE_PATH=/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/include
      EASYBUILD_BUILDPATH=/tmp/moldach
          LD_LIBRARY_PATH=/opt/software/slurm/lib
             LIBRARY_PATH=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/int
                          el2016.4/openmpi/2.1.1/lib:/cvmfs/soft.computecanada.ca/easybuild/soft
                          ware/2017/Core/ifort/2016.4.258/compilers_and_libraries_2016.4.258/lin
                          ux/compiler/lib/intel64:/cvmfs/restricted.computecanada.ca/easybuild/s
                          oftware/2017/Core/ifort/2016.4.258/compilers_and_libraries_2016.4.258/
                          linux/compiler/lib/intel64:/cvmfs/soft.computecanada.ca/easybuild/soft
                          ware/2017/Core/icc/2016.4.258/compilers_and_libraries_2016.4.258/linux
                          /compiler/lib/intel64:/cvmfs/restricted.computecanada.ca/easybuild/sof
                          tware/2017/Core/icc/2016.4.258/compilers_and_libraries_2016.4.258/linu
                          x/compiler/lib/intel64:/cvmfs/soft.computecanada.ca/nix/var/nix/profil
                          es/gcc-5.4.0/lib64:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/g
                          cc-5.4.0/lib:/cvmfs/soft.computecanada.ca/easybuild/software/2017/Core
                          /imkl/11.3.4.258/mkl/lib/intel64:/cvmfs/restricted.computecanada.ca/ea
                          sybuild/software/2017/Core/imkl/11.3.4.258/mkl/lib/intel64:/cvmfs/rest
                          ricted.computecanada.ca/easybuild/software/2017/Core/imkl/11.3.4.258/l
                          ib/intel64:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib
        LMOD_PACKAGE_PATH=/cvmfs/soft.computecanada.ca/config/lmod/
                  MANPATH=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/int
                          el2016.4/openmpi/2.1.1/share/man:/cvmfs/restricted.computecanada.ca/ea
                          sybuild/software/2017/Core/ifort/2016.4.258/compilers_and_libraries_20
                          16.4.258/linux/man/common:/cvmfs/restricted.computecanada.ca/easybuild
                          /software/2017/Core/icc/2016.4.258/compilers_and_libraries_2016.4.258/
                          linux/man/common:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/gcc
                          -5.4.0/share/man:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.
                          09/share/man:/cvmfs/soft.computecanada.ca/custom/software/lmod/lmod/sh
                          are/man::/opt/puppetlabs/puppet/share/man
      MIC_LD_LIBRARY_PATH=/cvmfs/restricted.computecanada.ca/easybuild/software/2017/Core/imkl/1
                          1.3.4.258/mkl/lib/mic:/cvmfs/restricted.computecanada.ca/easybuild/sof
                          tware/2017/Core/imkl/11.3.4.258/lib/intel64_lin_mic
               MODULEPATH=/opt/software/modulefiles:/cvmfs/soft.computecanada.ca/easybuild/modul
                          es/2017/avx2/MPI/intel2016.4/openmpi2.1:/cvmfs/soft.computecanada.ca/e
                          asybuild/modules/2017/avx2/Compiler/intel2016.4:/home/moldach/.local/e
                          asybuild/modules/2017/Core:/cvmfs/soft.computecanada.ca/easybuild/modu
                          les/2017/Core:/cvmfs/soft.computecanada.ca/custom/modules
                     PATH=/home/moldach/bin/snakemake/bin:/home/moldach/nb_distribution/:/home/m
                          oldach/miniconda3/condabin:/home/moldach/projects/def-mtgraovac/moldac
                          h/bin:/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compil
                          er/intel2016.4/openmpi/2.1.1/bin:/cvmfs/restricted.computecanada.ca/ea
                          sybuild/software/2017/Core/ifort/2016.4.258/compilers_and_libraries_20
                          16.4.258/linux/bin/intel64:/cvmfs/restricted.computecanada.ca/easybuil
                          d/software/2017/Core/icc/2016.4.258/compilers_and_libraries_2016.4.258
                          /linux/bin/intel64:/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/g
                          cc-5.4.0/bin:/cvmfs/restricted.computecanada.ca/easybuild/software/201
                          7/Core/imkl/11.3.4.258/mkl/bin:/cvmfs/restricted.computecanada.ca/easy
                          build/software/2017/Core/imkl/11.3.4.258/bin:/cvmfs/soft.computecanada
                          .ca/custom/bin/computecanada:/cvmfs/soft.computecanada.ca/easybuild/bi
                          n:/cvmfs/soft.computecanada.ca/custom/bin:/cvmfs/soft.computecanada.ca
                          /nix/var/nix/profiles/16.09/bin:/cvmfs/soft.computecanada.ca/nix/var/n
                          ix/profiles/16.09/sbin:/opt/software/bin:/opt/software/slurm/bin:/usr/
                          local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/opt/
                          dell/srvadmin/bin
          PKG_CONFIG_PATH=/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/int
                          el2016.4/openmpi/2.1.1/lib/pkgconfig:/cvmfs/soft.computecanada.ca/nix/
                          var/nix/profiles/16.09/lib/pkgconfig:/cvmfs/soft.computecanada.ca/nix/
                          var/nix/profiles/16.09/share/pkgconfig
               PYTHONPATH=/cvmfs/soft.computecanada.ca/custom/python/site-packages
       REQUESTS_CA_BUNDLE=<not set>
     RSNT_LD_LIBRARY_PATH=/opt/software/slurm/lib
            SSL_CERT_FILE=/etc/pki/tls/certs/ca-bundle.crt
__LMOD_Priority_MODULEPATH=/opt/software/modulefiles:-10
__LMOD_REF_COUNT_MODULEPATH=/opt/software/modulefiles:1;/cvmfs/soft.computecanada.ca/easybuild/mod
                          ules/2017/avx2/MPI/intel2016.4/openmpi2.1:1;/cvmfs/soft.computecanada.
                          ca/easybuild/modules/2017/avx2/Compiler/intel2016.4:1;/home/moldach/.l
                          ocal/easybuild/modules/2017/Core:1;/cvmfs/soft.computecanada.ca/easybu
                          ild/modules/2017/Core:1;/cvmfs/soft.computecanada.ca/custom/modules:1

     active environment : None
            shell level : 0
       user config file : /home/moldach/.condarc
 populated config files : /home/moldach/.condarc
          conda version : 4.8.4
    conda-build version : not installed
         python version : 3.8.3.final.0
       virtual packages : __glibc=2.17
       base environment : /home/moldach/miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/moldach/miniconda3/pkgs
                          /home/moldach/.conda/pkgs
       envs directories : /home/moldach/miniconda3/envs
                          /home/moldach/.conda/envs
               platform : linux-64
             user-agent : conda/4.8.4 requests/2.23.0 CPython/3.8.3 Linux/3.10.0-1062.12.1.el7.x86_64 centos/7.7.1908 glibc/2.17
                UID:GID : 3019658:3019658
             netrc file : None
           offline mode : False


An unexpected error has occurred. Conda has prepared the above report.

moldach avatar Aug 27 '20 18:08 moldach

Relevant issue: #191 Relevant stackoverflow question: https://stackoverflow.com/q/64000711/3998252

ManavalanG avatar Sep 22 '20 22:09 ManavalanG