Issue with fastqc.smk

[rjg2186]

Hi @Hoohm

I am using the dropSeqPipe v0.4 and have installed snakemake using conda on custom directory. When I execute the pipeline, I am getting below error with fastqc.smk. Can you please let me know how to fix this. Thanks

Snakemake installation
conda create -c conda-forge -c bioconda -p ./conda_env snakemake python=3.6.8 
conda config --set env_prompt '({name})' 
conda activate ./conda_env 
snakemake --cores 10 -s /dir/dropSeqPipe-0.4/Snakefile --use-conda --directory test/

RuleException:
AttributeError in line 47 of /dir/dropSeqPipe-0.4/rules/fastqc.smk:
'NoneType' object has no attribute 'group'
  File "/working_dir/conda_env/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 2189, in run_wrapper
  File "/dir/dropSeqPipe-0.4/rules/fastqc.smk", line 47, in __rule_multiqc_fastqc_reads
  File "/working_dir/conda_env/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 529, in _callback
  File "/working_dir/conda_env/lib/python3.6/concurrent/futures/thread.py", line 56, in run
  File "/working_dir/conda_env/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 515, in cached_or_run
  File "/working_dir/conda_env/lib/python3.6/site-packages/snakemake/executors/__init__.py", line 2201, in run_wrapper
[Thu Jul  2 12:16:36 2020]
Finished job 21.
7 of 48 steps (15%) done
[Thu Jul  2 12:35:19 2020]
Finished job 23.
8 of 48 steps (17%) done
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: /test/.snakemake/log/2020-07-02T115209.118464.snakemake.log

[vuzun]

Hi @rjg2186 ,

Have you solved this issue? I'm having the same error with Nadia data for Arabidopsis.

Thanks!

[rjg2186]

Hi @vuzun

No, I even tried with the latest code from github. Still facing the same issue.

Thanks.

[Hoohm]

I'll take a look into it this week. I'll keep you posted :)

[Hoohm]

Are your R1 and R2 files having the exact same sample name?

It should be like sample1_R1.fastq.gz, sample1_R2.fastq.gz

the prefix has to be exactly the same, in this example: sample1_

[rjg2186]

Hi @Hoohm

Yes, the prefix is identical for the paired end read files. Three months back, I have used the same version of the dropSeqPipe and it worked fine. I am not sure the reason for getting an error now.

Is there anything to do with the version of Python ?

Thanks

[seb-mueller]

This is odd indeed. Maybe conda has introduced some sort of incompatibility. The errors seems to to come from fastqc.smk i.e. this line pointing at the multiqc wrapper.

Since qc is not essential to run the rest, could you try to leave it out to see if it only affects this rule? i.e. something like the below (I haven't tested this, but it's about specifiying the rules map filter merge directly sparing out qc):

snakemake --notemp --skip-script-cleanup --keep-going --use-conda -rp --cores 10 -s /dir/dropSeqPipe-0.4/Snakefile --use-conda --directory test/ map filter merge

[Hoohm]

Could you run the pipeline with the -p flag? It will print out the command it tries to execute and we can have a closer look at what's not working.

[rjg2186]

Hi @Hoohm

It looks like MutiQC is using an older version of python which conflicts with snakemake current requirement of python3>.

Below is the error and the command causing the error

Activating conda environment: /dir/.snakemake/conda/ab69dd99
/dir/.snakemake/conda/ab69dd99/lib/python2.7/site-packages/multiqc-1.2-py2.7.egg/multiqc/utils/config.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  configs = yaml.load(f)
/dir/.snakemake/conda/ab69dd99/lib/python2.7/site-packages/multiqc-1.2-py2.7.egg/multiqc/utils/config.py:51: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  sp = yaml.load(f)
[WARNING]         multiqc : MultiQC Version v1.9 now available!
[INFO   ]         multiqc : This is MultiQC v1.2
[INFO   ]         multiqc : Template    : default
[INFO   ]         multiqc : Searching 'results/logs/fastqc'
[INFO   ]         multiqc : Only using modules fastqc
[ERROR  ]         multiqc : Oops! The 'fastqc' MultiQC module broke... 
                    Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues 
                    (if possible, include a log file that triggers the error) 
============================================================
Module fastqc raised an exception: Traceback (most recent call last):
  File "/dir/.snakemake/conda/ab69dd99/lib/python2.7/site-packages/multiqc-1.2-py2.7.egg/EGG-INFO/scripts/multiqc", line 409, in multiqc
    mod = config.avail_modules[this_module].load()
  File "/dir/.snakemake/conda/ab69dd99/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2443, in load
    return self.resolve()
  File "/dir/.snakemake/conda/ab69dd99/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2449, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/dir/.snakemake/conda/ab69dd99/lib/python2.7/site-packages/multiqc-1.2-py2.7.egg/multiqc/modules/fastqc/__init__.py", line 3, in <module>
    from .fastqc import MultiqcModule
  File "/dir/.snakemake/conda/ab69dd99/lib/python2.7/site-packages/multiqc-1.2-py2.7.egg/multiqc/modules/fastqc/fastqc.py", line 24, in <module>
    from multiqc.modules.base_module import BaseMultiqcModule
  File "/dir/.snakemake/conda/ab69dd99/lib/python2.7/site-packages/multiqc-1.2-py2.7.egg/multiqc/modules/base_module.py", line 10, in <module>
    import markdown
  File "/dir/.snakemake/conda/ab69dd99/lib/python2.7/site-packages/markdown/__init__.py", line 27, in <module>
    raise ImportError('A recent version of Python 3 is required.')
ImportError: A recent version of Python 3 is required.
============================================================

subprocess.CalledProcessError: Command ' set -euo pipefail;  multiqc -m fastqc --ignore *_R1* --force -o results/reports -n fastqc_reads.html results/logs/fastqc ' returned non-zero exit status 1.

[rjg2186]

Hi @Hoohm

I just changed the wrapper to point to master/bio/multiqc and it's working fine now.

Thanks

[Hoohm]

Yea I was suspecting this would fix it. I was hoping to change multiqc version for a long time but I always had some conflicts coming up and that's why we kept the older version.

Gonna test it on our side and update asap

On Fri, 17 Jul 2020 at 14:43, rjg2186 [email protected] wrote:

Hi @Hoohm https://github.com/Hoohm

I just changed the wrapper to point to master/bio/multiqc and it's working fine now.

Thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Hoohm/dropSeqPipe/issues/108#issuecomment-660086046, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJVO2D3DQAA7HWUBV7SRUDR4BBP5ANCNFSM4OPDMXXA .

--

Roelli Patrick Division of Animal Physiology and Immunology TUM School of Life Sciences Weihenstephan Technische Universität München Weihenstephaner Berg 3 85354 Freising Germany

https://github.com/Hoohm https://github.com/Hoohm