demultiplex
demultiplex copied to clipboard
nf-core
Some singularity images are missing when using `nf-core download demultiplex`
Description of the bug
After downloading the singularity images with nf-core download demultiplex, nextflow still attempts to download the bclconvert singularity image (similar problems occur with bcl2fastq and bases2fastq).
I suspect this has to do with how the images are named, i.e. the singularity_images folder still contains a file named nf-core-bclconvert-4.2.4.img but nextflow insists on downloading quay.io-nf-core-bclconvert-4.2.4.img.
Another (smaller) issue is that multiQC is downloaded twice (v1.17 and v1.18).
Command used and terminal output
$ nf-core download demultiplex
,--./,-.
___ __ __ __ ___ /,-._.--~\
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/tools version 2.11.1 - https://nf-co.re
WARNING Could not find GitHub authentication token. Some API requests may fail.
? Select release / branch: 1.4.0 [release]
? Include the nf-core's default institutional configuration files into the download? Yes
In addition to the pipeline code, this tool can download software containers.
? Download software container images: singularity
Nextflow and nf-core can use an environment variable called $NXF_SINGULARITY_CACHEDIR that is a path to a directory where remote Singularity images are stored. This allows downloaded images to be cached in a
central location.
? Define $NXF_SINGULARITY_CACHEDIR for a shared Singularity image download folder? [y/n]: n
If transferring the downloaded files to another system, it can be convenient to have everything compressed in a single file.
This is not recommended when downloading Singularity images, as it can take a long time and saves very little space.
? Choose compression type: none
INFO Saving 'nf-core/demultiplex'
Pipeline revision: '1.4.0'
Use containers: 'singularity'
Container library: 'quay.io'
Output directory: 'nf-core-demultiplex_1.4.0'
Include default institutional configuration: 'True'
INFO Downloading centralised configs from GitHub
INFO Downloading workflow files from GitHub
INFO Processing workflow revision 1.4.0, found 10 container images in total.
Pulling singularity images ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% • 10/10 completed
$ cd nf-core-demultiplex_1.4.0
$ ls singularity-images
depot.galaxyproject.org-singularity-falco-1.2.1--h867801b_3.img
depot.galaxyproject.org-singularity-fastp-0.23.4--h5f740d0_0.img
depot.galaxyproject.org-singularity-fqtk-0.2.1--h9f5acd7_0.img
depot.galaxyproject.org-singularity-multiqc-1.17--pyhdfd78af_0.img
depot.galaxyproject.org-singularity-multiqc-1.18--pyhdfd78af_0.img
depot.galaxyproject.org-singularity-sgdemux-1.1.1--ha982bd6_0.img
depot.galaxyproject.org-singularity-ubuntu-20.04.img
nf-core-bases2fastq-1.1.0.img
nf-core-bcl2fastq-2.20.0.422.img
nf-core-bclconvert-4.2.4.img
$ NXF_ANSI_LOG=false NXF_SINGULARITY_CACHEDIR=singularity-images nextflow run 1_4_0/main.nf -profile singularity,test --outdir /tmp/test_demux
N E X T F L O W ~ version 23.04.3
Launching `1_4_0/main.nf` [special_neumann] DSL2 - revision: a8c81121bd
------------------------------------------------------
,--./,-.
___ __ __ __ ___ /,-._.--~'
|\ | |__ __ / ` / \ |__) |__ } {
| \| | \__, \__/ | \ |___ \`-._,-`-,
`._,._,'
nf-core/demultiplex v1.4.0
------------------------------------------------------
Core Nextflow options
runName : special_neumann
containerEngine : singularity
launchDir : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0
workDir : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/work
projectDir : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/1_4_0
userName : adrien
profile : singularity,test
configFiles :
Workflow options
skip_tools : []
Input/output options
input : https://raw.githubusercontent.com/nf-core/test-datasets/demultiplex/samplesheet/1.3.0/flowcell_input.csv
outdir : /tmp/test_demux
Demultiplexing options
demultiplexer : bclconvert
Institutional config options
custom_config_base : /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/1_4_0/../configs/
config_profile_name : Test profile
config_profile_description: Minimal test dataset to check pipeline function
Max job request options
max_cpus : 2
max_memory : 6.GB
max_time : 6.h
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use nf-core/demultiplex for your analysis please cite:
* The pipeline
https://doi.org/10.5281/zenodo.7153103
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
* Software dependencies
https://github.com/nf-core/demultiplex/blob/master/CITATIONS.md
Pulling Singularity image docker://quay.io/nf-core/bclconvert:4.2.4 [cache /tmp/sandbox/bug_repeat/nf-core-demultiplex_1.4.0/singularity-images/quay.io-nf-core-bclconvert-4.2.4.img]
[28/c0e23c] Submitted process > NFCORE_DEMULTIPLEX:DEMULTIPLEX:BCL_DEMULTIPLEX:BCLCONVERT (220422_M11111_0222_000000000-K9H97.1)
...
Relevant files
No response
System information
nf-core, version 2.11.1 nextflow, version 23.04.3 build 5875 nf-core/demultixplex, version 1.4.0
I'd say you're right on point with your issue, but I feel like the first part is more of a tools issue than demultiplex.
I'm guessing multiqc issues are due to custom/dumpsoftwareversions, and this is a demultiplex issue, but hopefully we're getting rid of this module soon
Right, it seems like tools is getting confused with this line https://github.com/nf-core/modules/blob/3fb69a931aa3c968ac87bc17961f30da4c9f17b5/modules/nf-core/bclconvert/main.nf#L5 (compare to the url for falco for instance: https://github.com/nf-core/modules/blob/3fb69a931aa3c968ac87bc17961f30da4c9f17b5/modules/nf-core/falco/main.nf#L7-L9)
Just created a follow-up issue in nf-core/tools, we can leave this one open for awareness.
The main issue should be fixed with the nf-core tools release 2.13.1 today! 👌🎉
(I would still appreciate seeing an update to Nextflow in the future, but for now symlinks ensure that the images are found nonetheless.)
Leaving this open as a reminder of the dissociative identity disorder of MultiQC.
I think we can close this. Modules subworkflows have been updated in the meanwhile. There's also no guarantee that all modules use the same version of the multiqc container (the multiqc container is sometimes used not for multiqc, but just as a python container that happens to have a couple of useful packages installed).
