tools
tools copied to clipboard
nf-core
Inconsistent number of containers with `pipelines download` subcommand
Description of the bug
Hi I have an error in my CI/CD, where the number of downloads containers is different from the ones defined in the pipeline:
https://github.com/bigbio/quantms/actions/runs/14731287117/job/41358046501.
I have tried to debug internally, the command, and looks like even when the pipeline detect more than one container it only downloads one. See
INFO Saving 'bigbio/quantms' download.py:252
Pipeline revision: 'dev'
Use containers: 'singularity'
Container library: 'quay.io, docker.io, community.wave.seqera.io/library/, ghcr.io'
Using $NXF_SINGULARITY_CACHEDIR': /hps/nobackup/juan/pride/reanalysis/temp/singularity'
Output directory: './quantms'
Include default institutional configuration: 'True'
INFO Downloading centralised configs from GitHub download.py:265
DEBUG Downloading https://github.com/nf-core/configs/archive/master.zip download.py:662
DEBUG Starting new HTTPS connection (1): github.com:443 connectionpool.py:1049
DEBUG https://github.com:443 "GET /nf-core/configs/archive/master.zip HTTP/1.1" 302 0 connectionpool.py:544
DEBUG Starting new HTTPS connection (1): codeload.github.com:443 connectionpool.py:1049
DEBUG https://codeload.github.com:443 "GET /nf-core/configs/zip/refs/heads/master HTTP/1.1" 200 None connectionpool.py:544
INFO Downloading workflow files from GitHub download.py:269
DEBUG Downloading https://github.com/bigbio/quantms/archive/15ae4a21e4ebd025ad977998e423847ba16c459b.zip download.py:631
DEBUG Starting new HTTPS connection (1): github.com:443 connectionpool.py:1049
DEBUG https://github.com:443 "GET /bigbio/quantms/archive/15ae4a21e4ebd025ad977998e423847ba16c459b.zip HTTP/1.1" 302 0 connectionpool.py:544
DEBUG Starting new HTTPS connection (1): codeload.github.com:443 connectionpool.py:1049
DEBUG https://codeload.github.com:443 "GET /bigbio/quantms/zip/15ae4a21e4ebd025ad977998e423847ba16c459b HTTP/1.1" 200 None connectionpool.py:544
DEBUG Editing 'params.custom_config_base' in './quantms/dev/nextflow.config' download.py:685
DEBUG Replacing 'https://raw.githubusercontent.com/nf-core/configs/${params.custom_config_version}' with '${projectDir}/../configs/' download.py:692
DEBUG Updating './quantms/dev/nextflow.config' download.py:704
DEBUG Fetching container names for workflow download.py:720
DEBUG Got 'quantms/dev' as path utils.py:280
DEBUG Found a config cache, loading: /homes/pst_prd/.nextflow/nf-core/wf-config-cache-e27b78d997f0d816ae3345d51.json utils.py:314
DEBUG https://depot.galaxyproject.org/singularity/quantms-utils:0.0.21--pyh7e72e81_0 matches and will be saved as quantms-utils:0.0.21--pyh7e72e81_0 download.py:1028
DEBUG https://containers.biocontainers.pro/s3/SingImgsRepo/diann/v1.8.1_cv1/diann_v1.8.1_cv1.img matches and will be saved as diann_v1.8.1_cv1.img download.py:1028
DEBUG docker.io/biocontainers/diann:v1.8.1_cv1 matches and will be saved as diann:v1.8.1_cv1 download.py:1028
DEBUG https://depot.galaxyproject.org/singularity/mulled-v2-796b0610595ad1995b121d0b85375902097b78d4:a3a3220eb9ee55710d743438b2ab9092867c98c6-0 matches and will be saved as download.py:1028
mulled-v2-796b0610595ad1995b121d0b85375902097b78d4:a3a3220eb9ee55710d743438b2ab9092867c98c6-0
DEBUG docker.io/biocontainers/diann:v1.8.1_cv1 matches and will be saved as diann:v1.8.1_cv1 download.py:1028
DEBUG docker.io/biocontainers/diann:v1.8.1_cv1 matches and will be saved as diann:v1.8.1_cv1 download.py:1028
DEBUG docker.io/biocontainers/diann:v1.8.1_cv1 matches and will be saved as diann:v1.8.1_cv1 download.py:1028
DEBUG https://depot.galaxyproject.org/singularity/quantms-rescoring:0.0.10--pyhdfd78af_0 matches and will be saved as quantms-rescoring:0.0.10--pyhdfd78af_0 download.py:1028
DEBUG https://depot.galaxyproject.org/singularity/bioconductor-msstats:4.10.0--r43hf17093f_0 matches and will be saved as bioconductor-msstats:4.10.0--r43hf17093f_0 download.py:1028
DEBUG https://depot.galaxyproject.org/singularity/bioconductor-msstatstmt:2.10.0--r43hdfd78af_0 matches and will be saved as bioconductor-msstatstmt:2.10.0--r43hdfd78af_0 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG https://depot.galaxyproject.org/singularity/msgf_plus:2024.03.26--hdfd78af_0 matches and will be saved as msgf_plus:2024.03.26--hdfd78af_0 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG ghcr.io/bigbio/openms-tools-thirdparty:2025.04.14 matches and will be saved as openms-tools-thirdparty:2025.04.14 download.py:1028
DEBUG https://depot.galaxyproject.org/singularity/pmultiqc:0.0.26--pyhdfd78af_0 matches and will be saved as pmultiqc:0.0.26--pyhdfd78af_0 download.py:1028
DEBUG docker.io/biocontainers/diann:v1.8.1_cv1 matches and will be saved as diann:v1.8.1_cv1 download.py:1028
DEBUG quay.io/bigbio/tdf2mzml:latest matches and will be saved as tdf2mzml:latest download.py:1028
DEBUG https://depot.galaxyproject.org/singularity/thermorawfileparser:1.3.4--ha8f3691_0 matches and will be saved as thermorawfileparser:1.3.4--ha8f3691_0 download.py:1028
DEBUG https://depot.galaxyproject.org/singularity/multiqc:1.19--pyhdfd78af_0 matches and will be saved as multiqc:1.19--pyhdfd78af_0 download.py:1028
DEBUG https://depot.galaxyproject.org/singularity/multiqc:1.27--pyhdfd78af_0 matches and will be saved as multiqc:1.27--pyhdfd78af_0 download.py:1028
INFO Processing workflow revision dev, found 1 container image in total. download.py:1152
Image file exists at destination ━━━━━━━━━━━━━━━━━━━━━━━━ 100% • 1/1 completed
Command used and terminal output
nf-core -v pipelines download bigbio/quantms --revision dev --outdir ./quantms --compress "none" --container-system 'singularity' --container-library "quay.io" -l "docker.io" -l "community.wave.seqera.io/library/" -l "ghcr.io" --container-cache-utilisation 'amend' --download-configuration 'yes' --force
System information
I am having trouble recreating this issue. Did you solve this with the template update in you PR or only by removing the CI workflow?
At the first glance, I do not sport anything out of the ordinary there. The containers are correctly identified by nf-core download as shown by the debug statements. But since they are already contained in the NXF_SINGULARITY_CACHEDIR, they are not downloaded anymore because of --container-cache-utilisation 'amend'.
This issue has been tagged as awaiting-changes or awaiting-feedback by an nf-core contributor. Remove stale label or add a comment otherwise this issue will be closed in 20 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue was closed because it has been tagged as awaiting-changes or awaiting-feedback by an nf-core contributor and then staled for 20 days with no activity.