rnaseq
rnaseq copied to clipboard
Use channel topic for tool versions
This PR refactors the pipeline to use an experimental topic channel (https://github.com/nextflow-io/nextflow/pull/4459) to collect the tool versions.
~~The topic channel works, but multiqc is failing with this error:~~
nf-core lint
overall result: Failed :x:
Posted for pipeline commit 8f7e5bd
+| ✅ 144 tests passed |+
#| ❔ 6 tests were ignored |#
!| ❗ 4 tests had warnings |!
-| ❌ 1 tests failed |-
:x: Test failures:
- actions_ci - Minimum pipeline NF version '23.04.0' is not tested in '.github/workflows/ci.yml'
:heavy_exclamation_mark: Test warnings:
-
files_exist - File not found:
.github/workflows/awstest.yml
-
files_exist - File not found:
.github/workflows/awsfulltest.yml
-
pipeline_todos - TODO string in
methods_description_template.yml
: #Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline -
pipeline_todos - TODO string in
WorkflowRnaseq.groovy
: Optionally add in-text citation tools to this list.
:grey_question: Tests ignored:
-
files_unchanged - File ignored due to lint config:
assets/email_template.html
-
files_unchanged - File ignored due to lint config:
assets/email_template.txt
-
files_unchanged - File ignored due to lint config:
lib/NfcoreTemplate.groovy
-
files_unchanged - File ignored due to lint config:
.gitignore
or.prettierignore
orpyproject.toml
-
actions_awstest - 'awstest.yml' workflow not found:
/home/runner/work/rnaseq/rnaseq/.github/workflows/awstest.yml
- multiqc_config - multiqc_config
:white_check_mark: Tests passed:
-
files_exist - File found:
.gitattributes
-
files_exist - File found:
.gitignore
-
files_exist - File found:
.nf-core.yml
-
files_exist - File found:
.editorconfig
-
files_exist - File found:
.prettierignore
-
files_exist - File found:
.prettierrc.yml
-
files_exist - File found:
CHANGELOG.md
-
files_exist - File found:
CITATIONS.md
-
files_exist - File found:
CODE_OF_CONDUCT.md
-
files_exist - File found:
CODE_OF_CONDUCT.md
-
files_exist - File found:
LICENSE
orLICENSE.md
orLICENCE
orLICENCE.md
-
files_exist - File found:
nextflow_schema.json
-
files_exist - File found:
nextflow.config
-
files_exist - File found:
README.md
-
files_exist - File found:
.github/.dockstore.yml
-
files_exist - File found:
.github/CONTRIBUTING.md
-
files_exist - File found:
.github/ISSUE_TEMPLATE/bug_report.yml
-
files_exist - File found:
.github/ISSUE_TEMPLATE/config.yml
-
files_exist - File found:
.github/ISSUE_TEMPLATE/feature_request.yml
-
files_exist - File found:
.github/PULL_REQUEST_TEMPLATE.md
-
files_exist - File found:
.github/workflows/branch.yml
-
files_exist - File found:
.github/workflows/ci.yml
-
files_exist - File found:
.github/workflows/linting_comment.yml
-
files_exist - File found:
.github/workflows/linting.yml
-
files_exist - File found:
assets/email_template.html
-
files_exist - File found:
assets/email_template.txt
-
files_exist - File found:
assets/sendmail_template.txt
-
files_exist - File found:
assets/nf-core-rnaseq_logo_light.png
-
files_exist - File found:
conf/modules.config
-
files_exist - File found:
conf/test.config
-
files_exist - File found:
conf/test_full.config
-
files_exist - File found:
docs/images/nf-core-rnaseq_logo_light.png
-
files_exist - File found:
docs/images/nf-core-rnaseq_logo_dark.png
-
files_exist - File found:
docs/output.md
-
files_exist - File found:
docs/README.md
-
files_exist - File found:
docs/README.md
-
files_exist - File found:
docs/usage.md
-
files_exist - File found:
lib/nfcore_external_java_deps.jar
-
files_exist - File found:
lib/NfcoreTemplate.groovy
-
files_exist - File found:
lib/Utils.groovy
-
files_exist - File found:
lib/WorkflowMain.groovy
-
files_exist - File found:
main.nf
-
files_exist - File found:
assets/multiqc_config.yml
-
files_exist - File found:
conf/base.config
-
files_exist - File found:
conf/igenomes.config
-
files_exist - File found:
lib/WorkflowRnaseq.groovy
-
files_exist - File found:
modules.json
-
files_exist - File found:
pyproject.toml
-
files_exist - File not found check:
Singularity
-
files_exist - File not found check:
parameters.settings.json
-
files_exist - File not found check:
pipeline_template.yml
-
files_exist - File not found check:
.nf-core.yaml
-
files_exist - File not found check:
bin/markdown_to_html.r
-
files_exist - File not found check:
conf/aws.config
-
files_exist - File not found check:
.github/workflows/push_dockerhub.yml
-
files_exist - File not found check:
.github/ISSUE_TEMPLATE/bug_report.md
-
files_exist - File not found check:
.github/ISSUE_TEMPLATE/feature_request.md
-
files_exist - File not found check:
docs/images/nf-core-rnaseq_logo.png
-
files_exist - File not found check:
.markdownlint.yml
-
files_exist - File not found check:
.yamllint.yml
-
files_exist - File not found check:
lib/Checks.groovy
-
files_exist - File not found check:
lib/Completion.groovy
-
files_exist - File not found check:
lib/Workflow.groovy
-
files_exist - File not found check:
.travis.yml
-
nextflow_config - Config variable found:
manifest.name
-
nextflow_config - Config variable found:
manifest.nextflowVersion
-
nextflow_config - Config variable found:
manifest.description
-
nextflow_config - Config variable found:
manifest.version
-
nextflow_config - Config variable found:
manifest.homePage
-
nextflow_config - Config variable found:
timeline.enabled
-
nextflow_config - Config variable found:
trace.enabled
-
nextflow_config - Config variable found:
report.enabled
-
nextflow_config - Config variable found:
dag.enabled
-
nextflow_config - Config variable found:
process.cpus
-
nextflow_config - Config variable found:
process.memory
-
nextflow_config - Config variable found:
process.time
-
nextflow_config - Config variable found:
params.outdir
-
nextflow_config - Config variable found:
params.input
-
nextflow_config - Config variable found:
params.validationShowHiddenParams
-
nextflow_config - Config variable found:
params.validationSchemaIgnoreParams
-
nextflow_config - Config variable found:
manifest.mainScript
-
nextflow_config - Config variable found:
timeline.file
-
nextflow_config - Config variable found:
trace.file
-
nextflow_config - Config variable found:
report.file
-
nextflow_config - Config variable found:
dag.file
-
nextflow_config - Config variable (correctly) not found:
params.nf_required_version
-
nextflow_config - Config variable (correctly) not found:
params.container
-
nextflow_config - Config variable (correctly) not found:
params.singleEnd
-
nextflow_config - Config variable (correctly) not found:
params.igenomesIgnore
-
nextflow_config - Config variable (correctly) not found:
params.name
-
nextflow_config - Config variable (correctly) not found:
params.enable_conda
-
nextflow_config - Config
timeline.enabled
had correct value:true
-
nextflow_config - Config
report.enabled
had correct value:true
-
nextflow_config - Config
trace.enabled
had correct value:true
-
nextflow_config - Config
dag.enabled
had correct value:true
-
nextflow_config - Config
manifest.name
began withnf-core/
-
nextflow_config - Config variable
manifest.homePage
began with https://github.com/nf-core/ -
nextflow_config - Config
dag.file
ended with.html
-
nextflow_config - Config variable
manifest.nextflowVersion
started with >= or !>= -
nextflow_config - Config
manifest.version
ends indev
:3.14.0dev
-
nextflow_config - Config
params.custom_config_version
is set tomaster
-
nextflow_config - Config
params.custom_config_base
is set tohttps://raw.githubusercontent.com/nf-core/configs/master
- nextflow_config - Lines for loading custom profiles found
-
files_unchanged -
.gitattributes
matches the template -
files_unchanged -
.prettierrc.yml
matches the template -
files_unchanged -
CODE_OF_CONDUCT.md
matches the template -
files_unchanged -
LICENSE
matches the template -
files_unchanged -
.github/.dockstore.yml
matches the template -
files_unchanged -
.github/CONTRIBUTING.md
matches the template -
files_unchanged -
.github/ISSUE_TEMPLATE/bug_report.yml
matches the template -
files_unchanged -
.github/ISSUE_TEMPLATE/config.yml
matches the template -
files_unchanged -
.github/ISSUE_TEMPLATE/feature_request.yml
matches the template -
files_unchanged -
.github/PULL_REQUEST_TEMPLATE.md
matches the template -
files_unchanged -
.github/workflows/branch.yml
matches the template -
files_unchanged -
.github/workflows/linting_comment.yml
matches the template -
files_unchanged -
.github/workflows/linting.yml
matches the template -
files_unchanged -
assets/sendmail_template.txt
matches the template -
files_unchanged -
assets/nf-core-rnaseq_logo_light.png
matches the template -
files_unchanged -
docs/images/nf-core-rnaseq_logo_light.png
matches the template -
files_unchanged -
docs/images/nf-core-rnaseq_logo_dark.png
matches the template -
files_unchanged -
docs/README.md
matches the template -
files_unchanged -
lib/nfcore_external_java_deps.jar
matches the template - actions_ci - '.github/workflows/ci.yml' is triggered on expected events
-
readme - README Nextflow minimum version badge matched config. Badge:
23.04.0
, Config:23.04.0
- readme - README Zenodo placeholder was replaced with DOI.
- pipeline_name_conventions - Name adheres to nf-core convention
- template_strings - Did not find any Jinja template strings (353 files)
- schema_lint - Schema lint passed
- schema_lint - Schema title + description lint passed
- schema_lint - Input mimetype lint passed: 'text/csv'
- schema_params - Schema matched params returned from nextflow config
-
system_exit - No
System.exit
calls found - actions_schema_validation - Workflow validation passed: branch.yml
- actions_schema_validation - Workflow validation passed: clean-up.yml
- actions_schema_validation - Workflow validation passed: linting.yml
- actions_schema_validation - Workflow validation passed: fix-linting.yml
- actions_schema_validation - Workflow validation passed: ci.yml
- actions_schema_validation - Workflow validation passed: cloud_tests_full.yml
- actions_schema_validation - Workflow validation passed: cloud_tests_small.yml
- actions_schema_validation - Workflow validation passed: linting_comment.yml
- actions_schema_validation - Workflow validation passed: release-announcments.yml
- merge_markers - No merge markers found in pipeline files
-
modules_json - Only installed modules found in
modules.json
- modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
Run details
- nf-core/tools version 2.10
- Run at
2023-12-12 20:14:54
-225 lines!
I think you managed to hit the small window when the MultiQC bioconda build was broken. If you try again now it should be fine 🤞🏻
This is awesome! Do you think it'd be possible to go even further and use a topic to pick up all of the MultiQC inputs, not just the versions?
This is indeed awesome!! 🤩
When this is out in a release, we will first need to bump the minimum version of NF on nf-core/modules as well as in pipeline repos. Might need to delay a little to enable users to bump their NF version installs.
Do you think it'd be possible to go even further and use a topic to pick up all of the MultiQC inputs, not just the versions?
Presumably, you would need to explicitly add topic: multiqc
within all relevant modules for this to work?
output:
path "<SOME_MULTIQC_FILE", topic: multiqc
Is it possible to use emit
and topic
together in an output channel? There may be some instances where you want to use the output of a module in a workflow independent of a topic
.
I think you managed to hit the small window when the MultiQC bioconda build was broken
Okay I deleted the conda environment for multiqc but still got the error with a new build... does conda use a local cache that could be preventing a fresh download?
When this is out in a release, we will first need to bump the minimum version of NF on nf-core/modules as well as in pipeline repos.
This will be the challenge. It might be better to only patch the nf-core modules for the rnaseq pipeline and update the modules after another stable release (or two).
Is it possible to use emit and topic together in an output channel? There may be some instances where you want to use the output of a module in a workflow independent of a topic.
~~Currently it is not possible, not sure how difficult it would be to support. I can look at the multiqc inputs and see if a topic channel would be appropriate.~~
In any case, I tested rnaseq on master and it fails with the same error, so that at least makes me confident that there's no issue with the topic channels
As for the multiqc inputs, a topic might not be the best choice because it is not obvious that the corresponding process outputs are intended solely for multiqc.
For example, the FASTP
process:
output:
tuple val(meta), path('*.fastp.fastq.gz') , optional:true, emit: reads
tuple val(meta), path('*.json') , emit: json
tuple val(meta), path('*.html') , emit: html
tuple val(meta), path('*.log') , emit: log
path "versions.yml" , topic: versions
tuple val(meta), path('*.fail.fastq.gz') , optional:true, emit: reads_fail
tuple val(meta), path('*.merged.fastq.gz'), optional:true, emit: reads_merged
Some of these outputs are sent to multiqc, but you couldn't tell from the process definition. I think forcing a channel topic here would be more trouble than it's worth.
Shall we move this forward? Channel topics are available in 23.11.0-edge. Let me know how you want to handle it. Or maybe someone from nf-core can take it forward, don't really need me anymore 😄
Is it possible to use emit and topic together in an output channel? There may be some instances where you want to use the output of a module in a workflow independent of a topic.
I was wrong about this, you can use emit
and topic
together. So if you guys want you can add emit: ch_versions
back to the process outputs.
Also would love to hear back about my comment regarding the use of env
output versus the experimental cmd
output. If you prefer the env
approach then we can implement that right away, for the cmd
approach we can implement separately if/when it is merged.
I updated the CI and tests are passing now. The failing tests should work if they use 23.11.0-edge, but not sure how you wanted to manage that.