Store output of different pathways in different folders

Open bjlang opened this issue 1 year ago • 2 comments

PR checklist

[x] This comment contains a description of changes (with reason).
[ ] If you've fixed a bug or added code that should be tested, add tests!
[ ] If you've added a new tool - have you followed the pipeline conventions in the contribution docs
[ ] If necessary, also make a PR on the nf-core/differentialabundance branch on the nf-core/test-datasets repository.
[x] Make sure your code lints (nf-core lint).
[x] Ensure the test suite passes (nf-test test main.nf.test -profile test,docker).
[ ] Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
[ ] Usage Documentation in docs/usage.md is updated.
[ ] Output Documentation in docs/output.md is updated.
[ ] CHANGELOG.md is updated.
[ ] README.md is updated (including new tool citations and authors/contributors).

Oct 01 '24 14:10 bjlang

`nf-core lint` overall result: Passed :white_check_mark: :warning:

Posted for pipeline commit a6c2b3d

+| ✅ 301 tests passed       |+
#| ❔   4 tests were ignored |#
!| ❗   3 tests had warnings |!

:heavy_exclamation_mark: Test warnings:

pipeline_todos - TODO string in main.nf: Optionally add in-text citation tools to this list.
pipeline_todos - TODO string in main.nf: Optionally add bibliographic entries to this list.
pipeline_todos - TODO string in main.nf: Only uncomment below if logic in toolCitationText/toolBibliographyText has been filled!

:grey_question: Tests ignored:

nextflow_config - Config default ignored: params.report_file
nextflow_config - Config default ignored: params.logo_file
nextflow_config - Config default ignored: params.css_file
nextflow_config - Config default ignored: params.citations_file

:white_check_mark: Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-differentialabundance_logo_light.png
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/images/nf-core-differentialabundance_logo_light.png
files_exist - File found: docs/images/nf-core-differentialabundance_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: modules.json
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: docs/images/nf-core-differentialabundance_logo.png
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/NfcoreTemplate.groovy
files_exist - File not found check: lib/Utils.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: lib/WorkflowMain.groovy
files_exist - File not found check: lib/WorkflowDifferentialabundance.groovy
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: pipeline_template.yml
files_exist - File not found check: Singularity
files_exist - File not found check: lib/nfcore_external_java_deps.jar
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.validationShowHiddenParams
nextflow_config - Config variable found: params.validationSchemaIgnoreParams
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: 1.6.0dev
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
nextflow_config - nextflow.config contains configuration profile test
nextflow_config - Config default value correct: params.study_name= study
nextflow_config - Config default value correct: params.study_type= rnaseq
nextflow_config - Config default value correct: params.study_abundance_type= counts
nextflow_config - Config default value correct: params.pathway= diff_prop
nextflow_config - Config default value correct: params.affy_cel_files_archive= null
nextflow_config - Config default value correct: params.querygse= null
nextflow_config - Config default value correct: params.observations_id_col= sample
nextflow_config - Config default value correct: params.observations_type= sample
nextflow_config - Config default value correct: params.features_id_col= gene_id
nextflow_config - Config default value correct: params.features_name_col= gene_name
nextflow_config - Config default value correct: params.features_type= gene
nextflow_config - Config default value correct: params.features_metadata_cols= gene_id,gene_name,gene_biotype
nextflow_config - Config default value correct: params.features_gtf_feature_type= transcript
nextflow_config - Config default value correct: params.features_gtf_table_first_field= gene_id
nextflow_config - Config default value correct: params.affy_file_name_col= file
nextflow_config - Config default value correct: params.affy_background= true
nextflow_config - Config default value correct: params.affy_bgversion= 2
nextflow_config - Config default value correct: params.affy_cdfname= null
nextflow_config - Config default value correct: params.affy_build_annotation= true
nextflow_config - Config default value correct: params.proteus_measurecol_prefix= LFQ intensity
nextflow_config - Config default value correct: params.proteus_norm_function= normalizeMedian
nextflow_config - Config default value correct: params.proteus_plotsd_method= violin
nextflow_config - Config default value correct: params.proteus_plotmv_loess= true
nextflow_config - Config default value correct: params.proteus_palette_name= Set1
nextflow_config - Config default value correct: params.filtering_min_abundance= 1.0
nextflow_config - Config default value correct: params.filtering_min_samples= 1.0
nextflow_config - Config default value correct: params.filtering_min_proportion_not_na= 0.5
nextflow_config - Config default value correct: params.exploratory_clustering_method= ward.D2
nextflow_config - Config default value correct: params.exploratory_cor_method= spearman
nextflow_config - Config default value correct: params.exploratory_n_features= 500
nextflow_config - Config default value correct: params.exploratory_whisker_distance= 1.5
nextflow_config - Config default value correct: params.exploratory_mad_threshold= -5
nextflow_config - Config default value correct: params.exploratory_main_variable= auto_pca
nextflow_config - Config default value correct: params.exploratory_assay_names= raw,normalised,variance_stabilised
nextflow_config - Config default value correct: params.exploratory_final_assay= variance_stabilised
nextflow_config - Config default value correct: params.exploratory_palette_name= Set1
nextflow_config - Config default value correct: params.differential_feature_id_column= gene_id
nextflow_config - Config default value correct: params.differential_fc_column= log2FoldChange
nextflow_config - Config default value correct: params.differential_pval_column= pvalue
nextflow_config - Config default value correct: params.differential_qval_column= padj
nextflow_config - Config default value correct: params.differential_min_fold_change= 2.0
nextflow_config - Config default value correct: params.differential_max_pval= 1.0
nextflow_config - Config default value correct: params.differential_max_qval= 0.05
nextflow_config - Config default value correct: params.differential_feature_name_column= gene_name
nextflow_config - Config default value correct: params.differential_foldchanges_logged= true
nextflow_config - Config default value correct: params.differential_palette_name= Set1
nextflow_config - Config default value correct: params.deseq2_test= Wald
nextflow_config - Config default value correct: params.deseq2_fit_type= parametric
nextflow_config - Config default value correct: params.deseq2_sf_type= ratio
nextflow_config - Config default value correct: params.deseq2_min_replicates_for_replace= 7
nextflow_config - Config default value correct: params.deseq2_independent_filtering= true
nextflow_config - Config default value correct: params.deseq2_lfc_threshold= 0
nextflow_config - Config default value correct: params.deseq2_alt_hypothesis= greaterAbs
nextflow_config - Config default value correct: params.deseq2_p_adjust_method= BH
nextflow_config - Config default value correct: params.deseq2_alpha= 0.1
nextflow_config - Config default value correct: params.deseq2_minmu= 0.5
nextflow_config - Config default value correct: params.deseq2_vs_method= vst
nextflow_config - Config default value correct: params.deseq2_shrink_lfc= true
nextflow_config - Config default value correct: params.deseq2_cores= 1
nextflow_config - Config default value correct: params.deseq2_vs_blind= true
nextflow_config - Config default value correct: params.deseq2_vst_nsub= 1000
nextflow_config - Config default value correct: params.limma_spacing= null
nextflow_config - Config default value correct: params.limma_block= null
nextflow_config - Config default value correct: params.limma_correlation= null
nextflow_config - Config default value correct: params.limma_method= ls
nextflow_config - Config default value correct: params.limma_proportion= 0.01
nextflow_config - Config default value correct: params.limma_stdev_coef_lim= 0.1,4
nextflow_config - Config default value correct: params.limma_winsor_tail_p= 0.05,0.1
nextflow_config - Config default value correct: params.limma_lfc= 0
nextflow_config - Config default value correct: params.limma_adjust_method= BH
nextflow_config - Config default value correct: params.limma_p_value= 1.0
nextflow_config - Config default value correct: params.gsea_permute= phenotype
nextflow_config - Config default value correct: params.gsea_nperm= 1000
nextflow_config - Config default value correct: params.gsea_scoring_scheme= weighted
nextflow_config - Config default value correct: params.gsea_metric= Signal2Noise
nextflow_config - Config default value correct: params.gsea_sort= real
nextflow_config - Config default value correct: params.gsea_order= descending
nextflow_config - Config default value correct: params.gsea_set_max= 500
nextflow_config - Config default value correct: params.gsea_set_min= 15
nextflow_config - Config default value correct: params.gsea_norm= meandiv
nextflow_config - Config default value correct: params.gsea_rnd_type= no_balance
nextflow_config - Config default value correct: params.gsea_make_sets= true
nextflow_config - Config default value correct: params.gsea_num= 100
nextflow_config - Config default value correct: params.gsea_plot_top_x= 20
nextflow_config - Config default value correct: params.gsea_rnd_seed= timestamp
nextflow_config - Config default value correct: params.gprofiler2_significant= true
nextflow_config - Config default value correct: params.gprofiler2_measure_underrepresentation= false
nextflow_config - Config default value correct: params.gprofiler2_evcodes= false
nextflow_config - Config default value correct: params.gprofiler2_max_qval= 0.05
nextflow_config - Config default value correct: params.gprofiler2_domain_scope= annotated
nextflow_config - Config default value correct: params.gprofiler2_min_diff= 1
nextflow_config - Config default value correct: params.gprofiler2_palette_name= Blues
nextflow_config - Config default value correct: params.shinyngs_build_app= true
nextflow_config - Config default value correct: params.shinyngs_shinyapps_account= null
nextflow_config - Config default value correct: params.shinyngs_shinyapps_app_name= null
nextflow_config - Config default value correct: params.gene_sets_files= null
nextflow_config - Config default value correct: params.report_title= null
nextflow_config - Config default value correct: params.report_author= null
nextflow_config - Config default value correct: params.report_description= null
nextflow_config - Config default value correct: params.report_scree= true
nextflow_config - Config default value correct: params.report_round_digits= 4
nextflow_config - Config default value correct: params.custom_config_version= master
nextflow_config - Config default value correct: params.custom_config_base= https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Config default value correct: params.max_cpus= 16
nextflow_config - Config default value correct: params.max_memory= 128.GB
nextflow_config - Config default value correct: params.max_time= 240.h
nextflow_config - Config default value correct: params.publish_dir_mode= copy
nextflow_config - Config default value correct: params.validate_params= true
nextflow_config - Config default value correct: params.pipelines_testdata_base_path= https://raw.githubusercontent.com/nf-core/test-datasets/
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - assets/nf-core-differentialabundance_logo_light.png matches the template
files_unchanged - docs/images/nf-core-differentialabundance_logo_light.png matches the template
files_unchanged - docs/images/nf-core-differentialabundance_logo_dark.png matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 23.10.0, Config: 23.10.0
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (224 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: release-announcements.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: download_pipeline.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - assets/multiqc_config.yml found and not ignored.
multiqc_config - assets/multiqc_config.yml contains report_section_order
multiqc_config - assets/multiqc_config.yml contains export_plots
multiqc_config - assets/multiqc_config.yml contains report_comment
multiqc_config - assets/multiqc_config.yml follows the ordering scheme of the minimally required plugins.
multiqc_config - assets/multiqc_config.yml contains a matching 'report_comment'.
multiqc_config - assets/multiqc_config.yml contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'
base_config - conf/base.config found and not ignored.
modules_config - conf/modules.config found and not ignored.
modules_config - GUNZIP_GTF found in conf/modules.config and Nextflow scripts.
modules_config - GTF_TO_TABLE found in conf/modules.config and Nextflow scripts.
modules_config - VALIDATOR found in conf/modules.config and Nextflow scripts.
modules_config - AFFY_JUSTRMA_RAW found in conf/modules.config and Nextflow scripts.
modules_config - AFFY_JUSTRMA_NORM found in conf/modules.config and Nextflow scripts.
modules_config - PROTEUS found in conf/modules.config and Nextflow scripts.
modules_config - GEOQUERY_GETGEO found in conf/modules.config and Nextflow scripts.
modules_config - DESEQ2_NORM found in conf/modules.config and Nextflow scripts.
modules_config - DESEQ2_DIFFERENTIAL found in conf/modules.config and Nextflow scripts.
modules_config - LIMMA_DIFFERENTIAL found in conf/modules.config and Nextflow scripts.
modules_config - FILTER_DIFFTABLE found in conf/modules.config and Nextflow scripts.
modules_config - GSEA_GSEA found in conf/modules.config and Nextflow scripts.
modules_config - GPROFILER2_GOST found in conf/modules.config and Nextflow scripts.
modules_config - PLOT_EXPLORATORY found in conf/modules.config and Nextflow scripts.
modules_config - PLOT_DIFFERENTIAL found in conf/modules.config and Nextflow scripts.
modules_config - SHINYNGS_APP found in conf/modules.config and Nextflow scripts.
modules_config - RMARKDOWNNOTEBOOK found in conf/modules.config and Nextflow scripts.
modules_config - MAKE_REPORT_BUNDLE found in conf/modules.config and Nextflow scripts.
modules_config - CUSTOM_MATRIXFILTER found in conf/modules.config and Nextflow scripts.
modules_config - CUSTOM_TABULARTOGSEACLS found in conf/modules.config and Nextflow scripts.
modules_config - CUSTOM_TABULARTOGSEAGCT found in conf/modules.config and Nextflow scripts.
modules_config - TABULAR_TO_GSEA_CHIP found in conf/modules.config and Nextflow scripts.
modules_config - PROPR found in conf/modules.config and Nextflow scripts.
modules_config - PROPD found in conf/modules.config and Nextflow scripts.
modules_config - FILTERVAR found in conf/modules.config and Nextflow scripts.
modules_config - GREA_DIFF found in conf/modules.config and Nextflow scripts.
modules_config - GREA_COR found in conf/modules.config and Nextflow scripts.
modules_config - MYGENE found in conf/modules.config and Nextflow scripts.
nfcore_yml - Repository type in .nf-core.yml is valid: pipeline
nfcore_yml - nf-core version in .nf-core.yml is set to the latest version: 2.14.1

Run details

nf-core/tools version 2.14.1
Run at 2024-10-01 14:34:12

Oct 01 '24 14:10 github-actions[bot]

hello @bjlang ! If we have two pathways with common steps, like for example:

path  | method_diff   | args_diff         | method_cor | args_cor | method_enrich   | args_enrich
-------------------------------------------------------------------------------------------------
path1 | method_diff_1 | arg1='a' arg3='b' | empty      | empty    | method_enrich_2 | arg4 = 'c'
path2 | method_diff_1 | arg1='a' arg3='b' | empty      | empty    | method_enrich_1 | empty

the pipeline will run three processes:

> process 1 with an input like:
tuple val meta [
             method_diff = method_diff_1, 
             arg_diff = other arguments from modules.config + "arg1='a' arg3='b'"
     ],
     file input,
     etc
> process 2 with an input like:
tuple val meta [
             method_diff = method_diff_1, 
             arg_diff = other arguments from modules.config + "arg1='a' arg3='b'",
             method_enrich = method_enrich_2,
             arg_enrich = other arguments from modules.config + "arg4 = 'c'"
     ],
     file from process 1,
     etc
> process 3 with an input like:
tuple val meta [
             method_diff = method_diff_1, 
             arg_diff = other arguments from modules.config + "arg1='a' arg3='b'",
             method_enrich = method_enrich_1,
             arg_enrich = arguments from modules.config
     ],
     file from process 1,
     etc

So if we use pathway specific outdir nomenclature, wouldn't it be confusing to have two outdir with the same output? Or is it actually possible to output the same files into two different pathway-specific output directories?

Oct 02 '24 09:10 suzannejin

Store output of different pathways in different folders

PR checklist

nf-core lint overall result: Passed :white_check_mark: :warning:

:heavy_exclamation_mark: Test warnings:

:grey_question: Tests ignored:

:white_check_mark: Tests passed:

Run details

`nf-core lint` overall result: Passed :white_check_mark: :warning: