DSL2 dev merge checklist

Open edmundmiller opened this issue 3 years ago • 1 comments

[x] Merge #242 to get up to date with the template
[x] Try to resolve nf-core lint stuff, eg. updating modules
[ ] Finish multicore Bismark module PR: https://github.com/nf-core/modules/pull/1036
[ ] Check + fix any other CI issues (test dataset on GitHub)
[ ] Manual AWS test run
[ ] Manual AWS test_full run on both dev and dsl2

'cc @ewels

Sep 14 '22 02:09 edmundmiller

Tower tests URLs: (check nested when complete, test top-level when compared)

Profile: `test`

[ ] Aligner: bismark
- [ ] dsl2: [ tower]
- [x] dev: [ tower - nostalgic_bose ]
[ ] Aligner: bismark_hisat
- [ ] dsl2: [ tower ]
- [x] dev: [ tower - maniac_sax ]
[ ] Aligner: bwameth
- [ ] dsl2: [ tower ]
- [x] dev: [ tower - mad_mahavira ]

Profile: `test_full`

[ ] Aligner: bismark
- [ ] dsl2: [ tower - voluminous_lovelace ]
- [ ] dev: [ tower - angry_lavoisier ]
[ ] Aligner: bismark_hisat
- [ ] dsl2: [ tower ]
- [ ] dev: [ tower ]
[ ] Aligner: bwameth
- [ ] dsl2: [ tower ]
- [ ] dev: [ tower ]

Sep 28 '22 09:09 ewels

Testing on GitPod with `-profile test`

Bismark

master file tree

gitpod /workspace/methylseq/test_master/test_bismark (master) $ tree results/
results/
├── bismark_alignments
│   ├── logs
│   │   ├── SRR389222_sub1_trimmed_bismark_bt2_SE_report.txt
│   │   ├── SRR389222_sub2_trimmed_bismark_bt2_SE_report.txt
│   │   └── SRR389222_sub3_trimmed_bismark_bt2_SE_report.txt
│   └── where_are_my_files.txt
├── bismark_deduplicated
│   ├── logs
│   │   ├── SRR389222_sub1_trimmed_bismark_bt2.deduplication_report.txt
│   │   ├── SRR389222_sub2_trimmed_bismark_bt2.deduplication_report.txt
│   │   └── SRR389222_sub3_trimmed_bismark_bt2.deduplication_report.txt
│   ├── SRR389222_sub1_trimmed_bismark_bt2.deduplicated.bam
│   ├── SRR389222_sub2_trimmed_bismark_bt2.deduplicated.bam
│   └── SRR389222_sub3_trimmed_bismark_bt2.deduplicated.bam
├── bismark_methylation_calls
│   ├── bedGraph
│   │   ├── SRR389222_sub1_trimmed_bismark_bt2.deduplicated.bedGraph.gz
│   │   ├── SRR389222_sub2_trimmed_bismark_bt2.deduplicated.bedGraph.gz
│   │   └── SRR389222_sub3_trimmed_bismark_bt2.deduplicated.bedGraph.gz
│   ├── logs
│   │   ├── SRR389222_sub1_trimmed_bismark_bt2.deduplicated_splitting_report.txt
│   │   ├── SRR389222_sub2_trimmed_bismark_bt2.deduplicated_splitting_report.txt
│   │   └── SRR389222_sub3_trimmed_bismark_bt2.deduplicated_splitting_report.txt
│   ├── m-bias
│   │   ├── SRR389222_sub1_trimmed_bismark_bt2.deduplicated.M-bias.txt
│   │   ├── SRR389222_sub2_trimmed_bismark_bt2.deduplicated.M-bias.txt
│   │   └── SRR389222_sub3_trimmed_bismark_bt2.deduplicated.M-bias.txt
│   ├── methylation_calls
│   │   ├── CHG_OB_SRR389222_sub1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CHG_OB_SRR389222_sub2_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CHG_OB_SRR389222_sub3_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CHG_OT_SRR389222_sub1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CHG_OT_SRR389222_sub2_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CHG_OT_SRR389222_sub3_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CHH_OB_SRR389222_sub1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CHH_OB_SRR389222_sub2_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CHH_OB_SRR389222_sub3_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CHH_OT_SRR389222_sub1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CHH_OT_SRR389222_sub2_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CHH_OT_SRR389222_sub3_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CpG_OB_SRR389222_sub1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CpG_OB_SRR389222_sub2_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CpG_OB_SRR389222_sub3_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CpG_OT_SRR389222_sub1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── CpG_OT_SRR389222_sub2_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   └── CpG_OT_SRR389222_sub3_trimmed_bismark_bt2.deduplicated.txt.gz
│   └── methylation_coverage
│       ├── SRR389222_sub1_trimmed_bismark_bt2.deduplicated.bismark.cov.gz
│       ├── SRR389222_sub2_trimmed_bismark_bt2.deduplicated.bismark.cov.gz
│       └── SRR389222_sub3_trimmed_bismark_bt2.deduplicated.bismark.cov.gz
├── bismark_reports
│   ├── SRR389222_sub1_trimmed_bismark_bt2_SE_report.html
│   ├── SRR389222_sub2_trimmed_bismark_bt2_SE_report.html
│   └── SRR389222_sub3_trimmed_bismark_bt2_SE_report.html
├── bismark_summary
│   ├── bismark_summary_report.html
│   └── bismark_summary_report.txt
├── fastqc
│   ├── SRR389222_sub1_fastqc.html
│   ├── SRR389222_sub2_fastqc.html
│   ├── SRR389222_sub3_fastqc.html
│   └── zips
│       ├── SRR389222_sub1_fastqc.zip
│       ├── SRR389222_sub2_fastqc.zip
│       └── SRR389222_sub3_fastqc.zip
├── MultiQC
│   ├── multiqc_data
│   │   ├── mqc_bismark_alignment_1.txt
│   │   ├── mqc_bismark_deduplication_1.txt
│   │   ├── mqc_bismark_mbias_CHG_R1.txt
│   │   ├── mqc_bismark_mbias_CHH_R1.txt
│   │   ├── mqc_bismark_mbias_CpG_R1.txt
│   │   ├── mqc_bismark_strand_alignment_1.txt
│   │   ├── mqc_cutadapt_filtered_reads_plot_1.txt
│   │   ├── mqc_cutadapt_trimmed_sequences_plot_Counts.txt
│   │   ├── mqc_cutadapt_trimmed_sequences_plot_Obs_Exp.txt
│   │   ├── mqc_fastqc_overrepresented_sequencesi_plot_1.txt
│   │   ├── mqc_fastqc_per_base_n_content_plot_1.txt
│   │   ├── mqc_fastqc_per_base_sequence_quality_plot_1.txt
│   │   ├── mqc_fastqc_per_sequence_gc_content_plot_Counts.txt
│   │   ├── mqc_fastqc_per_sequence_gc_content_plot_Percentages.txt
│   │   ├── mqc_fastqc_per_sequence_quality_scores_plot_1.txt
│   │   ├── mqc_fastqc_sequence_counts_plot_1.txt
│   │   ├── mqc_fastqc_sequence_duplication_levels_plot_1.txt
│   │   ├── mqc_qualimap_coverage_histogram_1.txt
│   │   ├── mqc_qualimap_gc_content_1.txt
│   │   ├── mqc_qualimap_genome_fraction_1.txt
│   │   ├── multiqc_bismark_alignment.txt
│   │   ├── multiqc_bismark_dedup.txt
│   │   ├── multiqc_bismark_methextract.txt
│   │   ├── multiqc_cutadapt.txt
│   │   ├── multiqc_data.json
│   │   ├── multiqc_fastqc.txt
│   │   ├── multiqc_general_stats.txt
│   │   ├── multiqc.log
│   │   ├── multiqc_qualimap_bamqc_genome_results.txt
│   │   └── multiqc_sources.txt
│   ├── multiqc_plots
│   │   ├── pdf
│   │   │   ├── mqc_bismark_alignment_1_pc.pdf
│   │   │   ├── mqc_bismark_alignment_1.pdf
│   │   │   ├── mqc_bismark_deduplication_1_pc.pdf
│   │   │   ├── mqc_bismark_deduplication_1.pdf
│   │   │   ├── mqc_bismark_mbias_CHG_R1.pdf
│   │   │   ├── mqc_bismark_mbias_CHH_R1.pdf
│   │   │   ├── mqc_bismark_mbias_CpG_R1.pdf
│   │   │   ├── mqc_bismark_strand_alignment_1_pc.pdf
│   │   │   ├── mqc_bismark_strand_alignment_1.pdf
│   │   │   ├── mqc_cutadapt_filtered_reads_plot_1_pc.pdf
│   │   │   ├── mqc_cutadapt_filtered_reads_plot_1.pdf
│   │   │   ├── mqc_cutadapt_trimmed_sequences_plot_Counts.pdf
│   │   │   ├── mqc_cutadapt_trimmed_sequences_plot_Obs_Exp.pdf
│   │   │   ├── mqc_fastqc_overrepresented_sequencesi_plot_1.pdf
│   │   │   ├── mqc_fastqc_per_base_n_content_plot_1.pdf
│   │   │   ├── mqc_fastqc_per_base_sequence_quality_plot_1.pdf
│   │   │   ├── mqc_fastqc_per_sequence_gc_content_plot_Counts.pdf
│   │   │   ├── mqc_fastqc_per_sequence_gc_content_plot_Percentages.pdf
│   │   │   ├── mqc_fastqc_per_sequence_quality_scores_plot_1.pdf
│   │   │   ├── mqc_fastqc_sequence_counts_plot_1_pc.pdf
│   │   │   ├── mqc_fastqc_sequence_counts_plot_1.pdf
│   │   │   ├── mqc_fastqc_sequence_duplication_levels_plot_1.pdf
│   │   │   ├── mqc_qualimap_coverage_histogram_1.pdf
│   │   │   ├── mqc_qualimap_gc_content_1.pdf
│   │   │   └── mqc_qualimap_genome_fraction_1.pdf
│   │   ├── png
│   │   │   ├── mqc_bismark_alignment_1_pc.png
│   │   │   ├── mqc_bismark_alignment_1.png
│   │   │   ├── mqc_bismark_deduplication_1_pc.png
│   │   │   ├── mqc_bismark_deduplication_1.png
│   │   │   ├── mqc_bismark_mbias_CHG_R1.png
│   │   │   ├── mqc_bismark_mbias_CHH_R1.png
│   │   │   ├── mqc_bismark_mbias_CpG_R1.png
│   │   │   ├── mqc_bismark_strand_alignment_1_pc.png
│   │   │   ├── mqc_bismark_strand_alignment_1.png
│   │   │   ├── mqc_cutadapt_filtered_reads_plot_1_pc.png
│   │   │   ├── mqc_cutadapt_filtered_reads_plot_1.png
│   │   │   ├── mqc_cutadapt_trimmed_sequences_plot_Counts.png
│   │   │   ├── mqc_cutadapt_trimmed_sequences_plot_Obs_Exp.png
│   │   │   ├── mqc_fastqc_overrepresented_sequencesi_plot_1.png
│   │   │   ├── mqc_fastqc_per_base_n_content_plot_1.png
│   │   │   ├── mqc_fastqc_per_base_sequence_quality_plot_1.png
│   │   │   ├── mqc_fastqc_per_sequence_gc_content_plot_Counts.png
│   │   │   ├── mqc_fastqc_per_sequence_gc_content_plot_Percentages.png
│   │   │   ├── mqc_fastqc_per_sequence_quality_scores_plot_1.png
│   │   │   ├── mqc_fastqc_sequence_counts_plot_1_pc.png
│   │   │   ├── mqc_fastqc_sequence_counts_plot_1.png
│   │   │   ├── mqc_fastqc_sequence_duplication_levels_plot_1.png
│   │   │   ├── mqc_qualimap_coverage_histogram_1.png
│   │   │   ├── mqc_qualimap_gc_content_1.png
│   │   │   └── mqc_qualimap_genome_fraction_1.png
│   │   └── svg
│   │       ├── mqc_bismark_alignment_1_pc.svg
│   │       ├── mqc_bismark_alignment_1.svg
│   │       ├── mqc_bismark_deduplication_1_pc.svg
│   │       ├── mqc_bismark_deduplication_1.svg
│   │       ├── mqc_bismark_mbias_CHG_R1.svg
│   │       ├── mqc_bismark_mbias_CHH_R1.svg
│   │       ├── mqc_bismark_mbias_CpG_R1.svg
│   │       ├── mqc_bismark_strand_alignment_1_pc.svg
│   │       ├── mqc_bismark_strand_alignment_1.svg
│   │       ├── mqc_cutadapt_filtered_reads_plot_1_pc.svg
│   │       ├── mqc_cutadapt_filtered_reads_plot_1.svg
│   │       ├── mqc_cutadapt_trimmed_sequences_plot_Counts.svg
│   │       ├── mqc_cutadapt_trimmed_sequences_plot_Obs_Exp.svg
│   │       ├── mqc_fastqc_overrepresented_sequencesi_plot_1.svg
│   │       ├── mqc_fastqc_per_base_n_content_plot_1.svg
│   │       ├── mqc_fastqc_per_base_sequence_quality_plot_1.svg
│   │       ├── mqc_fastqc_per_sequence_gc_content_plot_Counts.svg
│   │       ├── mqc_fastqc_per_sequence_gc_content_plot_Percentages.svg
│   │       ├── mqc_fastqc_per_sequence_quality_scores_plot_1.svg
│   │       ├── mqc_fastqc_sequence_counts_plot_1_pc.svg
│   │       ├── mqc_fastqc_sequence_counts_plot_1.svg
│   │       ├── mqc_fastqc_sequence_duplication_levels_plot_1.svg
│   │       ├── mqc_qualimap_coverage_histogram_1.svg
│   │       ├── mqc_qualimap_gc_content_1.svg
│   │       └── mqc_qualimap_genome_fraction_1.svg
│   └── multiqc_report.html
├── pipeline_info
│   ├── execution_report.html
│   ├── execution_timeline.html
│   ├── execution_trace.txt
│   ├── execution_trace.txt.1
│   ├── pipeline_dag.dot
│   ├── pipeline_report.html
│   ├── pipeline_report.txt
│   ├── results_description.html
│   └── software_versions.csv
├── qualimap
│   ├── SRR389222_sub1_trimmed_bismark_bt2.deduplicated_qualimap
│   │   ├── css
│   │   │   ├── agogo.css
│   │   │   ├── ajax-loader.gif
│   │   │   ├── basic.css
│   │   │   ├── bgfooter.png
│   │   │   ├── bgtop.png
│   │   │   ├── comment-bright.png
│   │   │   ├── comment-close.png
│   │   │   ├── comment.png
│   │   │   ├── doctools.js
│   │   │   ├── down.png
│   │   │   ├── down-pressed.png
│   │   │   ├── file.png
│   │   │   ├── jquery.js
│   │   │   ├── minus.png
│   │   │   ├── plus.png
│   │   │   ├── pygments.css
│   │   │   ├── qualimap_logo_small.png
│   │   │   ├── report.css
│   │   │   ├── searchtools.js
│   │   │   ├── underscore.js
│   │   │   ├── up.png
│   │   │   ├── up-pressed.png
│   │   │   └── websupport.js
│   │   ├── genome_results.txt
│   │   ├── images_qualimapReport
│   │   │   ├── genome_coverage_0to50_histogram.png
│   │   │   ├── genome_coverage_across_reference.png
│   │   │   ├── genome_coverage_histogram.png
│   │   │   ├── genome_coverage_quotes.png
│   │   │   ├── genome_gc_content_per_window.png
│   │   │   ├── genome_mapping_quality_across_reference.png
│   │   │   ├── genome_mapping_quality_histogram.png
│   │   │   ├── genome_reads_content_per_read_position.png
│   │   │   └── genome_uniq_read_starts_histogram.png
│   │   ├── qualimapReport.html
│   │   └── raw_data_qualimapReport
│   │       ├── coverage_across_reference.txt
│   │       ├── coverage_histogram.txt
│   │       ├── duplication_rate_histogram.txt
│   │       ├── genome_fraction_coverage.txt
│   │       ├── mapped_reads_gc-content_distribution.txt
│   │       ├── mapped_reads_nucleotide_content.txt
│   │       ├── mapping_quality_across_reference.txt
│   │       └── mapping_quality_histogram.txt
│   ├── SRR389222_sub2_trimmed_bismark_bt2.deduplicated_qualimap
│   │   ├── css
│   │   │   ├── agogo.css
│   │   │   ├── ajax-loader.gif
│   │   │   ├── basic.css
│   │   │   ├── bgfooter.png
│   │   │   ├── bgtop.png
│   │   │   ├── comment-bright.png
│   │   │   ├── comment-close.png
│   │   │   ├── comment.png
│   │   │   ├── doctools.js
│   │   │   ├── down.png
│   │   │   ├── down-pressed.png
│   │   │   ├── file.png
│   │   │   ├── jquery.js
│   │   │   ├── minus.png
│   │   │   ├── plus.png
│   │   │   ├── pygments.css
│   │   │   ├── qualimap_logo_small.png
│   │   │   ├── report.css
│   │   │   ├── searchtools.js
│   │   │   ├── underscore.js
│   │   │   ├── up.png
│   │   │   ├── up-pressed.png
│   │   │   └── websupport.js
│   │   ├── genome_results.txt
│   │   ├── images_qualimapReport
│   │   │   ├── genome_coverage_0to50_histogram.png
│   │   │   ├── genome_coverage_across_reference.png
│   │   │   ├── genome_coverage_histogram.png
│   │   │   ├── genome_coverage_quotes.png
│   │   │   ├── genome_gc_content_per_window.png
│   │   │   ├── genome_mapping_quality_across_reference.png
│   │   │   ├── genome_mapping_quality_histogram.png
│   │   │   ├── genome_reads_content_per_read_position.png
│   │   │   └── genome_uniq_read_starts_histogram.png
│   │   ├── qualimapReport.html
│   │   └── raw_data_qualimapReport
│   │       ├── coverage_across_reference.txt
│   │       ├── coverage_histogram.txt
│   │       ├── duplication_rate_histogram.txt
│   │       ├── genome_fraction_coverage.txt
│   │       ├── mapped_reads_gc-content_distribution.txt
│   │       ├── mapped_reads_nucleotide_content.txt
│   │       ├── mapping_quality_across_reference.txt
│   │       └── mapping_quality_histogram.txt
│   └── SRR389222_sub3_trimmed_bismark_bt2.deduplicated_qualimap
│       ├── css
│       │   ├── agogo.css
│       │   ├── ajax-loader.gif
│       │   ├── basic.css
│       │   ├── bgfooter.png
│       │   ├── bgtop.png
│       │   ├── comment-bright.png
│       │   ├── comment-close.png
│       │   ├── comment.png
│       │   ├── doctools.js
│       │   ├── down.png
│       │   ├── down-pressed.png
│       │   ├── file.png
│       │   ├── jquery.js
│       │   ├── minus.png
│       │   ├── plus.png
│       │   ├── pygments.css
│       │   ├── qualimap_logo_small.png
│       │   ├── report.css
│       │   ├── searchtools.js
│       │   ├── underscore.js
│       │   ├── up.png
│       │   ├── up-pressed.png
│       │   └── websupport.js
│       ├── genome_results.txt
│       ├── images_qualimapReport
│       │   ├── genome_coverage_0to50_histogram.png
│       │   ├── genome_coverage_across_reference.png
│       │   ├── genome_coverage_histogram.png
│       │   ├── genome_coverage_quotes.png
│       │   ├── genome_gc_content_per_window.png
│       │   ├── genome_mapping_quality_across_reference.png
│       │   ├── genome_mapping_quality_histogram.png
│       │   ├── genome_reads_content_per_read_position.png
│       │   └── genome_uniq_read_starts_histogram.png
│       ├── qualimapReport.html
│       └── raw_data_qualimapReport
│           ├── coverage_across_reference.txt
│           ├── coverage_histogram.txt
│           ├── duplication_rate_histogram.txt
│           ├── genome_fraction_coverage.txt
│           ├── mapped_reads_gc-content_distribution.txt
│           ├── mapped_reads_nucleotide_content.txt
│           ├── mapping_quality_across_reference.txt
│           └── mapping_quality_histogram.txt
└── trim_galore
    ├── FastQC
    │   ├── SRR389222_sub1_trimmed_fastqc.html
    │   ├── SRR389222_sub1_trimmed_fastqc.zip
    │   ├── SRR389222_sub2_trimmed_fastqc.html
    │   ├── SRR389222_sub2_trimmed_fastqc.zip
    │   ├── SRR389222_sub3_trimmed_fastqc.html
    │   └── SRR389222_sub3_trimmed_fastqc.zip
    ├── logs
    │   ├── SRR389222_sub1.fastq.gz_trimming_report.txt
    │   ├── SRR389222_sub2.fastq.gz_trimming_report.txt
    │   └── SRR389222_sub3.fastq.gz_trimming_report.txt
    └── where_are_my_files.txt

37 directories, 302 files

dev file tree

gitpod /workspace/methylseq/test_dev/test_bismark (master) $ tree results/
results/
├── bismark
│   ├── alignments
│   │   └── logs
│   │       ├── SRR389222_sub1_T1_trimmed_bismark_bt2_SE_report.txt
│   │       ├── SRR389222_sub2_T1_trimmed_bismark_bt2_SE_report.txt
│   │       └── SRR389222_sub3_T1_trimmed_bismark_bt2_SE_report.txt
│   ├── deduplicated
│   │   ├── logs
│   │   │   ├── SRR389222_sub1_T1_trimmed_bismark_bt2.deduplication_report.txt
│   │   │   ├── SRR389222_sub2_T1_trimmed_bismark_bt2.deduplication_report.txt
│   │   │   └── SRR389222_sub3_T1_trimmed_bismark_bt2.deduplication_report.txt
│   │   ├── SRR389222_sub1_T1_trimmed_bismark_bt2.deduplicated.bam
│   │   ├── SRR389222_sub2_T1_trimmed_bismark_bt2.deduplicated.bam
│   │   └── SRR389222_sub3_T1_trimmed_bismark_bt2.deduplicated.bam
│   ├── methylation_calls
│   │   ├── bedGraph
│   │   │   ├── SRR389222_sub1_T1_trimmed_bismark_bt2.deduplicated.bedGraph.gz
│   │   │   ├── SRR389222_sub2_T1_trimmed_bismark_bt2.deduplicated.bedGraph.gz
│   │   │   └── SRR389222_sub3_T1_trimmed_bismark_bt2.deduplicated.bedGraph.gz
│   │   ├── mbias
│   │   │   ├── SRR389222_sub1_T1_trimmed_bismark_bt2.deduplicated.M-bias.txt
│   │   │   ├── SRR389222_sub2_T1_trimmed_bismark_bt2.deduplicated.M-bias.txt
│   │   │   └── SRR389222_sub3_T1_trimmed_bismark_bt2.deduplicated.M-bias.txt
│   │   ├── methylation_calls
│   │   │   ├── CHG_OB_SRR389222_sub1_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CHG_OB_SRR389222_sub2_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CHG_OB_SRR389222_sub3_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CHG_OT_SRR389222_sub1_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CHG_OT_SRR389222_sub2_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CHG_OT_SRR389222_sub3_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CHH_OB_SRR389222_sub1_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CHH_OB_SRR389222_sub2_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CHH_OB_SRR389222_sub3_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CHH_OT_SRR389222_sub1_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CHH_OT_SRR389222_sub2_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CHH_OT_SRR389222_sub3_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CpG_OB_SRR389222_sub1_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CpG_OB_SRR389222_sub2_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CpG_OB_SRR389222_sub3_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CpG_OT_SRR389222_sub1_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   ├── CpG_OT_SRR389222_sub2_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   │   └── CpG_OT_SRR389222_sub3_T1_trimmed_bismark_bt2.deduplicated.txt.gz
│   │   ├── methylation_coverage
│   │   │   ├── SRR389222_sub1_T1_trimmed_bismark_bt2.deduplicated.bismark.cov.gz
│   │   │   ├── SRR389222_sub2_T1_trimmed_bismark_bt2.deduplicated.bismark.cov.gz
│   │   │   └── SRR389222_sub3_T1_trimmed_bismark_bt2.deduplicated.bismark.cov.gz
│   │   ├── splitting_report
│   │   │   ├── SRR389222_sub1_T1_trimmed_bismark_bt2.deduplicated_splitting_report.txt
│   │   │   ├── SRR389222_sub2_T1_trimmed_bismark_bt2.deduplicated_splitting_report.txt
│   │   │   └── SRR389222_sub3_T1_trimmed_bismark_bt2.deduplicated_splitting_report.txt
│   │   └── stranded_CpG_report
│   ├── reports
│   │   ├── SRR389222_sub1_T1_trimmed_bismark_bt2_SE_report.html
│   │   ├── SRR389222_sub2_T1_trimmed_bismark_bt2_SE_report.html
│   │   └── SRR389222_sub3_T1_trimmed_bismark_bt2_SE_report.html
│   └── summary
│       ├── bismark_summary_report.html
│       └── bismark_summary_report.txt
├── fastqc
│   ├── SRR389222_sub1_T1_SRR389222_sub1_fastqc.html
│   ├── SRR389222_sub2_T1_SRR389222_sub2_fastqc.html
│   ├── SRR389222_sub3_T1_SRR389222_sub3_fastqc.html
│   └── zips
│       ├── SRR389222_sub1_T1_SRR389222_sub1_fastqc.zip
│       ├── SRR389222_sub2_T1_SRR389222_sub2_fastqc.zip
│       └── SRR389222_sub3_T1_SRR389222_sub3_fastqc.zip
├── multiqc
│   └── bismark
│       ├── multiqc_data
│       │   ├── mqc_bismark_alignment_1.txt
│       │   ├── mqc_bismark_deduplication_1.txt
│       │   ├── mqc_bismark_mbias_CHG_R1.txt
│       │   ├── mqc_bismark_mbias_CHH_R1.txt
│       │   ├── mqc_bismark_mbias_CpG_R1.txt
│       │   ├── mqc_bismark_strand_alignment_1.txt
│       │   ├── mqc_cutadapt_filtered_reads_plot_1.txt
│       │   ├── mqc_cutadapt_trimmed_sequences_plot_3_Counts.txt
│       │   ├── mqc_cutadapt_trimmed_sequences_plot_3_Obs_Exp.txt
│       │   ├── mqc_fastqc_overrepresented_sequences_plot_1.txt
│       │   ├── mqc_fastqc_per_base_n_content_plot_1.txt
│       │   ├── mqc_fastqc_per_base_sequence_quality_plot_1.txt
│       │   ├── mqc_fastqc_per_sequence_gc_content_plot_Counts.txt
│       │   ├── mqc_fastqc_per_sequence_gc_content_plot_Percentages.txt
│       │   ├── mqc_fastqc_per_sequence_quality_scores_plot_1.txt
│       │   ├── mqc_fastqc_sequence_counts_plot_1.txt
│       │   ├── mqc_fastqc_sequence_duplication_levels_plot_1.txt
│       │   ├── mqc_qualimap_coverage_histogram_1.txt
│       │   ├── mqc_qualimap_gc_content_1.txt
│       │   ├── mqc_qualimap_genome_fraction_1.txt
│       │   ├── multiqc_bismark_alignment.txt
│       │   ├── multiqc_bismark_dedup.txt
│       │   ├── multiqc_bismark_methextract.txt
│       │   ├── multiqc_citations.txt
│       │   ├── multiqc_cutadapt.txt
│       │   ├── multiqc_data.json
│       │   ├── multiqc_fastqc.txt
│       │   ├── multiqc_general_stats.txt
│       │   ├── multiqc.log
│       │   ├── multiqc_qualimap_bamqc_genome_results.txt
│       │   └── multiqc_sources.txt
│       ├── multiqc_plots
│       │   ├── pdf
│       │   │   ├── mqc_bismark_alignment_1_pc.pdf
│       │   │   ├── mqc_bismark_alignment_1.pdf
│       │   │   ├── mqc_bismark_deduplication_1_pc.pdf
│       │   │   ├── mqc_bismark_deduplication_1.pdf
│       │   │   ├── mqc_bismark_mbias_CHG_R1.pdf
│       │   │   ├── mqc_bismark_mbias_CHH_R1.pdf
│       │   │   ├── mqc_bismark_mbias_CpG_R1.pdf
│       │   │   ├── mqc_bismark_strand_alignment_1_pc.pdf
│       │   │   ├── mqc_bismark_strand_alignment_1.pdf
│       │   │   ├── mqc_cutadapt_filtered_reads_plot_1_pc.pdf
│       │   │   ├── mqc_cutadapt_filtered_reads_plot_1.pdf
│       │   │   ├── mqc_cutadapt_trimmed_sequences_plot_3_Counts.pdf
│       │   │   ├── mqc_cutadapt_trimmed_sequences_plot_3_Obs_Exp.pdf
│       │   │   ├── mqc_fastqc_overrepresented_sequences_plot_1.pdf
│       │   │   ├── mqc_fastqc_per_base_n_content_plot_1.pdf
│       │   │   ├── mqc_fastqc_per_base_sequence_quality_plot_1.pdf
│       │   │   ├── mqc_fastqc_per_sequence_gc_content_plot_Counts.pdf
│       │   │   ├── mqc_fastqc_per_sequence_gc_content_plot_Percentages.pdf
│       │   │   ├── mqc_fastqc_per_sequence_quality_scores_plot_1.pdf
│       │   │   ├── mqc_fastqc_sequence_counts_plot_1_pc.pdf
│       │   │   ├── mqc_fastqc_sequence_counts_plot_1.pdf
│       │   │   ├── mqc_fastqc_sequence_duplication_levels_plot_1.pdf
│       │   │   ├── mqc_qualimap_coverage_histogram_1.pdf
│       │   │   ├── mqc_qualimap_gc_content_1.pdf
│       │   │   └── mqc_qualimap_genome_fraction_1.pdf
│       │   ├── png
│       │   │   ├── mqc_bismark_alignment_1_pc.png
│       │   │   ├── mqc_bismark_alignment_1.png
│       │   │   ├── mqc_bismark_deduplication_1_pc.png
│       │   │   ├── mqc_bismark_deduplication_1.png
│       │   │   ├── mqc_bismark_mbias_CHG_R1.png
│       │   │   ├── mqc_bismark_mbias_CHH_R1.png
│       │   │   ├── mqc_bismark_mbias_CpG_R1.png
│       │   │   ├── mqc_bismark_strand_alignment_1_pc.png
│       │   │   ├── mqc_bismark_strand_alignment_1.png
│       │   │   ├── mqc_cutadapt_filtered_reads_plot_1_pc.png
│       │   │   ├── mqc_cutadapt_filtered_reads_plot_1.png
│       │   │   ├── mqc_cutadapt_trimmed_sequences_plot_3_Counts.png
│       │   │   ├── mqc_cutadapt_trimmed_sequences_plot_3_Obs_Exp.png
│       │   │   ├── mqc_fastqc_overrepresented_sequences_plot_1.png
│       │   │   ├── mqc_fastqc_per_base_n_content_plot_1.png
│       │   │   ├── mqc_fastqc_per_base_sequence_quality_plot_1.png
│       │   │   ├── mqc_fastqc_per_sequence_gc_content_plot_Counts.png
│       │   │   ├── mqc_fastqc_per_sequence_gc_content_plot_Percentages.png
│       │   │   ├── mqc_fastqc_per_sequence_quality_scores_plot_1.png
│       │   │   ├── mqc_fastqc_sequence_counts_plot_1_pc.png
│       │   │   ├── mqc_fastqc_sequence_counts_plot_1.png
│       │   │   ├── mqc_fastqc_sequence_duplication_levels_plot_1.png
│       │   │   ├── mqc_qualimap_coverage_histogram_1.png
│       │   │   ├── mqc_qualimap_gc_content_1.png
│       │   │   └── mqc_qualimap_genome_fraction_1.png
│       │   └── svg
│       │       ├── mqc_bismark_alignment_1_pc.svg
│       │       ├── mqc_bismark_alignment_1.svg
│       │       ├── mqc_bismark_deduplication_1_pc.svg
│       │       ├── mqc_bismark_deduplication_1.svg
│       │       ├── mqc_bismark_mbias_CHG_R1.svg
│       │       ├── mqc_bismark_mbias_CHH_R1.svg
│       │       ├── mqc_bismark_mbias_CpG_R1.svg
│       │       ├── mqc_bismark_strand_alignment_1_pc.svg
│       │       ├── mqc_bismark_strand_alignment_1.svg
│       │       ├── mqc_cutadapt_filtered_reads_plot_1_pc.svg
│       │       ├── mqc_cutadapt_filtered_reads_plot_1.svg
│       │       ├── mqc_cutadapt_trimmed_sequences_plot_3_Counts.svg
│       │       ├── mqc_cutadapt_trimmed_sequences_plot_3_Obs_Exp.svg
│       │       ├── mqc_fastqc_overrepresented_sequences_plot_1.svg
│       │       ├── mqc_fastqc_per_base_n_content_plot_1.svg
│       │       ├── mqc_fastqc_per_base_sequence_quality_plot_1.svg
│       │       ├── mqc_fastqc_per_sequence_gc_content_plot_Counts.svg
│       │       ├── mqc_fastqc_per_sequence_gc_content_plot_Percentages.svg
│       │       ├── mqc_fastqc_per_sequence_quality_scores_plot_1.svg
│       │       ├── mqc_fastqc_sequence_counts_plot_1_pc.svg
│       │       ├── mqc_fastqc_sequence_counts_plot_1.svg
│       │       ├── mqc_fastqc_sequence_duplication_levels_plot_1.svg
│       │       ├── mqc_qualimap_coverage_histogram_1.svg
│       │       ├── mqc_qualimap_gc_content_1.svg
│       │       └── mqc_qualimap_genome_fraction_1.svg
│       └── multiqc_report.html
├── pipeline_info
│   ├── execution_report_2022-11-04_09-47-39.html
│   ├── execution_timeline_2022-11-04_09-47-39.html
│   ├── execution_trace_2022-11-04_09-47-39.txt
│   ├── pipeline_dag_2022-11-04_09-47-39.html
│   ├── samplesheet.valid.csv
│   └── software_versions.yml
├── qualimap
│   ├── SRR389222_sub1_T1
│   │   ├── css
│   │   │   ├── agogo.css
│   │   │   ├── ajax-loader.gif
│   │   │   ├── basic.css
│   │   │   ├── bgfooter.png
│   │   │   ├── bgtop.png
│   │   │   ├── comment-bright.png
│   │   │   ├── comment-close.png
│   │   │   ├── comment.png
│   │   │   ├── doctools.js
│   │   │   ├── down.png
│   │   │   ├── down-pressed.png
│   │   │   ├── file.png
│   │   │   ├── jquery.js
│   │   │   ├── minus.png
│   │   │   ├── plus.png
│   │   │   ├── pygments.css
│   │   │   ├── qualimap_logo_small.png
│   │   │   ├── report.css
│   │   │   ├── searchtools.js
│   │   │   ├── underscore.js
│   │   │   ├── up.png
│   │   │   ├── up-pressed.png
│   │   │   └── websupport.js
│   │   ├── genome_results.txt
│   │   ├── images_qualimapReport
│   │   │   ├── genome_coverage_0to50_histogram.png
│   │   │   ├── genome_coverage_across_reference.png
│   │   │   ├── genome_coverage_histogram.png
│   │   │   ├── genome_coverage_quotes.png
│   │   │   ├── genome_gc_content_per_window.png
│   │   │   ├── genome_mapping_quality_across_reference.png
│   │   │   ├── genome_mapping_quality_histogram.png
│   │   │   ├── genome_reads_content_per_read_position.png
│   │   │   └── genome_uniq_read_starts_histogram.png
│   │   ├── qualimapReport.html
│   │   └── raw_data_qualimapReport
│   │       ├── coverage_across_reference.txt
│   │       ├── coverage_histogram.txt
│   │       ├── duplication_rate_histogram.txt
│   │       ├── genome_fraction_coverage.txt
│   │       ├── mapped_reads_gc-content_distribution.txt
│   │       ├── mapped_reads_nucleotide_content.txt
│   │       ├── mapping_quality_across_reference.txt
│   │       └── mapping_quality_histogram.txt
│   ├── SRR389222_sub2_T1
│   │   ├── css
│   │   │   ├── agogo.css
│   │   │   ├── ajax-loader.gif
│   │   │   ├── basic.css
│   │   │   ├── bgfooter.png
│   │   │   ├── bgtop.png
│   │   │   ├── comment-bright.png
│   │   │   ├── comment-close.png
│   │   │   ├── comment.png
│   │   │   ├── doctools.js
│   │   │   ├── down.png
│   │   │   ├── down-pressed.png
│   │   │   ├── file.png
│   │   │   ├── jquery.js
│   │   │   ├── minus.png
│   │   │   ├── plus.png
│   │   │   ├── pygments.css
│   │   │   ├── qualimap_logo_small.png
│   │   │   ├── report.css
│   │   │   ├── searchtools.js
│   │   │   ├── underscore.js
│   │   │   ├── up.png
│   │   │   ├── up-pressed.png
│   │   │   └── websupport.js
│   │   ├── genome_results.txt
│   │   ├── images_qualimapReport
│   │   │   ├── genome_coverage_0to50_histogram.png
│   │   │   ├── genome_coverage_across_reference.png
│   │   │   ├── genome_coverage_histogram.png
│   │   │   ├── genome_coverage_quotes.png
│   │   │   ├── genome_gc_content_per_window.png
│   │   │   ├── genome_mapping_quality_across_reference.png
│   │   │   ├── genome_mapping_quality_histogram.png
│   │   │   ├── genome_reads_content_per_read_position.png
│   │   │   └── genome_uniq_read_starts_histogram.png
│   │   ├── qualimapReport.html
│   │   └── raw_data_qualimapReport
│   │       ├── coverage_across_reference.txt
│   │       ├── coverage_histogram.txt
│   │       ├── duplication_rate_histogram.txt
│   │       ├── genome_fraction_coverage.txt
│   │       ├── mapped_reads_gc-content_distribution.txt
│   │       ├── mapped_reads_nucleotide_content.txt
│   │       ├── mapping_quality_across_reference.txt
│   │       └── mapping_quality_histogram.txt
│   └── SRR389222_sub3_T1
│       ├── css
│       │   ├── agogo.css
│       │   ├── ajax-loader.gif
│       │   ├── basic.css
│       │   ├── bgfooter.png
│       │   ├── bgtop.png
│       │   ├── comment-bright.png
│       │   ├── comment-close.png
│       │   ├── comment.png
│       │   ├── doctools.js
│       │   ├── down.png
│       │   ├── down-pressed.png
│       │   ├── file.png
│       │   ├── jquery.js
│       │   ├── minus.png
│       │   ├── plus.png
│       │   ├── pygments.css
│       │   ├── qualimap_logo_small.png
│       │   ├── report.css
│       │   ├── searchtools.js
│       │   ├── underscore.js
│       │   ├── up.png
│       │   ├── up-pressed.png
│       │   └── websupport.js
│       ├── genome_results.txt
│       ├── images_qualimapReport
│       │   ├── genome_coverage_0to50_histogram.png
│       │   ├── genome_coverage_across_reference.png
│       │   ├── genome_coverage_histogram.png
│       │   ├── genome_coverage_quotes.png
│       │   ├── genome_gc_content_per_window.png
│       │   ├── genome_mapping_quality_across_reference.png
│       │   ├── genome_mapping_quality_histogram.png
│       │   ├── genome_reads_content_per_read_position.png
│       │   └── genome_uniq_read_starts_histogram.png
│       ├── qualimapReport.html
│       └── raw_data_qualimapReport
│           ├── coverage_across_reference.txt
│           ├── coverage_histogram.txt
│           ├── duplication_rate_histogram.txt
│           ├── genome_fraction_coverage.txt
│           ├── mapped_reads_gc-content_distribution.txt
│           ├── mapped_reads_nucleotide_content.txt
│           ├── mapping_quality_across_reference.txt
│           └── mapping_quality_histogram.txt
└── trimgalore
    ├── fastqc
    │   ├── SRR389222_sub1_T1_trimmed_fastqc.html
    │   ├── SRR389222_sub2_T1_trimmed_fastqc.html
    │   ├── SRR389222_sub3_T1_trimmed_fastqc.html
    │   └── zips
    │       ├── SRR389222_sub1_T1_trimmed_fastqc.zip
    │       ├── SRR389222_sub2_T1_trimmed_fastqc.zip
    │       └── SRR389222_sub3_T1_trimmed_fastqc.zip
    └── logs
        ├── SRR389222_sub1_T1.fastq.gz_trimming_report.txt
        ├── SRR389222_sub2_T1.fastq.gz_trimming_report.txt
        └── SRR389222_sub3_T1.fastq.gz_trimming_report.txt

41 directories, 298 files

Nov 04 '22 10:11 ewels

I'm starting to think that comparing file outputs between versions is going to be basically impossible to do in a global manner. I think that we will have to just eyeball them this time and try to visually check that everything looks roughly correct..

Nov 04 '22 10:11 ewels

Done some manual inspections of the MultiQC reports from test runs and it looks good. I haven't done the in-depth analysis between full test runs that I was hoping for, but it started being way too much work. Hopefully going forward this will be much easier with the proper pipeline-level CI tests that we are now building.

Nov 08 '22 15:11 ewels

DSL2 dev merge checklist

Profile: test

Profile: test_full

Testing on GitPod with -profile test

Bismark

Profile: `test`

Profile: `test_full`

Testing on GitPod with `-profile test`