the chipseq pipeline runs forever even when errors were reported

[gudeqing]

Hi, I found the chipseq pipeline will run forever even when error reported!

The command used to launch the workflow was as follows:

 nextflow run nf-core/chipseq --input design.csv --fasta /testdata/genome.fa --gtf /testdata/genes.gtf -qs 5 --macs_gsize 2.7e9 -offline --outdir /enigma/local_storage/ -resume

Pipeline Configuration:

• Run Name: cheeky_cray
• Data Type: Paired-End
• Design File: design.csv
• Genome: Not supplied
• Fasta File: /testdata/genome.fa
• GTF File: /testdata/genes.gtf
• MACS2 Genome Size: 2.7e9
• Min Consensus Reps: 1
• MACS2 Narrow Peaks: No
• MACS2 Broad Cutoff: 0.1
• Trim R1: 0 bp
• Trim R2: 0 bp
• Trim 3' R1: 0 bp
• Trim 3' R2: 0 bp
• NextSeq Trim: 0 bp
• Fingerprint Bins: 500000
• Save Genome Index: No
• Max Resources: 128 GB memory, 16 cpus, 10d time per job
• Output Dir: /enigma/local_storage/
• Launch Dir: /enigma/local_storage
• Working Dir: /enigma/local_storage/work
• Script Dir: /root/.nextflow/assets/nf-core/chipseq
• User: root
• Config Profile: standard
• Date Started: 2022-08-26T07:40:32.843487Z
• Date Completed: 2022-08-26T08:17:04.528705Z
• Pipeline script file path: /root/.nextflow/assets/nf-core/chipseq/main.nf
• Pipeline script hash ID: 39c073c68caf34f6f725e82907fff2bd
• Pipeline repository Git URL: https://github.com/nf-core/chipseq
• Pipeline repository Git Commit: 6924b669422215f9021144b251e83fc9929be1fe
• Pipeline Git branch/tag: master
• Nextflow Version: 21.10.6
• Nextflow Build: 5660
• Nextflow Compile Timestamp: 21-12-2021 16:55 UTC

running status:

tail of the ,nexflow.log： Aug-26 07:40:35.833 [main] DEBUG nextflow.Session - Session await > all barriers passed Aug-26 07:40:35.833 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/mqc_spp_nsc_plot.png Aug-26 07:40:35.833 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/mqc_picard_insert_size_plot.png Aug-26 07:40:35.833 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/mqc_deseq2_sample_similarity_plot.png Aug-26 07:40:35.833 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/nf-core-chipseq_logo.png Aug-26 07:40:35.834 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/mqc_macs2_peak_count_plot.png Aug-26 07:40:35.834 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/mqc_samtools_stats_plot.png Aug-26 07:40:35.834 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/mqc_picard_deduplication_plot.png Aug-26 07:40:35.834 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/mqc_deseq2_pca_plot.png Aug-26 07:40:35.835 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/r_deseq2_ma_plot.png Aug-26 07:40:35.835 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/mqc_cutadapt_plot.png Aug-26 07:40:35.835 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/r_upsetr_intersect_plot.png Aug-26 07:40:35.835 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/igv_screenshot.png Aug-26 07:40:35.836 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/mqc_preseq_plot.png Aug-26 07:40:35.836 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/mqc_deeptools_plotProfile_plot.png Aug-26 07:40:35.836 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/mqc_spp_rsc_plot.png Aug-26 07:40:35.849 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/mqc_deeptools_plotFingerprint_plot.png Aug-26 07:40:35.850 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/docs/images/r_deseq2_volcano_plot.png Aug-26 07:40:35.851 [Actor Thread 4] DEBUG nextflow.util.CacheHelper - Hash asset file sha-256: /root/.nextflow/assets/nf-core/chipseq/bin/markdown_to_html.py Aug-26 07:40:35.920 [Actor Thread 13] DEBUG nextflow.sort.BigSort - Sort completed -- entries: 1; slices: 1; internal sort time: 0.001 s; external sort time: 0.012 s; total time: 0.013 s Aug-26 07:40:35.931 [Actor Thread 13] DEBUG nextflow.file.FileCollector - Saved collect-files list to: /tmp/eaf0fedf6e6e51441522aa82438c6ceb.collect-file Aug-26 07:40:35.938 [Actor Thread 13] DEBUG nextflow.file.FileCollector - Deleting file collector temp dir: /tmp/nxf-5332452955947821456 Aug-26 07:45:35.811 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 5 -- tasks to be submitted are shown below ~> TaskHandler[id: 1; name: CHECK_DESIGN (design.csv); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/f7/5558e2114cac76f90c9bcf645f0c3d] ~> TaskHandler[id: 3; name: MAKE_GENE_BED (genes.gtf); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/17/47fa87ef9ca0eb84430ba45fa044a0] ~> TaskHandler[id: 4; name: MAKE_GENOME_FILTER (genome.fa); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/e0/ed57245c94a8c32a708c64395e9fc1] ~> TaskHandler[id: 5; name: get_software_versions; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/70/408ad1bfd0414bb74be618bfc9d71c] ~> TaskHandler[id: 6; name: output_documentation; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/b8/e098c250e24fd2563f6b587bf1e4e9] Aug-26 07:50:35.955 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 5 -- tasks to be submitted are shown below ~> TaskHandler[id: 1; name: CHECK_DESIGN (design.csv); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/f7/5558e2114cac76f90c9bcf645f0c3d] ~> TaskHandler[id: 3; name: MAKE_GENE_BED (genes.gtf); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/17/47fa87ef9ca0eb84430ba45fa044a0] ~> TaskHandler[id: 4; name: MAKE_GENOME_FILTER (genome.fa); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/e0/ed57245c94a8c32a708c64395e9fc1] ~> TaskHandler[id: 5; name: get_software_versions; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/70/408ad1bfd0414bb74be618bfc9d71c] ~> TaskHandler[id: 6; name: output_documentation; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/b8/e098c250e24fd2563f6b587bf1e4e9] Aug-26 07:55:36.087 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 5 -- tasks to be submitted are shown below ~> TaskHandler[id: 1; name: CHECK_DESIGN (design.csv); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/f7/5558e2114cac76f90c9bcf645f0c3d] ~> TaskHandler[id: 3; name: MAKE_GENE_BED (genes.gtf); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/17/47fa87ef9ca0eb84430ba45fa044a0] ~> TaskHandler[id: 4; name: MAKE_GENOME_FILTER (genome.fa); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/e0/ed57245c94a8c32a708c64395e9fc1] ~> TaskHandler[id: 5; name: get_software_versions; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/70/408ad1bfd0414bb74be618bfc9d71c] ~> TaskHandler[id: 6; name: output_documentation; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/b8/e098c250e24fd2563f6b587bf1e4e9] Aug-26 08:00:36.221 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 5 -- tasks to be submitted are shown below ~> TaskHandler[id: 1; name: CHECK_DESIGN (design.csv); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/f7/5558e2114cac76f90c9bcf645f0c3d] ~> TaskHandler[id: 3; name: MAKE_GENE_BED (genes.gtf); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/17/47fa87ef9ca0eb84430ba45fa044a0] ~> TaskHandler[id: 4; name: MAKE_GENOME_FILTER (genome.fa); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/e0/ed57245c94a8c32a708c64395e9fc1] ~> TaskHandler[id: 5; name: get_software_versions; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/70/408ad1bfd0414bb74be618bfc9d71c] ~> TaskHandler[id: 6; name: output_documentation; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/b8/e098c250e24fd2563f6b587bf1e4e9] Aug-26 08:05:36.352 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 5 -- tasks to be submitted are shown below ~> TaskHandler[id: 1; name: CHECK_DESIGN (design.csv); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/f7/5558e2114cac76f90c9bcf645f0c3d] ~> TaskHandler[id: 3; name: MAKE_GENE_BED (genes.gtf); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/17/47fa87ef9ca0eb84430ba45fa044a0] ~> TaskHandler[id: 4; name: MAKE_GENOME_FILTER (genome.fa); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/e0/ed57245c94a8c32a708c64395e9fc1] ~> TaskHandler[id: 5; name: get_software_versions; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/70/408ad1bfd0414bb74be618bfc9d71c] ~> TaskHandler[id: 6; name: output_documentation; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/b8/e098c250e24fd2563f6b587bf1e4e9] Aug-26 08:10:36.484 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 5 -- tasks to be submitted are shown below ~> TaskHandler[id: 1; name: CHECK_DESIGN (design.csv); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/f7/5558e2114cac76f90c9bcf645f0c3d] ~> TaskHandler[id: 3; name: MAKE_GENE_BED (genes.gtf); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/17/47fa87ef9ca0eb84430ba45fa044a0] ~> TaskHandler[id: 4; name: MAKE_GENOME_FILTER (genome.fa); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/e0/ed57245c94a8c32a708c64395e9fc1] ~> TaskHandler[id: 5; name: get_software_versions; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/70/408ad1bfd0414bb74be618bfc9d71c] ~> TaskHandler[id: 6; name: output_documentation; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/b8/e098c250e24fd2563f6b587bf1e4e9] Aug-26 08:15:36.612 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 5 -- tasks to be submitted are shown below ~> TaskHandler[id: 1; name: CHECK_DESIGN (design.csv); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/f7/5558e2114cac76f90c9bcf645f0c3d] ~> TaskHandler[id: 3; name: MAKE_GENE_BED (genes.gtf); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/17/47fa87ef9ca0eb84430ba45fa044a0] ~> TaskHandler[id: 4; name: MAKE_GENOME_FILTER (genome.fa); status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/e0/ed57245c94a8c32a708c64395e9fc1] ~> TaskHandler[id: 5; name: get_software_versions; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/70/408ad1bfd0414bb74be618bfc9d71c] ~> TaskHandler[id: 6; name: output_documentation; status: NEW; exit: -; error: -; workDir: /enigma/local_storage/work/b8/e098c250e24fd2563f6b587bf1e4e9]

[JoseEspinosa]

Hi @gudeqing, thanks for reporting Could it be that there are still running jobs, Nextflow tries to safely finish all the jobs before stopping. Otherwise, I can not figure out the reason, maybe @drpatelh has another better guess 🤔