sarek Error when write a file running Mutect2 with Sarek

Description of the bug

I'm running Sarek v3.2.3 on an HPC with Slurm as management system. When Mutect2 is running, execution stops due to an error when creating a file. I have made several tests:

Running the same Sarek pipeline with the same version but in a different directory and with different samples, Mutect2 terminates successfully. I have checked the quality of the bams files and it is correct.
Running another Sarek version (v 3.2.1) the same error comes up.

This two facts makes me suspect it could be a space or permissions issue, but I'm not quite clear. I leave the error here in case you can help me to clarify it. Thank you very much.

Command used and terminal output

nextflow run nf-core/sarek -r 3.2.3 --input ${SAMPLESHEET} --wes -profile singularity --max_cpus 24 --intervals ${INTERVALS} --outdir $OUTDIR --step variant_calling --tools mutect2 --email [email protected]

Execution cancelled -- Finishing pending tasks before exit
ERROR ~ Error executing process > 'NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:MUTECT2_PAIRED (B1R2_vs_G1)'

Caused by:
  Process `NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:MUTECT2_PAIRED (B1R2_vs_G1)` terminated with an error exit status (1)

Command executed:

  gatk --java-options "-Xmx29491M" Mutect2 \
      --input G1-lane_1.converted.cram --input B1R2-lane_1.converted.cram \
      --output B1R2_vs_G1.mutect2.chr9_66591388-67920552.vcf.gz \
      --reference Homo_sapiens_assembly38.fasta \
      --panel-of-normals 1000g_pon.hg38.vcf.gz \
      --germline-resource af-only-gnomad.hg38.vcf.gz \
      --intervals chr9_66591388-67920552.bed \
      --tmp-dir . \
      --f1r2-tar-gz B1R2_vs_G1.mutect2.chr9_66591388-67920552.f1r2.tar.gz --normal-sample P01_G1
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:MUTECT2_PAIRED":
      gatk4: $(echo $(gatk --version 2>&1) | sed 's/^.*(GATK) v//; s/ .*$//')
  END_VERSIONS

Command exit status:
  1

Command output:
  [0.001s][warning][os,container] Duplicate cpuset controllers detected. Picking /sys/fs/cgroup/cpuset, skipping /mnt/netapp1/Optcesga_FT2_RHEL7/2020/gentoo/22072020/var/singularity/mnt/session/final/sys/fs/cgroup/cpuset.

Command error:
  23:07:43.140 INFO  ProgressMeter -       chr9:133405690             60.2                437880           7274.5
  23:07:53.502 INFO  ProgressMeter -       chr9:133540799             60.4                438710           7267.4
  23:08:03.594 INFO  ProgressMeter -       chr9:133725565             60.5                439840           7265.9
  23:08:13.701 INFO  ProgressMeter -       chr9:133925123             60.7                441090           7266.3
  23:08:23.706 INFO  ProgressMeter -       chr9:134088694             60.9                442150           7263.8
  23:08:32.545 INFO  Mutect2 - 245998 read(s) filtered by: MappingQualityReadFilter 
  0 read(s) filtered by: MappingQualityAvailableReadFilter 
  0 read(s) filtered by: MappingQualityNotZeroReadFilter 
  0 read(s) filtered by: MappedReadFilter 
  0 read(s) filtered by: NotSecondaryAlignmentReadFilter 
  1900483 read(s) filtered by: NotDuplicateReadFilter 
  0 read(s) filtered by: PassesVendorQualityCheckReadFilter 
  0 read(s) filtered by: NonChimericOriginalAlignmentReadFilter 
  0 read(s) filtered by: NonZeroReferenceLengthAlignmentReadFilter 
  47 read(s) filtered by: ReadLengthReadFilter 
  0 read(s) filtered by: GoodCigarReadFilter 
  0 read(s) filtered by: WellformedReadFilter 
  2146528 total reads filtered out of 8009276 reads processed
  23:08:32.545 INFO  ProgressMeter -       chr9:134182911             61.0                442750           7256.1
  23:08:32.545 INFO  ProgressMeter - Traversal complete. Processed 442750 total regions in 61.0 minutes.
  23:08:32.553 INFO  VectorLoglessPairHMM - Time spent in setup for JNI call : 1.1748570740000002
  23:08:32.553 INFO  PairHMM - Total compute time in PairHMM computeLogLikelihoods() : 154.751703055
  23:08:32.553 INFO  SmithWatermanAligner - Total compute time in java Smith-Waterman : 2227.11 sec
  23:08:32.554 INFO  Mutect2 - Shutting down engine
  [September 11, 2023 at 11:08:32 PM GMT] org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2 done. Elapsed time: 61.03 minutes.
  Runtime.totalMemory()=3238002688
  htsjdk.samtools.SAMException: Could not write metrics to file: /tmp/untarred10106267742814969916/P01_B1R2.ref_histogram
  	at htsjdk.samtools.metrics.MetricsFile.write(MetricsFile.java:138)
  	at org.broadinstitute.hellbender.tools.walkers.readorientation.F1R2CountsCollector.writeHistograms(F1R2CountsCollector.java:164)
  	at org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2Engine.lambda$writeExtraOutputs$13(Mutect2Engine.java:435)
  	at java.base/java.util.Optional.ifPresent(Optional.java:178)
  	at org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2Engine.writeExtraOutputs(Mutect2Engine.java:434)
  	at org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2.onTraversalSuccess(Mutect2.java:297)
  	at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1102)
  	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:149)
  	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:198)
  	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:217)
  	at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
  	at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
  	at org.broadinstitute.hellbender.Main.main(Main.java:289)
  Caused by: java.io.FileNotFoundException: /tmp/untarred10106267742814969916/P01_B1R2.ref_histogram (No such file or directory)
  	at java.base/java.io.FileOutputStream.open0(Native Method)
  	at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293)
  	at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:235)
  	at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:184)
  	at java.base/java.io.FileWriter.<init>(FileWriter.java:96)
  	at htsjdk.samtools.metrics.MetricsFile.write(MetricsFile.java:135)
  	... 12 more
  .command.run: line 155: kill: (33) - No such process
  INFO:    Cleaning up image...

Work dir:
  /mnt/lustre/scratch/nlsas/home/usc/mg/translational_oncology/1_tools/01_nf_core_sarek/1_src/b1r2_job/work/2a/832da1404e27cee67fe5d89ea2ffe5

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details

Relevant files

No response

System information

Nextflow version: 23.04.2
Hardware: HPC Finisterrae III
Executor: slurm
Container engine: Singularity v3.9.7
OS: Linux
Version os nf-core/sarek: 3.2.3 and tested with 3.2.1 too

Sep 12 '23 10:09 mimifp

Many things concerning the mutect2-workflow has been improved in the upcoming release of Sarek. If possible, would you mind running your nf-cmd on the dev-branch. It might be that your problem has been solved on the dev-branch.

Sep 12 '23 10:09 asp8200

Ok, thanks for you response. Could you tell me how I can run the dev-branch version through Singularity on the HPC? I have never done it.

Sep 12 '23 14:09 mimifp

replace -r 3.2.3 with -r dev :)

Sep 12 '23 14:09 FriederikeHanssen

Ok, I'm trying. As soon as the execution has finished I will comment the results. Thanks.

Sep 13 '23 08:09 mimifp

Hi! The execution of dev version also gives an similar error.

`Error executing process > 'NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:MUTECT2_PAIRED (B1R3_vs_G1)'

Caused by:
  Process `NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:MUTECT2_PAIRED (B1R3_vs_G1)` terminated with an error exit status (1)

Command executed:

  gatk --java-options "-Xmx29491M" Mutect2 \
      --input G1-lane_1.converted.cram --input B1R3-lane_1.converted.cram \
      --output B1R3_vs_G1.mutect2.chr3_91256422-91257890.vcf.gz \
      --reference Homo_sapiens_assembly38.fasta \
      --panel-of-normals 1000g_pon.hg38.vcf.gz \
      --germline-resource af-only-gnomad.hg38.vcf.gz \
      --intervals chr3_91256422-91257890.bed \
      --tmp-dir . \
      --f1r2-tar-gz B1R3_vs_G1.mutect2.chr3_91256422-91257890.f1r2.tar.gz --normal-sample P01_G1
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:MUTECT2_PAIRED":
      gatk4: $(echo $(gatk --version 2>&1) | sed 's/^.*(GATK) v//; s/ .*$//')
  END_VERSIONS

Command exit status:
  1

Command output:
  [0.001s][warning][os,container] Duplicate cpuset controllers detected. Picking /sys/fs/cgroup/cpuset, skipping /mnt/netapp1/Optcesga_FT2_RHEL7/2020/gentoo/22072020/var/singularity/mnt/session/final/sys/fs/cgroup/cpuset.

Command error:
  10:00:47.329 INFO  ProgressMeter -       chr3:197249954             64.3                650030          10113.9
  10:00:57.576 INFO  ProgressMeter -       chr3:197490780             64.4                651560          10110.9
  10:01:07.733 INFO  ProgressMeter -       chr3:197704075             64.6                652870          10104.6
  10:01:17.808 INFO  ProgressMeter -       chr3:197838684             64.8                653760          10092.2
  10:01:27.937 INFO  ProgressMeter -       chr3:197974443             64.9                654690          10080.3
  10:01:37.462 INFO  Mutect2 - 183706 read(s) filtered by: MappingQualityReadFilter 
  0 read(s) filtered by: MappingQualityAvailableReadFilter 
  0 read(s) filtered by: MappingQualityNotZeroReadFilter 
  0 read(s) filtered by: MappedReadFilter 
  0 read(s) filtered by: NotSecondaryAlignmentReadFilter 
  2326197 read(s) filtered by: NotDuplicateReadFilter 
  0 read(s) filtered by: PassesVendorQualityCheckReadFilter 
  0 read(s) filtered by: NonChimericOriginalAlignmentReadFilter 
  0 read(s) filtered by: NonZeroReferenceLengthAlignmentReadFilter 
  23 read(s) filtered by: ReadLengthReadFilter 
  0 read(s) filtered by: GoodCigarReadFilter 
  0 read(s) filtered by: WellformedReadFilter 
  2509926 total reads filtered out of 9752634 reads processed
  10:01:37.462 INFO  ProgressMeter -       chr3:198232736             65.1                656109          10077.5
  10:01:37.462 INFO  ProgressMeter - Traversal complete. Processed 656109 total regions in 65.1 minutes.
  10:01:37.510 INFO  VectorLoglessPairHMM - Time spent in setup for JNI call : 1.280755679
  10:01:37.511 INFO  PairHMM - Total compute time in PairHMM computeLogLikelihoods() : 156.71105921900002
  10:01:37.511 INFO  SmithWatermanAligner - Total compute time in java Smith-Waterman : 2124.03 sec
  10:01:37.511 INFO  Mutect2 - Shutting down engine
  [September 14, 2023 at 10:01:37 AM GMT] org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2 done. Elapsed time: 65.12 minutes.
  Runtime.totalMemory()=3875536896
  htsjdk.samtools.SAMException: Could not write metrics to file: /tmp/untarred6217691579772485266/P01_B1R3.ref_histogram
  	at htsjdk.samtools.metrics.MetricsFile.write(MetricsFile.java:138)
  	at org.broadinstitute.hellbender.tools.walkers.readorientation.F1R2CountsCollector.writeHistograms(F1R2CountsCollector.java:164)
  	at org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2Engine.lambda$writeExtraOutputs$13(Mutect2Engine.java:435)
  	at java.base/java.util.Optional.ifPresent(Optional.java:178)
  	at org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2Engine.writeExtraOutputs(Mutect2Engine.java:434)
  	at org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2.onTraversalSuccess(Mutect2.java:297)
  	at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1102)
  	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:149)
  	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:198)
  	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:217)
  	at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
  	at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
  	at org.broadinstitute.hellbender.Main.main(Main.java:289)
  Caused by: java.io.FileNotFoundException: /tmp/untarred6217691579772485266/P01_B1R3.ref_histogram (No such file or directory)
  	at java.base/java.io.FileOutputStream.open0(Native Method)
  	at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293)
  	at java.base/java.io.FileOutputStream.(FileOutputStream.java:235)
  	at java.base/java.io.FileOutputStream.(FileOutputStream.java:184)
  	at java.base/java.io.FileWriter.(FileWriter.java:96)
  	at htsjdk.samtools.metrics.MetricsFile.write(MetricsFile.java:135)
  	... 12 more
  .command.run: line 155: kill: (33) - No such process
  INFO:    Cleaning up image...

Work dir:
  /mnt/lustre/scratch/nlsas/home/usc/mg/translational_oncology/1_tools/01_nf_core_sarek/1_src/b1r3_job/work/77/7236c701f379e331f929d09e493ba9

Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run``

I'm run Mutect2 on my own and it works, so the sample is not the reason for the error neither.

Sep 14 '23 13:09 mimifp

Cross-posted it to slack: https://nfcore.slack.com/archives/CKDDD9JB0/p1695024055740019 I googled for the error, but didn't find anything too helpful. It seems to indicate some issues on the set up side. Which java version are you using?

Sep 18 '23 08:09 FriederikeHanssen

Can you trying running the pipeline and adding:

process {
    scratch = false
}

to a custom.config, and then running:

nextflow run nf-core/sarek -r 3.2.3 --input ${SAMPLESHEET} --wes -profile singularity --max_cpus 24 --intervals ${INTERVALS} --outdir $OUTDIR --step variant_calling --tools mutect2 --email [email protected] -c custom.config

Sep 18 '23 08:09 FriederikeHanssen

Hey! DId this work?

Aug 20 '24 13:08 FriederikeHanssen

sarek sarek copied to clipboard

Error when write a file running Mutect2 with Sarek

Description of the bug

Command used and terminal output

Relevant files

System information

sarek
sarek copied to clipboard