Error when write a file running Mutect2 with Sarek

[mimifp]

Description of the bug

I'm running Sarek v3.2.3 on an HPC with Slurm as management system. When Mutect2 is running, execution stops due to an error when creating a file. I have made several tests:

• Running the same Sarek pipeline with the same version but in a different directory and with different samples, Mutect2 terminates successfully. I have checked the quality of the bams files and it is correct.
• Running another Sarek version (v 3.2.1) the same error comes up.

This two facts makes me suspect it could be a space or permissions issue, but I'm not quite clear. I leave the error here in case you can help me to clarify it. Thank you very much.

Command used and terminal output

nextflow run nf-core/sarek -r 3.2.3 --input ${SAMPLESHEET} --wes -profile singularity --max_cpus 24 --intervals ${INTERVALS} --outdir $OUTDIR --step variant_calling --tools mutect2 --email [email protected]

Execution cancelled -- Finishing pending tasks before exit
ERROR ~ Error executing process > 'NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:MUTECT2_PAIRED (B1R2_vs_G1)'

Caused by:
  Process `NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:MUTECT2_PAIRED (B1R2_vs_G1)` terminated with an error exit status (1)

Command executed:

  gatk --java-options "-Xmx29491M" Mutect2 \
      --input G1-lane_1.converted.cram --input B1R2-lane_1.converted.cram \
      --output B1R2_vs_G1.mutect2.chr9_66591388-67920552.vcf.gz \
      --reference Homo_sapiens_assembly38.fasta \
      --panel-of-normals 1000g_pon.hg38.vcf.gz \
      --germline-resource af-only-gnomad.hg38.vcf.gz \
      --intervals chr9_66591388-67920552.bed \
      --tmp-dir . \
      --f1r2-tar-gz B1R2_vs_G1.mutect2.chr9_66591388-67920552.f1r2.tar.gz --normal-sample P01_G1
  
  cat <<-END_VERSIONS > versions.yml
  "NFCORE_SAREK:SAREK:BAM_VARIANT_CALLING_SOMATIC_ALL:BAM_VARIANT_CALLING_SOMATIC_MUTECT2:MUTECT2_PAIRED":
      gatk4: $(echo $(gatk --version 2>&1) | sed 's/^.*(GATK) v//; s/ .*$//')
  END_VERSIONS

Command exit status:
  1

Command output:
  [0.001s][warning][os,container] Duplicate cpuset controllers detected. Picking /sys/fs/cgroup/cpuset, skipping /mnt/netapp1/Optcesga_FT2_RHEL7/2020/gentoo/22072020/var/singularity/mnt/session/final/sys/fs/cgroup/cpuset.

Command error:
  23:07:43.140 INFO  ProgressMeter -       chr9:133405690             60.2                437880           7274.5
  23:07:53.502 INFO  ProgressMeter -       chr9:133540799             60.4                438710           7267.4
  23:08:03.594 INFO  ProgressMeter -       chr9:133725565             60.5                439840           7265.9
  23:08:13.701 INFO  ProgressMeter -       chr9:133925123             60.7                441090           7266.3
  23:08:23.706 INFO  ProgressMeter -       chr9:134088694             60.9                442150           7263.8
  23:08:32.545 INFO  Mutect2 - 245998 read(s) filtered by: MappingQualityReadFilter 
  0 read(s) filtered by: MappingQualityAvailableReadFilter 
  0 read(s) filtered by: MappingQualityNotZeroReadFilter 
  0 read(s) filtered by: MappedReadFilter 
  0 read(s) filtered by: NotSecondaryAlignmentReadFilter 
  1900483 read(s) filtered by: NotDuplicateReadFilter 
  0 read(s) filtered by: PassesVendorQualityCheckReadFilter 
  0 read(s) filtered by: NonChimericOriginalAlignmentReadFilter 
  0 read(s) filtered by: NonZeroReferenceLengthAlignmentReadFilter 
  47 read(s) filtered by: ReadLengthReadFilter 
  0 read(s) filtered by: GoodCigarReadFilter 
  0 read(s) filtered by: WellformedReadFilter 
  2146528 total reads filtered out of 8009276 reads processed
  23:08:32.545 INFO  ProgressMeter -       chr9:134182911             61.0                442750           7256.1
  23:08:32.545 INFO  ProgressMeter - Traversal complete. Processed 442750 total regions in 61.0 minutes.
  23:08:32.553 INFO  VectorLoglessPairHMM - Time spent in setup for JNI call : 1.1748570740000002
  23:08:32.553 INFO  PairHMM - Total compute time in PairHMM computeLogLikelihoods() : 154.751703055
  23:08:32.553 INFO  SmithWatermanAligner - Total compute time in java Smith-Waterman : 2227.11 sec
  23:08:32.554 INFO  Mutect2 - Shutting down engine
  [September 11, 2023 at 11:08:32 PM GMT] org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2 done. Elapsed time: 61.03 minutes.
  Runtime.totalMemory()=3238002688
  htsjdk.samtools.SAMException: Could not write metrics to file: /tmp/untarred10106267742814969916/P01_B1R2.ref_histogram
  	at htsjdk.samtools.metrics.MetricsFile.write(MetricsFile.java:138)
  	at org.broadinstitute.hellbender.tools.walkers.readorientation.F1R2CountsCollector.writeHistograms(F1R2CountsCollector.java:164)
  	at org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2Engine.lambda$writeExtraOutputs$13(Mutect2Engine.java:435)
  	at java.base/java.util.Optional.ifPresent(Optional.java:178)
  	at org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2Engine.writeExtraOutputs(Mutect2Engine.java:434)
  	at org.broadinstitute.hellbender.tools.walkers.mutect.Mutect2.onTraversalSuccess(Mutect2.java:297)
  	at org.broadinstitute.hellbender.engine.GATKTool.doWork(GATKTool.java:1102)
  	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.runTool(CommandLineProgram.java:149)
  	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMainPostParseArgs(CommandLineProgram.java:198)
  	at org.broadinstitute.hellbender.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:217)
  	at org.broadinstitute.hellbender.Main.runCommandLineProgram(Main.java:160)
  	at org.broadinstitute.hellbender.Main.mainEntry(Main.java:203)
  	at org.broadinstitute.hellbender.Main.main(Main.java:289)
  Caused by: java.io.FileNotFoundException: /tmp/untarred10106267742814969916/P01_B1R2.ref_histogram (No such file or directory)
  	at java.base/java.io.FileOutputStream.open0(Native Method)
  	at java.base/java.io.FileOutputStream.open(FileOutputStream.java:293)
  	at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:235)
  	at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:184)
  	at java.base/java.io.FileWriter.<init>(FileWriter.java:96)
  	at htsjdk.samtools.metrics.MetricsFile.write(MetricsFile.java:135)
  	... 12 more
  .command.run: line 155: kill: (33) - No such process
  INFO:    Cleaning up image...

Work dir:
  /mnt/lustre/scratch/nlsas/home/usc/mg/translational_oncology/1_tools/01_nf_core_sarek/1_src/b1r2_job/work/2a/832da1404e27cee67fe5d89ea2ffe5

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

 -- Check '.nextflow.log' file for details

Relevant files

No response

System information

• Nextflow version: 23.04.2
• Hardware: HPC Finisterrae III
• Executor: slurm
• Container engine: Singularity v3.9.7
• OS: Linux
• Version os nf-core/sarek: 3.2.3 and tested with 3.2.1 too