There's a mount issue using apptainer

[alexanderchang1]

Description of the bug

I get this error that kills the pipeline using the apptainer pipeline

Caused by: Process NFCORE_SAREK:SAREK:PREPARE_GENOME:BWAMEM1_INDEX (genome.fasta) terminated with an error exit status (255)

Command executed:

mkdir bwa bwa
index

-p bwa/genome
genome.fasta

cat <<-END_VERSIONS > versions.yml "NFCORE_SAREK:SAREK:PREPARE_GENOME:BWAMEM1_INDEX": bwa: $(echo $(bwa 2>&1) | sed 's/^.Version: //; s/Contact:.$//') END_VERSIONS

Command exit status: 255

Command output: (empty)

Command error: WARNING: Skipping mount /var/apptainer/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container FATAL: container creation failed: mount hook function failure: mount /tmp/nxf.7ZcU6MZzH9->/tmp/nxf.7ZcU6MZzH9 error: while mounting /tmp/nxf.7ZcU6MZzH9: destination /tmp/nxf.7ZcU6MZzH9 doesn't exist in container

Work dir: /ocean/projects/aurora/achang4/sarek/Part2_FASTQ2SAREK/work/70/36e5cb0114519a69f081bd329e4960

Command used and terminal output

nextflow run nf-core/sarek -r 3.4.0 -profile test,apptainer \
    --input /ocean/projects/aurora/achang4/sarek/Part1.2_ASSEMBLE_BAM_INPUT/samplesheet.csv \
    --outdir /ocean/projects/aurora/achang4/sarek/Part2_FASTQ2SAREK/results/ \
    --genome GATK.GRCh38 \
    --tools mutect2,strelka,manta \
    -c slurm.config  \
    -resume

Relevant files

nextflow.zip

System information

Nextflow Version 23.10.0 build 5889 Hardware: HPC Executor: Slurm Container enginer: Apptainer OS: Ubuntu Version of Sarek: 3.4.0

[alexanderchang1]

It looks like the issue is when you specify the apptainer profile, this issue doesn't replicate when you specify singularity profile.

Edit: Spoke too soon, same crashing error.

FATAL: container creation failed: mount hook function failure: mount /etc/resolv.conf->/etc/resolv.conf error: while mounting /etc/resolv.conf: destination /etc/resolv.conf doesn't exist in container

[asp8200]

Similar issue was reported for nf-core/eager.

I've occasionally experienced issues with having the pipelines fetch the containers. This problem might also be related to the pipeline fetching or creating containers. In general, I recommend fetching the containers using the nf-core tool with a command like

nf-core download sarek -r 3.4.0 -s singularity  --outdir sarek_download`.

The tool can also fail or fetch incorrect containers, but then I just fetch those manually with a command like, for instance,

apptainer pull  --name quay.io-biocontainers-bcftools-1.17--haef29d1_0.img docker://quay.io/biocontainers/bcftools:1.17--haef29d1_0

I also experienced that an apptainer pull may fail on one system but run on another. (That is probably due to some incorrect configuration of apptainer.)

Once you've fetched the containers, you collect them in a folder and have the environment variable NXF_SINGULARITY_CACHEDIR point to that folder, and then you export that variable before running nextflow.

By the way, I wasn't able to reproduce the problem with this command:

nextflow run main.nf -profile test,apptainer --tools strelka -c my_slurm.config --outdir results

Do you also experience the problem if you run that command?

Finally, I recommend that you report problems like this on nf-core/sarek Slack before creating GitHub-issue, as people there often respond faster. Cheers