Staging /dev causes apptainer/singularity processes to fail.

[alanhoyle]

Bug report

Expected behavior and actual behavior

Run a process that takes a file parameter as one of the inputs, set the file to be /dev/null, and run it in an apptainer/singularity container.

Under local executor and Docker (all I can test), everything works fine.

Under apptainer, the process fails due to some weird issue with /dev being staged and bound within the process.

Steps to reproduce the problem

Check out https://github.com/alanhoyle/nf-null-singularity-staging and run bash ./run_me_test.sh on a machine that has Singularity available.

The test script runs 6 total nextflow workflows. With the local executor, it runs using a real file as input, and then with /dev/null as the input file. Then it does the same thing using Docker container, and then the same thing using apptainer.

Program output

(Copy and paste the output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the .nextflow.log file.)

 N E X T F L O W   ~  version 25.04.1

Launching `./main.nf` [evil_hypatia] DSL2 - revision: de3f7f1720

executor >  local (1)
executor >  local (1)
[6b/53b117] print_input [  0%] 0 of 1 ✘
ERROR ~ Error executing process > 'print_input'

Caused by:
  Process `print_input` terminated with an error exit status (1)


Command executed:

  echo "Input file contents:"
  cat null
  echo "stage dir contents:" > output.txt
  ls -l >> output.txt
  echo "" >> output.txt

  echo "input file contents:" >> output.txt
  cat null >> output.txt

Command exit status:
  1

Command output:
  Input file contents:

Command error:
  Input file contents:
  cat: null: Permission denied

Work dir:
  /datastore/scratch/users/alanh/nextflow/6b/53b11745de73f15010e2b6e1b882cb

Container:
  /home/alanh/singularity/ubuntu-latest.img

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details

Environment

• Nextflow version: 25.04.1
• Java version: openjdk 23.0.2-internal 2025-01-21
• Operating system: Linux
• Bash version: GNU bash, version 5.1.8(1)-release (x86_64-redhat-linux-gnu)

Additional context

This seems to be an issue whenever -B /dev is added to an apptainer run or apptainer exec command to bind that dir into the container.

apptainer run docker://ubuntu:latest works fine, but apptainer run -B /dev docker://ubuntu:latest throws a ton of errors, even outside of the Nextflow context.

[alanhoyle]

This might be something that needs to be fixed upstream, so I put this bug report in for them: https://github.com/apptainer/apptainer/issues/2963

[alanhoyle]

Apptainer seems to think this is something that should be fixed at the Nextflow level. I think fixing it at both places would be even better. Any thoughts on the proposed solution below?

/modules/nextflow/src/main/groovy/nextflow/container/SingularityBuilder.groovy

    protected String composeVolumePath( String path, boolean readOnly = false ) {
        def result = "-B ${escape(path)}"
        if( readOnly )
            result += ":${escape(path)}:ro"
        if (path == "/dev" || path.startsWith("/dev/"))
            result = ""
        return result
    }