Error when using Shifter as container runtime in Nextflow >23.08.0-edge

[jackscanlan]

When using Shifter (shifter.enabled = true), processes using containers fail (see error output below). I first noticed this in Nextflow version 24.04.2, but after stepping through versions, 23.08.1-edge and higher (tested up to 24.04.2) throw this error, but 23.08.0-edge and lower do not.

Program output

Typical error output:

ERROR ~ Error executing process > 'FREYR:PARSE_INPUTS (Whole dataset)'

Caused by:
  Process `FREYR:PARSE_INPUTS (Whole dataset)` terminated with an error exit status (127)


Command executed:

  #!/usr/bin/env Rscript
  
  ### defining Nextflow environment variables as R variables
  ## input channel variables
  samplesheet =           "test_data/dual/samplesheet_read_dir.csv"
  loci_params =           "test_data/dual/loci_params.csv"
  
  ## global variables
  projectDir = "/group/pathogens/IAWS/Personal/JackS/dev/freyr"
  params_dict = "[help:null, data_folder:test_data/dual, refdir:reference, samplesheet:test_data/dual/samplesheet_read_dir.csv, loci_params:test_data/dual/loci_params.csv, extension:null, illumina:true, pacbio:false, nanopore:false, paired:true, high_sensitivity:true, threads:null, rdata:true, slurm_account:pathogens, max_memory:2.GB, max_cpus:1, max_time:10.m]"
  
  ### source functions and themes, load packages, and import Nextflow params
  ### from "bin/process_start.R"
  sys.source("/group/pathogens/IAWS/Personal/JackS/dev/freyr/bin/process_start.R", envir = .GlobalEnv)
  
  ### run module code
  sys.source(
      "/group/pathogens/IAWS/Personal/JackS/dev/freyr/bin/parse_inputs.R", # run script
      envir = .GlobalEnv # this allows import of existing objects like projectDir
  )
  
  ### save .RData for debugging
  if ("true" == "true") {
      save.image()
  } else {
      NULL
  }

Command exit status:
  127

Command output:
  
  2024-07-05T14:53:51 Pulling Image: docker:jackscanlan/piperline-multi:0.0.1, status: READY
  2a3f21b84278741a6d7226093a69f1463064515cf6d5dbdd3fe172039b8b6c05
  2a3f21b84278741a6d7226093a69f1463064515cf6d5dbdd3fe172039b8b6c05

Command error:
  /var/spool/slurm/d/job28977797/slurm_script: line 326: NXF_TASK_WORKDIR=/group/pathogens/IAWS/Personal/JackS/dev/freyr/work/68/bc374c9440715e61fb629278528cd2: No such file or directory

Work dir:
  /group/pathogens/IAWS/Personal/JackS/dev/freyr/work/68/bc374c9440715e61fb629278528cd2

Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line

 -- Check '.nextflow.log' file for details

Error seems to be coming from this part of the process' .command.run file:

nxf_launch() {
    shifterimg pull docker:jackscanlan/piperline-multi:0.0.1
    shifterimg lookup docker:jackscanlan/piperline-multi:0.0.1
    while ! shifterimg lookup docker:jackscanlan/piperline-multi:0.0.1; do
        sleep 5
        STATUS=$(shifterimg -v pull docker:jackscanlan/piperline-multi:0.0.1 | tail -n2 | head -n1 | awk '{print $6}')
        [[ $STATUS == "FAILURE" || -z $STATUS ]] && echo "Shifter failed to pull image 'docker:jackscanlan/piperline-multi:0.0.1'" >&2  && exit 1
    done
    ${NXF_TASK_WORKDIR:+"NXF_TASK_WORKDIR=$NXF_TASK_WORKDIR"} NXF_DEBUG=${NXF_DEBUG:=0} shifter --image docker:jackscanlan/piperline-multi:0.0.1 /bin/bash -c "eval $(nxf_container_env); /bin/bash /group/pathogens/IAWS/Personal/JackS/dev/freyr/work/68/bc374c9440715e61fb629278528cd2/.command.run nxf_trace"
}

.nextflow.log for the command: NXF_VER=23.08.1-edge nextflow run AVR-biosecurity-bioinformatics/freyr -profile basc_slurm,test

.nextflow.log for the command: NXF_VER=24.04.2 nextflow run AVR-biosecurity-bioinformatics/freyr -profile basc_slurm,test

Steps to reproduce the problem

If you have Shifter installed, you can reproduce using a test dataset with my pipeline:

# runs fine past first process requiring container ('PARSE_INPUTS')
NXF_VER=23.08.0-edge nextflow run AVR-biosecurity-bioinformatics/freyr -profile shifter,test

# throws error during first process requiring container ('PARSE_INPUTS')
NXF_VER=24.04.2 nextflow run AVR-biosecurity-bioinformatics/freyr -profile shifter,test

I've reproduced this with a different, independent pipeline on the same system:

git clone https://github.com/tuannguyen8390/nf-EXPLOR.git 

# runs fine past first process requiring container ('make_map_index' or 'make_gatk_index')
NXF_VER=23.08.0-edge nextflow run setup.nf -profile shifter

# throws error during first process requiring container ('make_map_index' or 'make_gatk_index')
NXF_VER=24.04.2 nextflow run setup.nf -profile shifter

Environment

• Nextflow version: 23.08.1-edge through to 24.04.2
• Java version: 17
• Operating system: Linux 3.10.0-1160.92.1.el7.x86_64
• Bash version: 4.2.46(2)-release
• Shifter version: 22.02.1