When using scratch=true nxf_unstage can fail silently

[muniheart]

Bug report

Issue #1172 persists in version 23.04.3

Expected behavior and actual behavior

I expect a process task to return a non-zero exit code when task output cannot be copied to workdir. I observe exit code of 0 and incomplete FASTQ output files in workdir despite error message in .command.log.

Steps to reproduce the problem

1. In nextflow.config, process.scratch=true
2. Provide insufficient storage space to file system containing workdir.

Program output

File .command.run nxf_unstage() is: nxf_unstage() { true cp .command.out /scratch/nextflow/humann3/work/53/488896a747de96a6730b41075ffb25/.command.out || true cp .command.err /scratch/nextflow/humann3/work/53/488896a747de96a6730b41075ffb25/.command.err || true cp .command.trace /scratch/nextflow/humann3/work/53/488896a747de96a6730b41075ffb25/.command.trace || true [[ ${nxf_main_ret:=0} != 0 ]] && return IFS=$'\n' for name in $(eval "ls -1d repaired_{1,2}.fastq" | sort | uniq); do nxf_fs_copy "$name" /scratch/nextflow/humann3/work/53/488896a747de96a6730b41075ffb25 || true done unset IFS }

File .command.log contains:

• mkdir -p /scratch/nextflow/humann3/work/53/488896a747de96a6730b41075ffb25/.
• cp -fRL repaired_2.fastq /scratch/nextflow/humann3/work/53/488896a747de96a6730b41075ffb25/. cp: error writing '/scratch/nextflow/humann3/work/53/488896a747de96a6730b41075ffb25/./repaired_2.fastq': Disk quota exceed ed
• true

Environment

• Nextflow version: [23.04.3]
• Java version: [OpenJDK Runtime Environment 18.9 (build 11.0.16+8)]
• Operating system: [Gentoo Base System release 2.6]
• Bash version: (GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu))