nextflow icon indicating copy to clipboard operation
nextflow copied to clipboard

clusterOptions does not work with CLI flags (LSF executor)

Open timini opened this issue 2 months ago • 3 comments

Bug report

when specigying a clusterOptions which includes a flag the Grid executor header is constructed wrongly by nextflow

e.g profile.config :

clusterOptions = { "-g /${_queue_group} -app ${params.fast_track ? params.lsf_fast_track_profile : task.process.tokenize(':')[-1].toLowerCase()} -sp ${params.hpc_priority} -P ${params.lsf_project_name} -rn" }

results in the following BSUB command (LSF executor)

#!/bin/bash
#BSUB -o /home/tim/nextflow/13/e0aeb3d2121ed180dc3272e28de8a0/.command.log
#BSUB -q high
#BSUB -W 24:00
#BSUB -J nf-DELIVERY_INTEGRITY_MOVE_DELIVERY_(HX23697398_LP5221738-DNA_H07)
#BSUB -rn -g
#BSUB /high -app
# NEXTFLOW TASK: DELIVERY_INTEGRITY:MOVE_DELIVERY (HX23697398/LP5221738-DNA_H07)
set -e
set -u
NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 1 ]] && set -x
NXF_ENTRY=${1:-nxf_main}

Expected behavior and actual behavior

I would expect flags (keys with no values) to be correctly rendered in the BSUB option lists

#!/bin/bash
#BSUB -o /home/tim/nextflow/13/e0aeb3d2121ed180dc3272e28de8a0/.command.log
#BSUB -q high
#BSUB -W 24:00
#BSUB -J nf-DELIVERY_INTEGRITY_MOVE_DELIVERY_(HX23697398_LP5221738-DNA_H07)
#BSUB -rn 
#BSUB -g /high
#BSUB  -app something
# NEXTFLOW TASK: DELIVERY_INTEGRITY:MOVE_DELIVERY (HX23697398/LP5221738-DNA_H07)
set -e
set -u
NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 1 ]] && set -x
NXF_ENTRY=${1:-nxf_main}

Steps to reproduce the problem

(Provide a test case that reproduce the problem either with a self-contained script or GitHub repository)

Program output

(Copy and paste here output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the .nextflow.log file.)

Environment

  • Nextflow version: [?]
  • Java version: [?]
  • Operating system: [macOS, Linux, etc]
  • Bash version: (use the command $SHELL --version)

Additional context

(Add any other context about the problem here)

timini avatar Apr 19 '24 14:04 timini

It is because the abstract grid executor assumes the cluster options are provided as key-value pairs:

https://github.com/nextflow-io/nextflow/blob/7c9d965e1d7b39ed58fb189d4649d8d57ed81fc0/modules/nextflow/src/main/groovy/nextflow/executor/AbstractGridExecutor.groovy#L95-L116

So you can probably make clusterOptions work by inserting a blank value after -rn:

clusterOptions = { "-g /${_queue_group} -app ${params.fast_track ? params.lsf_fast_track_profile : task.process.tokenize(':')[-1].toLowerCase()} -sp ${params.hpc_priority} -P ${params.lsf_project_name} -rn ''" }

bentsherman avatar Apr 19 '24 15:04 bentsherman

@pditommaso what if we allow clusterOptions to be a list of strings, then it becomes clear how to map the options to the directive lines:

clusterOptions = { [
  "-g /${_queue_group}",
  "-app ${params.fast_track ? params.lsf_fast_track_profile : task.process.tokenize(':')[-1].toLowerCase()}",
  "-sp ${params.hpc_priority}",
  "-P ${params.lsf_project_name}",
  "-rn" ] }

bentsherman avatar Apr 19 '24 15:04 bentsherman

It makes sense

pditommaso avatar Apr 21 '24 10:04 pditommaso