nextflow
nextflow copied to clipboard
clusterOptions does not work with CLI flags (LSF executor)
Bug report
when specigying a clusterOptions which includes a flag the Grid executor header is constructed wrongly by nextflow
e.g profile.config :
clusterOptions = { "-g /${_queue_group} -app ${params.fast_track ? params.lsf_fast_track_profile : task.process.tokenize(':')[-1].toLowerCase()} -sp ${params.hpc_priority} -P ${params.lsf_project_name} -rn" }
results in the following BSUB command (LSF executor)
#!/bin/bash
#BSUB -o /home/tim/nextflow/13/e0aeb3d2121ed180dc3272e28de8a0/.command.log
#BSUB -q high
#BSUB -W 24:00
#BSUB -J nf-DELIVERY_INTEGRITY_MOVE_DELIVERY_(HX23697398_LP5221738-DNA_H07)
#BSUB -rn -g
#BSUB /high -app
# NEXTFLOW TASK: DELIVERY_INTEGRITY:MOVE_DELIVERY (HX23697398/LP5221738-DNA_H07)
set -e
set -u
NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 1 ]] && set -x
NXF_ENTRY=${1:-nxf_main}
Expected behavior and actual behavior
I would expect flags (keys with no values) to be correctly rendered in the BSUB option lists
#!/bin/bash
#BSUB -o /home/tim/nextflow/13/e0aeb3d2121ed180dc3272e28de8a0/.command.log
#BSUB -q high
#BSUB -W 24:00
#BSUB -J nf-DELIVERY_INTEGRITY_MOVE_DELIVERY_(HX23697398_LP5221738-DNA_H07)
#BSUB -rn
#BSUB -g /high
#BSUB -app something
# NEXTFLOW TASK: DELIVERY_INTEGRITY:MOVE_DELIVERY (HX23697398/LP5221738-DNA_H07)
set -e
set -u
NXF_DEBUG=${NXF_DEBUG:=0}; [[ $NXF_DEBUG > 1 ]] && set -x
NXF_ENTRY=${1:-nxf_main}
Steps to reproduce the problem
(Provide a test case that reproduce the problem either with a self-contained script or GitHub repository)
Program output
(Copy and paste here output produced by the failing execution. Please highlight it as a code block. Whenever possible upload the .nextflow.log
file.)
Environment
- Nextflow version: [?]
- Java version: [?]
- Operating system: [macOS, Linux, etc]
- Bash version: (use the command
$SHELL --version
)
Additional context
(Add any other context about the problem here)
It is because the abstract grid executor assumes the cluster options are provided as key-value pairs:
https://github.com/nextflow-io/nextflow/blob/7c9d965e1d7b39ed58fb189d4649d8d57ed81fc0/modules/nextflow/src/main/groovy/nextflow/executor/AbstractGridExecutor.groovy#L95-L116
So you can probably make clusterOptions
work by inserting a blank value after -rn
:
clusterOptions = { "-g /${_queue_group} -app ${params.fast_track ? params.lsf_fast_track_profile : task.process.tokenize(':')[-1].toLowerCase()} -sp ${params.hpc_priority} -P ${params.lsf_project_name} -rn ''" }
@pditommaso what if we allow clusterOptions to be a list of strings, then it becomes clear how to map the options to the directive lines:
clusterOptions = { [
"-g /${_queue_group}",
"-app ${params.fast_track ? params.lsf_fast_track_profile : task.process.tokenize(':')[-1].toLowerCase()}",
"-sp ${params.hpc_priority}",
"-P ${params.lsf_project_name}",
"-rn" ] }
It makes sense