nextflow icon indicating copy to clipboard operation
nextflow copied to clipboard

Solve queueSize exceeded when using job arrays

Open jorgee opened this issue 6 months ago • 4 comments

close #5920

  • Modifying canSubmit to avoid to avoid exceed executor.queueSize parameter.
  • Printing warning when array size exceeds the queueSize. This can make the task array 'unsubmitable' (Not sure if we should abort the execution in this situation)
  • Add unit tests

Tested with this pipeline

process test {
    array 10 
    input:
    val x

    """
    echo $x
    """
}

workflow {
   Channel.of(1..20) | test
}

With a config with awsbatch executor and different executor.queueSize values:

  • size 15: Only one job submitted at a time in AWS Batch.
  • size 5: Prints a warning but run continues without the task array jobs submitted.

jorgee avatar May 07 '25 16:05 jorgee

Deploy Preview for nextflow-docs-staging ready!

Name Link
Latest commit 0a557a531bbf40622ab89d9b37283a9b17c83e61
Latest deploy log https://app.netlify.com/projects/nextflow-docs-staging/deploys/6846d81fd7c9ae00082f239c
Deploy Preview https://deploy-preview-6047--nextflow-docs-staging.netlify.app
Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

netlify[bot] avatar May 07 '25 16:05 netlify[bot]

  • change the warning to an error, since the run will hang anyway
  • make process maxForks aware of job arrays

bentsherman avatar May 29 '25 13:05 bentsherman

Because processor.forksCount is related to maxForks directive, instead here the aim is to controller the max numbers of task runs

pditommaso avatar May 30 '25 16:05 pditommaso

So "forks" here just refers to the number of concurrent tasks, therefore I see no reason to call it something else in the task handler

bentsherman avatar May 30 '25 17:05 bentsherman