prefect icon indicating copy to clipboard operation
prefect copied to clipboard

Add max concurrent submitted & running flow runs on local agent

Open dkomisar opened this issue 3 years ago • 0 comments

Summary

Our use case is poorly served by having agents submit as many flows as they can. We limit the number that actually run concurrently on the agent but they are still running on the agent as far as prefect is concerned. This PR adds some checks so that the agent will leave flow runs in the scheduled state if the number that have been submitted on the agent but not finished grows greater than a configurable parameter.

Steps Taken to QA Changes

Code has been tested manually. The agent can be tested by running it with the parameter set and then running flow runs and monitoring their status. Better testing can be looked into after we get some initial comments.

Checklist

  • [x] Closes https://github.com/PrefectHQ/prefect/issues/6231

This pull request is:

  • [ ] A documentation / typographical error fix
    • No tests or issue needed
  • [ ] A short code fix
    • Please reference the related issue by including "closes <link to issue>" in this Pull Request's summary section.
      • If no issue exists, please create a bug report issue
    • Please include tests. One-line fixes without tests will not be accepted unless it's related to the documentation only.
  • [x] A new feature implementation
    • Please reference the related issue by including "closes <link to issue>" in this Pull Request's summary section.
      • If no issue exists, please create a feature enhancement issue
    • Please include tests
    • Please make sure that your QA steps are both thorough and easy to reproduce by somebody with limited knowledge of the feature that you are submitting

Happy engineering!

dkomisar avatar Aug 04 '22 16:08 dkomisar