optimus
optimus copied to clipboard
raystack
Support for RateLimits for replay such that the scheduled jobs are not impacted.
Is your feature request related to a problem? Please describe. Replay feature can be handy as well it can be a problem where it disrupts the regular scheduled runs, by hitting the worker pool limits or the limits of the underlying Datastore. Would be nice to have a feature to limit the number of jobs due to replay.
Describe the solution you'd like A config which can be configured at project or namespace level to rate limit the number of active job runs due to replay, considering all the downstream runs. An option to force this validation, mainly for admin usage for replaying a higher priority job.
Describe alternatives you've considered Currently, the design doesn't allow a mechanism to identify jobs which are triggered through replay or scheduled runs so the sophistication of configured the jobs differently such that limits can be applied at a underlying datastore or any downstream level as well. Even assigning all the replay requests to a different pool at the scheduler level is not an option now. For all of that a custom scheduler with the state management managed by optimus is the solution but that will be a big change.
Additional context With secret management, Optimus will have a flexibility to configure various service accounts for various jobs so one can configure the different service accounts accordingly such that the resources are properly used across
Why is Even assigning all the replay requests to a different pool at the scheduler level is not an option now not an easy solution for now? Can you elaborate?
Pool is a configuration at a dag, for airflow. currently we can't differentiate between run from b/w regular scheduled job vs job due to replay request right?
