harbor icon indicating copy to clipboard operation
harbor copied to clipboard

Add maintenance mode that pauses all scheduled jobs while enabled

Open dkulchinsky opened this issue 3 years ago • 2 comments

Is your feature request related to a problem? Please describe. No

Describe the solution you'd like Harbor provides a read-only mode that blocks any writes to the system, this is useful however we identified cases where we would also like to have a Maintenance Mode that will automatically pause all scheduled jobs from executing while it is enabled (retention rules, garbage collection, replication, etc...)

Using maintenance mode is particularly useful during Harbor upgrades or infrastructure work related to where Harbor is deployed, in our case the most common case is K8s cluster upgrades, we would like to actively disable any jobs from executing while we're making these upgrades as currently it causes things to fail due to transient errors when services connect to each other and could be avoided.

Describe the main design/architecture of your solution Maintenance mode toggle (enable/disable) and optionally a duration (i.e. that turns off maintenance mode after a defined period) that can be set from the UI and API, while enabled the system will not execute any scheduled/recurring jobs.

Describe the development plan you've considered Don't have any specific plan here, but I suspect this is mostly going to affect the Jobservice scheduling engine that will need to check the state of the maintenance mode toggle and act not execute jobs if it's enabled.

Additional context N/A

dkulchinsky avatar Jun 09 '21 12:06 dkulchinsky

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] avatar Jul 05 '22 11:07 github-actions[bot]

still relevant

dkulchinsky avatar Jul 06 '22 01:07 dkulchinsky

We have enhanced the read-only mode in 2.6 which also blocks the write operation from the internal jobs. So to a certain extent, I think this should meet your requirement with this change

ywk253100 avatar Sep 02 '22 07:09 ywk253100

With this feature https://github.com/goharbor/harbor/issues/17512 integrated, The admin user could pause all pending tasks (include scheduler) and stop all the running jobs in the worker pool. it meets the requirement of maintenance mode, right?

stonezdj avatar Dec 13 '22 02:12 stonezdj

Hey @stonezdj, apologies for not replying sooner.

Closing this as this is now possible with the new Jobs UI

Thanks!

dkulchinsky avatar May 09 '23 16:05 dkulchinsky