autoscaling icon indicating copy to clipboard operation
autoscaling copied to clipboard

Epic: automated autoscaling release workflow

Open Omrigan opened this issue 5 months ago • 0 comments

Motivation

A continuation of https://github.com/neondatabase/cloud/issues/9672 and #837

We now have the script, but this is not a final state.

Requirements

Ideally, we want to:

  1. Have a button (e. g. in GitHub Actions) so that it is easy to do the deployments without the need to setup local environment to run scripts, etc
  2. Have a way to record actions which were taken as a part of a release for an audit
  3. Be able to specify a custom roll-out order for components
  4. Have codereviews/approvals for ops work
  5. Have a machine-readable compatibility matrix (e. g. for #580 and #761)
  6. Engineers not from the core autoscaling team during oncall are able to perform most common fixes, like:
  • pushing an emergency hotfix to the region
  • doing a rollback of a component
  • etc

DoD

  1. Deployment system covers all regular operational scenarios
  2. A subset of features from Requirements

Implementation ideas

  1. To have a desired state of things commited into a special branch (e. g. prod) and a idempotent reconciliation process, triggered by a push to that branch.

Tasks

- [ ] Gather info how about how other teams solve this problem
- [ ] Gather info about tools for automating release workflow for k8s
- [ ] https://github.com/neondatabase/autoscaling/issues/888

Omrigan avatar Mar 05 '24 18:03 Omrigan