Monitoring Module: Fix Prometheus Alerts Deployment

[sj-williams]

Background

Currently, our monitoring module uses a local-exec block to kubectl apply our custom prometheus alerts manifest. This has the unfortunate consequence of meaning that updates are not applied on a concourse job run after changes are merged into infra. Changes to alerts have to be applied manually.

We should switch the management of these manifests over to a kubectl_manifest terraform resource (or similar), so that changes are detected and managed natively by terraform.

Proposed user journey

Approach

Which part of the user docs does this impact

Communicate changes

• [ ] post for #cloud-platform-update
• [ ] Weeknotes item
• [ ] Show the Thing/P&A All Hands/User CoP
• [ ] Announcements channel

Questions / Assumptions

Definition of done

• [ ] readme has been updated
• [ ] user docs have been updated
• [ ] another team member has reviewed
• [ ] smoke tests are green
• [ ] prepare demo for the team

Reference

How to write good user stories