Increase Observability Capabilities

[cassandralfernandes]

Epic Overview

Monitoring & Observability

Problem Statements(s)

CiviForm is missing the kind of instrumentation and data presentation that makes maintaining and operating a web service less risky (i.e. observability or o11y). Entities have little more than server logs with the basic cloud-provider provided tools for searching through them. Looking through logs is onerous, and looking into critical metrics such as QPS, request latency, error ratios, etc is impossible.

Improving this circumstance is made more complicated since we cannot mandate a specific paid tool for civic entities to use for o11y. There are however open-source instrumentation tools that allow the implementer to choose their own paid data ingestion and presentation service. https://micrometer.io/ is one such example.

From @shanemc-goog:

Likewise the ability to launch new features an observe they're working as intended is missing.

We currently launch features with code pushes, so the lack of observability makes it hard to do this safely, or know if a rollback is needed.

Product Requirements

n/a

Design & Eng Requirements

link + any important details