CDCgov
Add a PagerDuty alert for when TestEvents fail to serialize
Background
See #4643
Action requested
Create a PagerDuty alert for when the custom exception from #4643 is thrown. As of writing the ticket it is called TestEventSerializationFailureException
Acceptance Criteria
A PagerDuty alert is created for when the custom exception from #4643 is thrown and is set for a reasonable threshold (TBD)
Additional context
This exception has been thrown a couple of times in production since it was enabled (13 times since the end of November). Looking into it a little further, however, the log isn't as useful as I'd hoped - it records the test event id and message, but the message is always just "no value present"; it doesn't specify the field. Furthermore, since this exception causes a cascading failure on a @Transactional method, the test event ID it records isn't actually saved to the database. We probably want to change the logs to be more verbose before trying to make an alert out of this.
Azure search query: https://portal.azure.com#@9ce70869-60db-44fd-abe8-d2767077fc8f/blade/Microsoft_OperationsManagementSuite_Workspace/Logs.ReactView/resourceId/%2Fsubscriptions%2F7d1e3999-6577-4cd5-b296-f518e5c8e677%2FresourceGroups%2Fprime-simple-report-prod%2Fproviders%2Fmicrosoft.insights%2Fcomponents%2Fprime-simple-report-prod-insights/source/LogsBlade.AnalyticsShareLinkToQuery/q/H4sIAAAAAAAAA0utSE4tKMnMzyvmqlEoz0gtSlWIVi%252BpLEhVj1VIzs8rSczMK1ZQCkktLnEtS80rCU4tykzMyaxKBGlxS8zMKS1KdYUZoWQNAKfTYONOAAAA/timespan/2022-11-01T21%3A34%3A40.000Z%2F2023-01-09T21%3A34%3A40.117Z