Staging - [Alerting] Queue Insights Failures alert
:broken_heart: Metric state changed to alerting
Queue Insights has thrown an unhandled exception and failed to generate its check. This could be caused by invalid data in the Matrix of Truth, or some other component failing.
Wiki Page: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki?wikiVersion=GBwikiMaster&pagePath=/FR%20Operations/Wiki%20for%20Grafana%20Alerts/%5BAlerts%5D%20Queue%20Insights&pageId=956&_a=edit
@dotnet/dnceng, please investigate
Automation information below, do not change
Grafana-Automated-Alert-Id-763d449c7cd747a786373befe76ad19b
:green_heart: Metric state changed to ok
Queue Insights has thrown an unhandled exception and failed to generate its check. This could be caused by invalid data in the Matrix of Truth, or some other component failing.
Wiki Page: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki?wikiVersion=GBwikiMaster&pagePath=/FR%20Operations/Wiki%20for%20Grafana%20Alerts/%5BAlerts%5D%20Queue%20Insights&pageId=956&_a=edit

@melotic I just noticed that the link to the wiki for this alert links to edit the wiki page. If you have time would you please change the link to point just to the regular page?
:broken_heart: Metric state changed to alerting
Queue Insights has thrown an unhandled exception and failed to generate its check. This could be caused by invalid data in the Matrix of Truth, or some other component failing.
Wiki Page: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki?wikiVersion=GBwikiMaster&pagePath=/FR%20Operations/Wiki%20for%20Grafana%20Alerts/%5BAlerts%5D%20Queue%20Insights&pageId=956&_a=edit
:green_heart: Metric state changed to ok
Queue Insights has thrown an unhandled exception and failed to generate its check. This could be caused by invalid data in the Matrix of Truth, or some other component failing.
Wiki Page: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki?wikiVersion=GBwikiMaster&pagePath=/FR%20Operations/Wiki%20for%20Grafana%20Alerts/%5BAlerts%5D%20Queue%20Insights&pageId=956&_a=edit

:broken_heart: Metric state changed to alerting
Queue Insights has thrown an unhandled exception and failed to generate its check. This could be caused by invalid data in the Matrix of Truth, or some other component failing.
Wiki Page: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki?wikiVersion=GBwikiMaster&pagePath=/FR%20Operations/Wiki%20for%20Grafana%20Alerts/%5BAlerts%5D%20Queue%20Insights&pageId=956&_a=edit
:green_heart: Metric state changed to ok
Queue Insights has thrown an unhandled exception and failed to generate its check. This could be caused by invalid data in the Matrix of Truth, or some other component failing.
Wiki Page: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki?wikiVersion=GBwikiMaster&pagePath=/FR%20Operations/Wiki%20for%20Grafana%20Alerts/%5BAlerts%5D%20Queue%20Insights&pageId=956&_a=edit
@AlitzelMendez it seems that the ConfigureAwait was not the issue :(
:broken_heart: Metric state changed to alerting
Queue Insights has thrown an unhandled exception and failed to generate its check. This could be caused by invalid data in the Matrix of Truth, or some other component failing.
Wiki Page: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki?wikiVersion=GBwikiMaster&pagePath=/FR%20Operations/Wiki%20for%20Grafana%20Alerts/%5BAlerts%5D%20Queue%20Insights&pageId=956&_a=edit
:green_heart: Metric state changed to ok
Queue Insights has thrown an unhandled exception and failed to generate its check. This could be caused by invalid data in the Matrix of Truth, or some other component failing.
Wiki Page: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki?wikiVersion=GBwikiMaster&pagePath=/FR%20Operations/Wiki%20for%20Grafana%20Alerts/%5BAlerts%5D%20Queue%20Insights&pageId=956&_a=edit

:green_heart: Metric state changed to ok
Queue Insights has thrown an unhandled exception and failed to generate its check. This could be caused by invalid data in the Matrix of Truth, or some other component failing.
Wiki Page: https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki?wikiVersion=GBwikiMaster&pagePath=/FR%20Operations/Wiki%20for%20Grafana%20Alerts/%5BAlerts%5D%20Queue%20Insights&pageId=956&_a=edit

Everything appears to be functioning normally. I am closing this alert. If we experience this problem again, we will investigate.
For context on what changed (and hopefully sticks this time). QueueInsights now retries when there's a sporadic Kusto exception