dnceng icon indicating copy to clipboard operation
dnceng copied to clipboard

Address existing open alerts, and fix Grafana Alerting automation not opening issues in GitHub

Open riarenas opened this issue 1 year ago • 2 comments

We have active alerts: https://dotnet-eng-grafana.westus2.cloudapp.azure.com/d/introduction/introduction?orgId=1&viewPanel=9 that haven't created a github issue.

Release Note Category

  • [ ] Feature changes/additions
  • [ ] Bug fixes
  • [ ] Internal Infrastructure Improvements

Release Note Description

riarenas avatar Oct 07 '24 13:10 riarenas

Looking at the current set of active grafana alerts, we have an issue associated with each of them. Closing this issue in favor of those. @riarenas thank you for bringing this to my attention.

ilyas1974 avatar Oct 09 '24 18:10 ilyas1974

Spoke too soon. @davfost could you please take a look the https://dotnet-eng-grafana.westus2.cloudapp.azure.com/d/buildAnalysis/build-analysis?viewPanel=46&orgId=1 (Build Analysis) and https://dotnet-eng-grafana.westus2.cloudapp.azure.com/d/arcadeAvailability/service-availability?viewPanel=19&orgId=1&refresh=30s (Helix API response time). The alerts for the mobile devices are being address by DDFUN.

ilyas1974 avatar Oct 09 '24 18:10 ilyas1974

I have been investigating this but have limited access to Grafana.

I am still waiting to get admin access, but it appears that after the latest deployment the issue was resolved.

Meaning the rule shows 'green' now.

davfost avatar Oct 16 '24 17:10 davfost

Note that grafana alerting is still broken: https://github.com/dotnet/dnceng/issues/4279

(I also made you admin in grafana)

riarenas avatar Oct 17 '24 13:10 riarenas