dotnet
Auto scale configuration is not cleaned up when Helix queue is deleted
The reunion and xaml helix queues were removed in https://dnceng.visualstudio.com/internal/_git/dotnet-helix-machines/pullrequest/32886 on August 1st.
Since then, the autoscaler has been logging errors of the form:
The Resource 'Microsoft.Compute/virtualMachineScaleSets/windows.amd64.server2022.open.reunion-a-scaleset' under resource group 'windows.amd64.server2022.open.reunion-westus2-v2-rg' was not found.
This happens because the blob container that the auto scaler reads from still has configuration files for these deleted queues.
We should make it so that deleting a queue in the helix-machines YAML also cleans up the auto scaler resources.
Release Note Category
- [ ] Feature changes/additions
- [ ] Bug fixes
- [ ] Internal Infrastructure Improvements
Release Note Description
I removed the files for the reunion and xaml queues so that we clean up the exception logging a bit.
I also cleaned up the ubuntu 1604 config files. These queues were removed in https://dnceng.visualstudio.com/internal/_git/dotnet-helix-machines/pullrequest/34407
Since these config files are re-deployed with each rollout, I'm deleting any that are older than today (May 1, the latest rollout). I confirmed that unmonitored queues still get this update, so they are safe.
Here is a list of what will be deleted:
az storage blob list --account-name helixosconfig --container-name configuration --query "sort_by([?properties.lastModified < ``2024-05-01``].{Name:name, LastModified:properties.lastModified}, &Name)" --output table
| Name | LastModified |
|---|---|
| debian.9.amd64.arcade.zip | 2022-06-29T15:26:59+00:00 |
| debian.9.amd64.open.arcade.zip | 2022-06-29T15:26:59+00:00 |
| debian.9.amd64.open.rt.zip | 2022-06-29T15:27:42+00:00 |
| debian.9.amd64.open.svc.zip | 2022-06-29T15:28:56+00:00 |
| debian.9.amd64.open.zip | 2022-06-29T15:20:52+00:00 |
| debian.9.amd64.rt.zip | 2022-06-29T15:27:40+00:00 |
| debian.9.amd64.svc.zip | 2022-06-29T15:28:56+00:00 |
| debian.9.amd64.zip | 2022-06-29T15:20:52+00:00 |
| osx.1015.amd64.appletv.open.zip | 2023-08-02T20:45:01+00:00 |
| osx.1015.amd64.appletv.zip | 2023-08-02T20:45:01+00:00 |
| osx.1015.amd64.iphone.open.zip | 2023-06-07T00:58:45+00:00 |
| osx.1015.amd64.iphone.zip | 2023-06-07T00:58:45+00:00 |
| osx.1015.amd64.open.zip | 2023-08-02T20:45:01+00:00 |
| osx.1100.amd64.scouting.open.zip | 2022-05-03T19:07:52+00:00 |
| osx.1100.amd64.scouting.zip | 2022-05-03T19:07:52+00:00 |
| osx.1200.amd64.scouting.open.zip | 2022-06-29T15:20:43+00:00 |
| osx.1200.amd64.scouting.zip | 2022-06-29T15:20:43+00:00 |
| raspbian.9.arm32.iot.open.zip | 2022-12-07T21:57:06+00:00 |
| raspbian.9.arm32.iot.zip | 2022-12-07T21:57:06+00:00 |
| redhat.7.amd64.arcade.zip | 2023-11-09T16:07:26+00:00 |
| redhat.7.amd64.open.arcade.zip | 2023-11-09T16:07:26+00:00 |
| ubuntu.1604.amd64.open.rt.zip | 2023-10-25T20:05:40+00:00 |
| ubuntu.1604.amd64.open.svc.zip | 2023-10-25T20:18:16+00:00 |
| ubuntu.1604.amd64.open.zip | 2023-10-25T19:50:48+00:00 |
| ubuntu.1604.amd64.perf.open.zip | 2022-04-20T19:38:16+00:00 |
| ubuntu.1604.amd64.perf.zip | 2022-04-20T19:38:16+00:00 |
| ubuntu.1604.amd64.rt.zip | 2023-10-25T20:05:40+00:00 |
| ubuntu.1604.amd64.svc.zip | 2023-10-25T20:18:16+00:00 |
| ubuntu.1604.amd64.zip | 2023-10-25T19:50:47+00:00 |
| ubuntu.1804.amd64.android.arcade.zip | 2023-10-18T02:39:36+00:00 |
| ubuntu.1804.amd64.android.open.arcade.zip | 2023-10-18T02:39:36+00:00 |
| ubuntu.1804.amd64.arcade.zip | 2023-10-18T02:39:36+00:00 |
| ubuntu.1804.amd64.open.arcade.zip | 2023-10-18T02:39:36+00:00 |
| ubuntu.1804.armarch.repro.zip | 2022-04-20T19:27:41+00:00 |
| ubuntu.2004.ppc64el.experimental.open.zip | 2022-06-18T00:57:53+00:00 |
| ubuntu.2004.ppc64el.experimental.zip | 2022-06-18T00:57:52+00:00 |
| windows.10.amd64.19h1.snake.perf.open.zip | 2022-12-07T21:57:20+00:00 |
| windows.10.amd64.client.open.reunion.zip | 2023-08-02T21:06:49+00:00 |
| windows.10.amd64.client.reunion.zip | 2023-08-02T21:06:49+00:00 |
| windows.10.amd64.client20h2.open.reunion.zip | 2023-08-02T21:06:50+00:00 |
| windows.10.amd64.client20h2.reunion.zip | 2023-08-02T21:06:49+00:00 |
| windows.10.amd64.client20h2.xaml.zip | 2023-08-02T21:06:35+00:00 |
| windows.10.amd64.client21h1.open.reunion.zip | 2022-12-07T21:59:18+00:00 |
| windows.10.amd64.client21h1.open.rt.zip | 2022-12-07T22:00:47+00:00 |
| windows.10.amd64.client21h1.open.svc.zip | 2022-12-07T22:01:30+00:00 |
| windows.10.amd64.client21h1.open.xaml.zip | 2022-12-07T21:58:06+00:00 |
| windows.10.amd64.client21h1.open.zip | 2022-12-07T21:57:18+00:00 |
| windows.10.amd64.client21h1.reunion.zip | 2022-12-07T21:59:18+00:00 |
| windows.10.amd64.client21h1.rt.zip | 2022-12-07T22:00:46+00:00 |
| windows.10.amd64.client21h1.svc.zip | 2022-12-07T22:01:30+00:00 |
| windows.10.amd64.client21h1.xaml.zip | 2022-12-07T21:58:06+00:00 |
| windows.10.amd64.client21h1.zip | 2022-12-07T21:57:18+00:00 |
| windows.10.amd64.clientpre.vs2019.pre.svc.zip | 2022-11-16T19:17:08+00:00 |
| windows.10.amd64.clientpre.vs2019.pre.zip | 2022-11-16T18:46:55+00:00 |
| windows.10.amd64.clientrs1.perf.zip | 2022-04-20T19:38:32+00:00 |
| windows.10.amd64.clientrs5.open.reunion.zip | 2023-08-02T21:06:49+00:00 |
| windows.10.amd64.clientrs5.open.xaml.zip | 2023-08-02T21:06:35+00:00 |
| windows.10.amd64.clientrs5.reunion.zip | 2023-08-02T21:06:49+00:00 |
| windows.10.amd64.clientrs5.xaml.zip | 2023-08-02T21:06:35+00:00 |
| windows.10.amd64.nperf.zip | 2022-04-20T19:38:36+00:00 |
| windows.10.amd64.server19h1.es.open.rt.zip | 2022-03-23T17:37:02+00:00 |
| windows.10.amd64.server19h1.es.open.svc.zip | 2022-03-23T17:38:55+00:00 |
| windows.10.amd64.server19h1.es.open.zip | 2022-03-23T17:34:02+00:00 |
| windows.10.amd64.server19h1.es.svc.zip | 2022-03-23T17:38:54+00:00 |
| windows.10.amd64.server19h1.es.zip | 2022-03-23T17:34:03+00:00 |
| windows.10.amd64.server20h2.open.rt.zip | 2022-07-27T19:22:48+00:00 |
| windows.10.amd64.server20h2.open.svc.zip | 2022-07-27T19:49:36+00:00 |
| windows.10.amd64.server20h2.open.zip | 2022-07-27T18:25:40+00:00 |
| windows.10.amd64.server20h2.rt.zip | 2022-07-27T19:22:47+00:00 |
| windows.10.amd64.server20h2.svc.zip | 2022-07-27T19:53:41+00:00 |
| windows.10.amd64.server20h2.zip | 2022-07-27T18:25:39+00:00 |
| windows.10.amd64.serverrs1.perf.open.zip | 2022-04-20T19:38:33+00:00 |
| windows.10.arm32.iot.open.zip | 2022-12-07T21:57:19+00:00 |
| windows.10.arm32.iot.zip | 2022-12-07T21:57:20+00:00 |
| windows.10.arm32.open.zip | 2022-12-07T21:57:20+00:00 |
| windows.10.arm32.zip | 2022-12-07T21:57:19+00:00 |
| windows.10.arm64.appcompat.zip | 2022-12-07T21:57:19+00:00 |
| windows.10.arm64.open.zip | 2023-01-30T16:58:32+00:00 |
| windows.10.arm64.tof.zip | 2022-12-07T21:57:19+00:00 |
| windows.10.arm64.zip | 2023-01-30T16:58:32+00:00 |
| windows.10.arm64v8.open.zip | 2023-01-30T16:58:32+00:00 |
| windows.10.arm64v8.zip | 2023-01-30T16:58:32+00:00 |
| windows.11.amd64.client.open.reunion.zip | 2023-08-02T21:06:49+00:00 |
| windows.11.amd64.client.open.xaml.zip | 2023-08-02T21:06:35+00:00 |
| windows.11.amd64.client.reunion.zip | 2023-08-02T21:06:49+00:00 |
| windows.11.amd64.client.xaml.zip | 2023-08-02T21:06:35+00:00 |
| windows.11.amd64.clientpre.open.svc.zip | 2022-06-18T01:08:31+00:00 |
| windows.11.amd64.clientpre.open.zip | 2022-06-18T00:57:42+00:00 |
| windows.11.amd64.clientpre.svc.zip | 2022-06-18T01:08:31+00:00 |
| windows.11.amd64.clientpre.zip | 2022-06-18T00:57:42+00:00 |
| windows.11.arm64.perf.surf.zip | 2023-11-09T15:52:43+00:00 |
| windows.7.amd64.open.rt.zip | 2024-04-17T19:55:18+00:00 |
| windows.7.amd64.open.svc.zip | 2024-04-17T19:55:46+00:00 |
| windows.7.amd64.open.zip | 2024-04-17T19:52:08+00:00 |
| windows.7.amd64.rt.zip | 2024-04-17T19:55:18+00:00 |
| windows.7.amd64.svc.zip | 2024-04-17T19:55:46+00:00 |
| windows.7.amd64.zip | 2024-04-17T19:52:08+00:00 |
| windows.amd64.server2022.open.reunion.zip | 2023-08-02T21:06:49+00:00 |
| windows.amd64.server2022.reunion.zip | 2023-08-02T21:06:49+00:00 |
| windows.server.amd64.vs2019.pre.open.svc.zip | 2022-04-20T19:58:22+00:00 |
| windows.server.amd64.vs2019.pre.open.zip | 2022-04-20T19:38:23+00:00 |
| windows.server.amd64.vs2019.pre.svc.zip | 2022-04-20T19:58:23+00:00 |
| windows.server.amd64.vs2019.pre.zip | 2022-04-20T19:38:23+00:00 |
Are we planning to keep this open to remind us to take manual action in the future❔ If not, does something track automating more deletions related to deleted queues❔
I think this issue is tracking a better, automated solution in the future.
It also seems like a good place to note impact and manual interventions in the meantime (which can serve as a measure of the impact this solution would have).
It seems that the service is still throwing exceptions after deleting these config files, what else needs to happen so that we cleanly stop monitoring a queue?