Deploy templates for unmonitored queues

[engyebrahim]

https://dev.azure.com/dnceng/internal/_workitems/edit/7373

Release Note Category

• [ ] Feature changes/additions
• [ ] Bug fixes
• [x] Internal Infrastructure Improvements

Release Note Description

Keep scale set templates up-to-date for all available VM-based queues.

[dougbu]

resetting this to Dev since change blocked CI and is not rolling out but we have the starting point in Enji's prior PR. not sure who should finish the work up b/c Enji is OOF and we're a bit low on people. @ilyas1974 this seems like a very useful improvement for future rollouts. should we add it to the Ops backlog and mark it as P1 or something❓

[ilyas1974]

Into the ops backlog is fine. Here is the guidance we put together for prioritization issues - https://dev.azure.com/dnceng/internal/_wiki/wikis/DNCEng%20Services%20Wiki/1225/Issue-Triage-Guidance

[dougbu]

triage doc doesn't mention rollout at all, let alone engineering impact of broken rollouts. I chose P2 based on "Higher impact non-customer-facing service issues" bullet there. of course there may also be a customer impact as well in the future if we don't fix this

[dougbu]

hoping @engyebrahim can get back to this since it is a rollout gotcha and time-waster

[dougbu]

hmm, I didn't mean to self-assign this one

[dougbu]

@engyebrahim you said you were looking at this recently. any progress to report❓ should the issue be assigned to you❓

[engyebrahim]

@dougbu I'm working still on it, i know now the area in code that causes the problem but still working on how to fix it

[dougbu]

@dougbu I'm working still on it, i know now the area in code that causes the problem but still working on how to fix it

I think that means this issue should be assigned to you. please remove your assignment if this is incorrect from your perspetive

[dougbu]

fix in !48927

[dougbu]

I need to drop this one. current branch contains an approximate fix but

1. it updates existing entries in the global table far too frequently. that should only happen when building the 'production' branch and an entry is found w/ the correct definition id. everywhere else, should instead add new entries
2. it repeats the search for deployed queues in multiple subscriptions. since queues reference images in their own subscriptions (after things are copied around IIRC), it should instead do that only for the HelixStaging sub. that's where everything in the two image-tracking tables find images to reuse

[missymessa]

We're not going to do this work. Closing