dotnet
User study feedback: Consider splitting Build Analysis into separate checks per pipeline
This suggestion would help our issue with running out of character space on the check as well as providing a logical division of errors for our customers to investigate (instead of smooshing all the errors together on the same view). This would primarily help the runtime repo, but would be useful for any repo that uses multiple pipelines for PR builds.
Isn't a big part of the value that you have a one stop shop for your problems? Having to bounce around a bunch of checks to investigate the problem seems counter to our goal (and the existing AzDO checks already do that if that's what people want).
Here's an example of a run where it would be useful just yesterday:
https://github.com/dotnet/runtime/runs/8500666297
There's a lot of failures, most of them are the same known thing that hasn't been fixed in several days, but duplicated across several legs. I especially do not care about the iOS/tvOS/Bionic/whatever legs (the change can only have an effect on NativeAOT legs), but there's just too many failures across all pipelines for build analysis to be useful. I ended up having to click through AzDO legs to find that yep, I would introduce a regression in one test that I care about.
It does say under the header "• [All failing tests from runtime-extra-platforms]" and lists no other pipelines, and then again at the bottom "2855 other tests are failing: [All failing tests from runtime-extra-platforms]"
The fact that there are 2000 existing failing tests seems like a big part of the problem. Though an unfortunately common one. :-(
I guess what I would want is some summary of test failures in each leg instead of each of the pipelines (i.e. - I only want the NativeAOT ones in the above case, not everything from extra-platforms).
I guess we could create some sort of count by leg, though it would just be numbers that you wouldn't really be able to take action on without going to AzDO. The goal of the feature isn't to make it so that you never have to go to AzDO... it's to try to given you enough information that most of the time most users can take action immediately. Just showing a summary isn't really actionable... what action would you take if we showed such a summary... other than "go to AzDO to investigate more"... at which point I'm not sure what value we've added to that specific scenario.