arcade
arcade copied to clipboard
dotnet
User study feedback: Would prefer to see more test failures than shared test failures
Customers would rather see more of the test failures on that are specific to their PRs rather than having a truncated subset and then a section of "shared" test failures.
Yeah... the shared concept is... odd... especially now that we have known issues. Should we remove the "shared with target branch" functionality altogether, and let known issues handle that? That seems to be what the product teams are heading toward anyway... and it's yet another sort of thing here... We have like 5 different ways to deal with "not my fault" failures, and they all render differently, and it's really confusing.
- Build retry
- Local Test Retry
- Known Issues
- Shared Test with Target Branch
- Infrastructure issue
I can see how our customers might be confused.
That's an interesting idea @ChadNedzlek. Maybe we float this on the teams channel and a few known suspects?
It does seem likely that as Known Issues use grows, the "Shared Test Failures" section may disappear, except on the first instance and at that point, someone's going to file a Known Issue on it.
Yeah, so it feels like wasting space differentiating them might be more confusing than providing value at this point. It made a lot of sense before we had known issues, but those seem to be the hot ticket these days, so it makes sense to pivot to make those more central.
Based on conversation in the Team's channel, we may remove the Shared Test Failure section entirely, but we want to have an understanding of how often it shows up. (So we need to get those metrics). Ideally, we're thinking that Known Issues will replace the need for Shared Test Failures, so that section will be unnecessary.
PR: https://dev.azure.com/dnceng/internal/_git/dotnet-helix-service/pullrequest/26328
