maui
maui copied to clipboard
dotnet
Add possibility to run UI tests per category (or any filter)
Description of Change
This adds a filter to our pipeline in order to be able to run only specific UI tests. The filter value needs to be a valid NUnit string. For example: TestCategory=ScrollView or for multiple categories: TestCategory=ScrollView|TestCategory=Button.
We primarily wanted to filter on categories, this change allows for more flexibility and also filter on method names and other things.
For more info see Microsoft Learn.
Build with filter TestCategory=ScrollView: https://dev.azure.com/xamarin/public/_build/results?buildId=116230&view=results
Issues Fixed
Fixes #21241
As discussed offline, what we do want to still add here is also running each category separately in the pipeline.
That does mean that we will need to keep the categories in code in sync with the categories we list in the pipeline. To make sure that we don't miss running anything what we want to do is:
- have runs for each category separately that we do know (in the pipeline)
- one run that excludes all the categories we do know (in the pipeline) so that it will run the tests that either have a category we don't know about (in the pipeline) or a test that doesn't have a category.
Ideally for the latter we throw some kind of error/warning/something discoverable so we can come look for what is going on and make sure that everything is in sync.
Reverting back to draft for now to see if I can make that all work in this one PR or we should split that up in multiple
Note to self:
- ~looks like test result files get overwritten, make them unique~
- ~group some categories together to split, but not too much?~
- Uncategorized, categories we didn't specify in YAML are not ran with these changes
- ~Add a note to the categories cs file that people should update the YAML~
