vstest icon indicating copy to clipboard operation
vstest copied to clipboard

Test execution ordering/filtering extension

Open NGloreous opened this issue 5 years ago • 0 comments

Description

Add an extensibility point to support filtering what tests to run and the ability to control the order in which they run.

There is really two asks here but I think it makes sense to combine them into one extensibility point.

The OneDrive team has a large set of tests which need to run. To accomplish this in a reasonable time period we stripe our tests across multiple machines. We also have custom logic that uses historical test runtime to do this effectively by ordering tests in a smart way (e.g. longer tests run first).

We can accomplish the filtering by using the Translation Layer extension but we rather just run vstest.console.exe directly for various reasons.

The ordering could possibly be accomplished by providing a hard code config but being able to dynamically control ordering provides more flexibility and could possibly integrate with the retry support feature to control the order to run retries.

NGloreous avatar Mar 06 '19 22:03 NGloreous