Parallelize data driven tests

[sam-byng]

Summary

MS Tests supports parallel tests across Assembly, Class and Method levels.

However this does not apply to methods that take input as DataRows / Data driven tests.

Can we add this feature? That would represent a huge gain for our iterative testing processes.

What would a design for this roughly look like?    

Background and Motivation

I've read in the testfx docs that ". Execution of data driven tests will not be parallelized - i.e. parallelizing over DataRow attributes is not supported." https://github.com/microsoft/testfx-docs/blob/main/RFCs/004-In-Assembly-Parallel-Execution.md#notes

Our use case involves running hundreds of .txt file through the same test.

Since each .txt file takes ~15 seconds. A single test run can take multiple hours.

Parallelization would greatly improve this!

Proposed Feature

Update the Parallelization code to expand the DataRows and then run tests.

I've read that this is fundamentally opposed to the implementation of data driven tests: Data driven test infrastructure goes through each row in series (at execution time?). Perhaps this would need a redraft.

Alternative Designs

[Evangelink]

Thanks for the suggestion.

[markm77]

Yes, I would also be super interested in this feature. Some exciting updates to MSTest recently which have got my attention but won't migrate to MSTest without this.

[nohwnd]

Assigning to me to do some initial investigation.

[nohwnd]

Seems to already be working, since the time (simple) data row tests became represented as their own tests:

// file GlobalUsings.cs
global using Microsoft.VisualStudio.TestTools.UnitTesting;

<!-- file mstest134.csproj -->
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <EnableMSTestRunner>true</EnableMSTestRunner>
    <OutputType>Exe</OutputType>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>

    <IsPackable>false</IsPackable>
    <IsTestProject>true</IsTestProject>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="MSTest" Version="3.3.1" />
  </ItemGroup>

</Project>

// file UnitTest1.cs
using System.Diagnostics;

[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.MethodLevel)]

namespace mstest134;

[TestClass]
public class UnitTest1
{
    [TestMethod]
    [DataRow(1)]
    [DataRow(2)]
    public void TestMethod1(int a)
    {
        Console.WriteLine("start");
        Debug.WriteLine("start");
        Thread.Sleep(3_000);
        Console.WriteLine("stop");
        Debug.WriteLine("stop");
    }
}

<!-- file settings.xml -->
<?xml version="1.0" encoding="utf-8" ?>
<RunSettings>
  <MSTest>
    <CaptureTraceOutput>false</CaptureTraceOutput>
  </MSTest>
</RunSettings>

cd S:\t\mstest134
.\bin\Debug\net8.0\mstest134.exe --settings settings.xml

---

This won't work for data rows that don't expand to separate tests, that is for data rows that don't use serializable data (e.g. object). In this issue I am describing the hurdles I had to go through to get my data to expand to separate tests (what I call there "new test UI").

https://github.com/microsoft/testfx/issues/1054