testfx icon indicating copy to clipboard operation
testfx copied to clipboard

Add TestCategories property to ITestDataRow for per-test-case categorization

Open Copilot opened this issue 7 months ago • 4 comments

This PR adds support for applying test categories to individual test data rows in parameterized tests, addressing a long-standing user request for more granular test categorization.

Changes

Core API Changes

  • Added TestCategories property to ITestDataRow interface
  • Implemented TestCategories property in TestDataRow<T> class with [DataMember] attribute
  • Property is nullable to maintain backward compatibility

Discovery Logic Updates

  • Updated TestDataSourceHelpers.TryHandleITestDataRow() to extract test categories from test data rows
  • Added backward-compatible overload to maintain existing API
  • Modified discovery logic in AssemblyEnumerator to merge test categories from test data rows with existing method/class/assembly level categories

Usage Example

[TestMethod]
[DynamicData(nameof(GetTestData), DynamicDataSourceType.Method)]
public void MyParameterizedTest(string input, string expected)
{
    Assert.AreEqual(expected, ProcessInput(input));
}

public static IEnumerable<object[]> GetTestData()
{
    // Fast unit test case
    yield return new TestDataRow<(string, string)>(("input1", "output1"))
    {
        TestCategories = new List<string> { "Unit", "Fast" },
        DisplayName = "Fast unit test"
    };
    
    // Slow integration test case
    yield return new TestDataRow<(string, string)>(("input2", "output2"))
    {
        TestCategories = new List<string> { "Integration", "Slow" },
        DisplayName = "Integration test"
    };
}

Benefits

  • Granular filtering: Can now filter tests by categories applied to individual test cases
  • Mixed scenarios: Single test method can contain both unit tests and integration tests with appropriate categories
  • Category inheritance: Test data row categories are merged with method/class/assembly level categories
  • Backward compatibility: Existing code continues to work unchanged

Test Coverage

  • Added comprehensive unit tests for TestDataRow and TestDataSourceHelpers
  • Added integration tests to verify end-to-end functionality
  • Created sample test project demonstrating real-world usage
  • Validated category filtering works correctly with test discovery

This enables scenarios like:

# Run only fast test cases
dotnet test --filter "TestCategory=Fast"

# Run only integration test cases  
dotnet test --filter "TestCategory=Integration"

Fixes #4089.

[!WARNING]

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • 5rqvsblobprodcus385.vsblob.vsassets.io
    • Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:quiet /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/testfx/testfx/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/testfx/testfx/artifacts/toolset/10.0.0-beta.25316.2.txt (dns block)

If you need me to access, download, or install something from one of these locations, you can either:


💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

Copilot avatar Jun 17 '25 08:06 Copilot

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

[!WARNING]

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • 5rqvsblobprodcus385.vsblob.vsassets.io
    • Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/testfx/testfx/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/testfx/testfx/artifacts/toolset/10.0.0-beta.25316.2.txt (dns block)
    • Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet build src/TestFramework/TestFramework/TestFramework.csproj --no-restore (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot avatar Jun 20 '25 09:06 Copilot

@copilot There is a merge conflict. Can you fix it?

Youssef1313 avatar Jun 20 '25 09:06 Youssef1313

@copilot There is a merge conflict. Can you fix it?

The merge conflict in PublicAPI.Unshipped.txt has been resolved. I merged both the TestCategories property entries and the new Assert methods from main branch. (0ae57ea)

Copilot avatar Jun 20 '25 09:06 Copilot

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

[!WARNING]

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • 5rqvsblobprodcus385.vsblob.vsassets.io
    • Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/testfx/testfx/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/testfx/testfx/artifacts/toolset/10.0.0-beta.25316.2.txt (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot avatar Jun 20 '25 14:06 Copilot

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

[!WARNING]

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • 5rqvsblobprodcus385.vsblob.vsassets.io
    • Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/testfx/testfx/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/testfx/testfx/artifacts/toolset/10.0.0-beta.25316.2.txt (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot avatar Jun 23 '25 07:06 Copilot

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

[!WARNING]

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • ytvvsblobprodcus310.vsblob.vsassets.io
    • Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/testfx/testfx/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/testfx/testfx/artifacts/toolset/10.0.0-beta.25352.1.txt (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot avatar Jul 08 '25 10:07 Copilot

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

[!WARNING]

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • ytvvsblobprodcus310.vsblob.vsassets.io
    • Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/testfx/testfx/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/testfx/testfx/artifacts/toolset/10.0.0-beta.25352.1.txt (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot avatar Jul 08 '25 10:07 Copilot

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

[!WARNING]

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • ytvvsblobprodcus310.vsblob.vsassets.io
    • Triggering command: /home/REDACTED/work/testfx/testfx/.dotnet/dotnet msbuild /m /nologo /clp:Summary /v:minimal /nr:true /warnaserror /p:TreatWarningsAsErrors=true /p:ContinuousIntegrationBuild=false /home/REDACTED/work/testfx/testfx/artifacts/toolset/restore.proj /t:__WriteToolsetLocation /clp:ErrorsOnly;NoSummary /p:__ToolsetLocationOutputFile=/home/REDACTED/work/testfx/testfx/artifacts/toolset/10.0.0-beta.25352.1.txt (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot avatar Jul 08 '25 10:07 Copilot

Codecov Report

Attention: Patch coverage is 66.66667% with 6 lines in your changes missing coverage. Please review.

Project coverage is 73.18%. Comparing base (591ca0a) to head (158ac3c). Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
....PlatformServices/Helpers/TestDataSourceHelpers.cs 44.44% 5 Missing :warning:
src/TestFramework/TestFramework/TestDataRow.cs 50.00% 1 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5795      +/-   ##
==========================================
- Coverage   73.57%   73.18%   -0.40%     
==========================================
  Files         615      615              
  Lines       37639    38743    +1104     
==========================================
+ Hits        27693    28354     +661     
- Misses       9946    10389     +443     
Flag Coverage Δ
Debug 73.18% <66.66%> (-0.40%) :arrow_down:
integration 69.13% <57.14%> (-4.45%) :arrow_down:
production 73.18% <66.66%> (-0.40%) :arrow_down:
unit 73.56% <91.66%> (-0.01%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...r.PlatformServices/Discovery/AssemblyEnumerator.cs 95.04% <100.00%> (-3.19%) :arrow_down:
src/TestFramework/TestFramework/TestDataRow.cs 87.50% <50.00%> (-12.50%) :arrow_down:
....PlatformServices/Helpers/TestDataSourceHelpers.cs 76.72% <44.44%> (-2.02%) :arrow_down:

... and 34 files with indirect coverage changes

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov-commenter avatar Jul 15 '25 12:07 codecov-commenter