[MTP] Error when VSTest arguments are passed via dotnet test when they will be ignored

[Copilot]

This PR implements validation to show errors when VSTest-specific command-line switches are used with Microsoft Testing Platform (MTP), helping users understand why their VSTest properties are being ignored.

Problem

Users were confused when using VSTest-specific properties with dotnet test while using MTP, because these properties get silently ignored. For example:

• dotnet test --filter something - filter is ignored
• dotnet test --logger trx - logger is ignored

This led to users wondering why their test filtering or logging wasn't working as expected.

Solution

Added comprehensive validation that produces an error when VSTest-specific properties are set with MTP:

Key Changes

1. New opt-out property: TestingPlatformIgnoreVSTestProperties (default: false) allows users to suppress the error when needed.

2. Validation target: Added _ValidateVSTestProperties target that checks for all VSTest properties mentioned in the issue:

   • VSTestSetting, VSTestListTests, VSTestTestCaseFilter, VSTestTestAdapterPath
   • VSTestLogger, VSTestDiag, VSTestResultsDirectory, VSTestCollect
   • VSTestBlame, VSTestBlameCrash, VSTestBlameHang

3. Clear error message: Provides helpful guidance including:

   • Lists which VSTest properties are set
   • Explains they will be ignored with MTP
   • Shows how to suppress the error
   • Links to migration documentation

4. Integration with existing workflow: The validation runs as part of the InvokeTestingPlatform target, ensuring it catches issues early in the build process.

Example Error Message

VSTest-specific properties are set but will be ignored when using Microsoft Testing Platform. 
The following properties are set: VSTestTestCaseFilter; VSTestLogger; VSTestBlame. 
To suppress this error, set TestingPlatformIgnoreVSTestProperties to true. 
For more information about migrating from VSTest to Microsoft Testing Platform, see: https://docs.microsoft.com/en-us/dotnet/core/testing/unit-testing-mstest-intro

Usage Examples

Error case:

dotnet test --filter Category=Unit --logger trx
# Now shows clear error explaining the issue

Opt-out:

dotnet test --filter Category=Unit -p:TestingPlatformIgnoreVSTestProperties=true
# Suppresses the error for migration scenarios

Project file opt-out:

<PropertyGroup>
  <TestingPlatformIgnoreVSTestProperties>true</TestingPlatformIgnoreVSTestProperties>
</PropertyGroup>

Testing

Added comprehensive integration tests covering:

• Error when VSTest properties are set
• No error when opt-out property is enabled
• No error when no VSTest properties are set

Breaking Change Notice

This is a breaking change as it will now show errors for previously "working" (but ignored) VSTest property usage. However, this is intentional to help users identify and fix configuration issues. Users can opt-out during migration using the TestingPlatformIgnoreVSTestProperties property.

Fixes #5697.

[!WARNING]

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

• 8xbvsblobprodcus382.vsblob.vsassets.io
  • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)
• bcnvsblobprodcus378.vsblob.vsassets.io
  • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)
• http://168.63.129.16:80/machine/
  • Triggering command: /usr/bin/python3 -u bin/WALinuxAgent-2.13.1.1-py3.9.egg -collect-logs (http block)
• i1qvsblobprodcus353.vsblob.vsassets.io
  • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)
• l49vsblobprodcus358.vsblob.vsassets.io
  • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)
• mfjvsblobprodcus373.vsblob.vsassets.io
  • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)
• s4uvsblobprodcus326.vsblob.vsassets.io
  • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)
• s8mvsblobprodcus38.vsblob.vsassets.io
  • Triggering command: dotnet build src/Platform/Microsoft.Testing.Platform.MSBuild/Microsoft.Testing.Platform.MSBuild.csproj --configuration Release --verbosity minimal (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to my firewall allow list

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[Copilot]

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

[!WARNING]

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

• http://168.63.129.16:80/machine/
  • Triggering command: /usr/bin/python3 -u bin/WALinuxAgent-2.13.1.1-py3.9.egg -collect-logs (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to my firewall allow list

[Copilot]

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

[!WARNING]

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

• http://168.63.129.16:80/machine/
  • Triggering command: /usr/bin/python3 -u bin/WALinuxAgent-2.13.1.1-py3.9.egg -collect-logs (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to my firewall allow list

[codecov-commenter]

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 76.37%. Comparing base (37692a0) to head (9cc0bd3). :warning: Report is 467 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5698      +/-   ##
==========================================
+ Coverage   73.89%   76.37%   +2.47%     
==========================================
  Files         602      602              
  Lines       36850    36850              
==========================================
+ Hits        27231    28144     +913     
+ Misses       9619     8706     -913     

Flag	Coverage Δ
Debug	76.37% <ø> (+2.47%)	:arrow_up:
integration	76.37% <ø> (+2.47%)	:arrow_up:
production	76.37% <ø> (+2.47%)	:arrow_up:
unit	76.36% <ø> (+2.47%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more. see 59 files with indirect coverage changes

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Youssef1313]

Drafting, this will break DotNetCoreCLI@2. We need to discuss.

[Copilot]

Drafting, this will break DotNetCoreCLI@2. We need to discuss.

You're absolutely right - the validation will break existing DotNetCoreCLI@2 tasks that pass VSTest properties. Should I modify the implementation to be less aggressive (e.g., warning instead of error, or different default behavior) or would you prefer to discuss the approach first before I make further changes?

[Youssef1313]

Closing in favor of #5722

[Youssef1313]

Going to take it as a "warning" in 3.11/1.9

[Youssef1313]

/backport to rel/3.11

[github-actions[bot]]

Started backporting to rel/3.11: https://github.com/microsoft/testfx/actions/runs/18141335925

[Youssef1313]

We probably don't need to merge it to main at all. Merging to 3.11 might be just sufficient.