[Bug]: Playwright MSTest.Sdk Uses MSTest Runner, Ignoring runsettings Configuration / Microsoft.Testing.Platform

[terius7]

Version

1.50.0

Steps to reproduce

When using Playwright version 1.50.0 with MSTest.Sdk version 3.8.1, the MSTest runner is used instead of the expected VSTest runner. As a result, the Playwright configuration section in the runsettings file is ignored. Key settings such as BrowserName, ExpectedTimeout, Headless, and Channel do not work.

However, when <UseVSTest>true</UseVSTest> is explicitly set in the .csproj file, and the VSTest runner is used, the Playwright settings in the runsettings file work as expected.

Expected behavior

• Playwright should respect the runsettings file when running tests with MSTest.Sdk.
• Playwright-specific settings such as BrowserName, ExpectedTimeout, Headless, and Channel should be applied correctly, regardless of whether MSTest or VSTest is used.

Actual behavior

• With MSTest runner, the Playwright settings in runsettings are ignored.
• With VSTest runner (<UseVSTest>true</UseVSTest>), the settings work as expected.

Additional context

This issue suggests that MSTest.Sdk is defaulting to the MSTest runner instead of the VSTest runner, which does not properly load the Playwright settings from runsettings. It would be helpful if Playwright could support runsettings consistently across different test runners.

Would appreciate any insights or fixes on this issue. Thanks!

Environment

- Playwright Version: 1.50.0
- MSTest.Sdk Version: 3.8.1
- .NET Version: net8.0

[mxschmitt]

Sounds similar to the issues we are encountering with https://github.com/microsoft/playwright-dotnet/issues/3093. We probably need to change how the config is getting parsed from a TestAdapter to something else. Or plumb the values through the other process somehow.

[0xced]

Sounds like another argument in favor of #3081. 😉

[Youssef1313]

You can probably plug your own configuration provider via TestingPlatformBuilderHook. I think you should be able to get it to work with MTP that way.

[alexaka1]

@Youssef1313 Can you elaborate on that? A quick search of the docs, I didn't find anything I could start with.

[Youssef1313]

It's supposed to be in https://learn.microsoft.com/dotnet/core/testing/microsoft-testing-platform-architecture-extensions but looks like the doc is missing some stuff, partially tracked by https://github.com/microsoft/testfx/issues/5552.

In general though, TestingPlatformBuilderHook is an MSBuild item that lets Microsoft.Testing.Platform entry point generation inject extension registration.

Then for a custom configuration provider, you can do something similar to this:

https://github.com/microsoft/testfx/blob/093c672c4dc15c0d4cbbd84c85e0a43774aa82c6/src/Platform/Microsoft.Testing.Extensions.VSTestBridge/Helpers/TestApplicationBuilderExtensions.cs#L36, or it might even be better to just get the IConfiguration from IServiceProvider and read settings from it right away, without providing a whole custom configuration provider. But really depends on your actual needs and I didn't look deeply in it.