[BUG] Setting headless to false in .runsetttings does nothing when using .NET 4.8

[sakull]

Context:

• Playwright Version: Microsoft.Playwright.MSTest - v1.27.1
• Operating System: Windows
• .NET version: net4.8
• Browser: chromium
• IDE: VS 2022

Code Snippet

https://github.com/sakull/mstest-runsettings-headless

Describe the bug

When running the tests on .NET 4.8, headless option is not read, .NET 6 works just fine.

[Prinsn]

Does not do anything with the standard installation instructions for a .NET 6 project, either.

I'm just following them verbatim for minimal spinup to validate that it works better than Selenium and I added it to our current runsettings file and it makes no difference.

However, it might be being used, because the test passes without setting the options.

After adding the settings:

  Message: 
Microsoft.Playwright.PlaywrightException : Browser closed.
==================== Browser output: ====================
<launching> C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe --disable-field-trial-config --disable-background-networking --enable-features=NetworkService,NetworkServiceInProcess --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=ImprovedCookieControls,LazyFrameLoading,GlobalMediaControls,DestroyProfileOnBrowserClose,MediaRouter,DialMediaRouteProvider,AcceptCHFrame,AutoExpandDetailsElement,CertificateTransparencyComponentUpdater,AvoidUnnecessaryBeforeUnloadCheckSync,Translate --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --disable-sync --force-color-profile=srgb --metrics-recording-only --no-first-run --enable-automation --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --no-sandbox --user-data-dir=C:\Users\MILLERJM\AppData\Local\Temp\playwright_chromiumdev_profile-5uqRWy --remote-debugging-pipe --no-startup-window
<launched> pid=5984
[pid=5984][err] [5984:19992:1109/125817.952:ERROR:device_event_log_impl.cc(215)] [12:58:17.952] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)

  Stack Trace: 
Connection.InnerSendMessageToServerAsync[T](String guid, String method, Object args) line 163
Connection.WrapApiCallAsync[T](Func`1 action, Boolean isInternal) line 482
BrowserContext.NewPageAsync() line 255
PageTest.PageSetup() line 37
GenericAdapter`1.BlockUntilCompleted()
NoMessagePumpStrategy.WaitForCompletion(AwaitAdapter awaiter)
AsyncToSyncAdapter.Await(Func`1 invoke)
SetUpTearDownItem.RunSetUpOrTearDownMethod(TestExecutionContext context, IMethodInfo method)
SetUpTearDownItem.RunSetUp(TestExecutionContext context)
<.ctor>b__0(TestExecutionContext context)
<>c__DisplayClass1_0.<Execute>b__0()
DelegatingTestCommand.RunTestMethodInThreadAbortSafeZone(TestExecutionContext context, Action action)

Also fails from dotnet test -- Playwright.BrowserName=chromium Playwright.LaunchOptions.Headless=false Playwright.LaunchOptions.Channel=msedge

[nohwnd]

I see that the options are used on net7 But not net48. Net7 test passes (and a window flashes on the screen) the other test fails.

I've just modified the csproj in the example above to build for both TFMs:

  <PropertyGroup>
    <TargetFrameworks>net4.8;net7.0</TargetFrameworks>
  </PropertyGroup>

And then run: dotnet test --settings .\.runsettings

[nohwnd]

@mxschmitt Running with --diag:logs\log.txt I can see that your runsettings provider is loaded in both cases, and in both cases the runsettings are passed down to including the playwright section.

30380, 3, 2022/11/10, 10:10:05.664, 2320259055406, testhost.net48.exe, SettingsProviderExtensionManager: Loading settings provider Microsoft.Playwright.TestAdapter.PlaywrightSettingsProvider

Debugging your actual implementation, to see what is actually passed there.

[nohwnd]

Sooo there are 2 problems happening. The original issue is caused by sharing the settings via static fields which does not happen to be shared in .NET Framework because there we use AppDomains, and the intialization runs in a different appdomain than the execution.

If you disable Appdomains by adding the setting to your runsettings you will run into another bug:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <RunConfiguration> 
    <DisableAppDomain>True</DisableAppDomain>
  </RunConfiguration>
  <Playwright>
    <BrowserName>chromium</BrowserName>
    <LaunchOptions>
      <Headless>false</Headless>
    </LaunchOptions>
  </Playwright>
</RunSettings>

A total of 1 test files matched the specified pattern.
No test is available in S:\c\mstest-runsettings-headless\bin\Debug\net4.8\PlaywrightTests.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again.

This is because adapters don't normally reference Microsoft.TestPlatform.ObjectModel dependency and only use it as build time reference. But Microsoft.Playwright.TestAdapter does, and the loading order of dlls prefers the user folder over the folder of the runner for this dependency. So if your dotnet test comes from net7, it expects object model 17.4.0 or newer, but you load 17.3.2 that Playwright.TestAdapter carries.

To work around this you can install the dependency directly into your csproj:

    <PackageReference Include="Microsoft.TestPlatform.ObjectModel" Version="17.4.0" />

So to work around this take the two steps above.

[nohwnd]

For reference:

# file .runsettings
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <RunConfiguration> 
    <DisableAppDomain>True</DisableAppDomain>
  </RunConfiguration>
  <Playwright>
    <BrowserName>chromium</BrowserName>
    <LaunchOptions>
      <Headless>false</Headless>
    </LaunchOptions>
  </Playwright>
</RunSettings>

<!-- file PlaywrightTests.csproj -->
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFrameworks>net4.8;net7.0</TargetFrameworks>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.4.0" />
    <PackageReference Include="Microsoft.TestPlatform.ObjectModel" Version="17.4.0" />
    <PackageReference Include="Microsoft.Playwright.MSTest" Version="1.27.1" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.2.10" />
    <PackageReference Include="MSTest.TestFramework" Version="2.2.10" />
    <PackageReference Include="coverlet.collector" Version="3.1.2" />
  </ItemGroup>

</Project>