In Microsoft.Testing.Platform , using dotnet test, the testResults folder is not created, but it is created when run test from the TestExplorer of vs 2022

[moh-hassan]

Using vs 2022 v17.13.6 and TestExplorer in Microsoft.Testing.Platform, i can generate TestResults in xml format, but when run the commandline from solution folder

dotnet test 

The TestResults folder is not created or updated. I followed the instructions here

I find this link which list unsupported setting

https://learn.microsoft.com/en-us/dotnet/core/testing/microsoft-testing-platform-extensions-vstest-bridge RunConfiguration element The following RunConfiguration elements are not supported by Microsoft.Testing.Platform:

Why vs 2022 testExplorere can generate the testResults folder? What alternative /command options to generate the testResults in Microsoft.Testing.Platform using dotnet test

[OsirisTerje]

@Youssef1313 Comments ?

[moh-hassan]

I find that MsBuild property RunSettingsFilePath is not honored by 'dotnet test' in MTP mode, but honored in VSTest mode.

In the test project in NUnit, i created the property that set runsettings like:

			<RunSettingsFilePath>$(MSBuildThisFileDirectory)..\nunit.runsettings</RunSettingsFilePath>

when run dotnet test from within solution folder (without any options):

1. VSTest mode nunit.runsettings is loaded and the mytest.xml is created in TestResults folder in bin folder

2. Mtp mode

nunit.runsettings IS NOT loaded and the mytest.xml IS NOT created

Also, if i try to run from within solution folder in MTP, the option is ignored:

dotnet test --settings ./nunit.runsettings

The output in console is:

NUnit Adapter 5.0.0.0: Test execution started
Running all tests in C:\Users\mohha\source\repos\Issue1152\1-nunit-runner\bin\Debug\net8.0\nunit-runner.dll
  Test Output folder checked/created : C:\Users\mohha\source\repos\Issue1152\1-nunit-runner\bin\Debug\net8.0\TestResults
   NUnit3TestExecutor discovered 6 of 6 NUnit test cases using Current Discovery mode, Non-Explicit run
   Test results written to C:\Users\mohha\source\repos\Issue1152\1-nunit-runner\bin\Debug\net8.0\TestResults\nunit-runner.xml
NUnit Adapter 5.0.0.0: Test execution complete
  nunit-runner test succeeded (1.3s)

But really, Test results IS NOT written

The generated command in the log insures that no setting file is used, as shown below:

=== COMMAND LINE ===
C:\Users\mohha\source\repos\Issue1152\2-nunit-runner-dotnet-test\bin\Debug\net8.0\nunit-runner-dotnet-test.exe --internal-msbuild-node testingplatform.pipe.14a59c49500f4d9e872bbee504a9a48a 

I think this may be a bug

On the other side, Vs 2022 Test Explorere GUI, honour this property in both MTP and VsTest modes.

[OsirisTerje]

@moh-hassan Can you check this with the 5.1.0-alpha.3 version of the adapter ?

[moh-hassan]

Thanks @OsirisTerje for reply. I tried version 5.1.0-alpha.3 of the adapter. Using MTP mode, vs 2022 Test Explorer (TE): the runsetting file is loaded using the Msbuild property RunSettingsFilePath and the testResult.xml is generated as configured in runsetting file.

But, using the commandLine dotnet test without any option, the runsetting file was not loaded the and no generation of testResult.xml. The issue is still exist.

I find that RunSettingsFilePath is removed support in MTP mode in the project testfx I have posted an issue in testfx repo

The workaround solution To continue support using RunSettingsFilePath,in commandline and vs 2022 Test Explorer, i added this new MsBuild property in MTP in the project csproj file:

<TestingPlatformCommandLineArguments>$(TestingPlatformCommandLineArguments) settings $(RunSettingsFilePath)</TestingPlatformCommandLineArguments> 

This property save me from modifying the commandLine like:

dotnet test -- --settings  absolute/path/to/runsetting

Now I get testResult.xml in both vs 2022 TE and dotnet test (without setting runsetting file) in MTP mode.

It seem that in MTP mode, Nunit3Adapter is receiving runsettings continents as xml and process it and not accessing RunSettingsFilePath property.

[Youssef1313]

RunSettingsFilePath isn't supported by VSTestBridge. Is that the request here?

We have received this request quite few times, but it's a low value for us to support and is easy for users to proceed without it. A simple MSBuild line that adds --settings ... to TestingPlatformCommandLineArguments should work.

If the request here is that RunSettingsFilePath isn't working, this is expected and is by-design.

[OsirisTerje]

@Youssef1313 The issue here is that the TestREsults folder is not created.

[Youssef1313]

My understanding is that user is trying to specify it in runsettings and specifying the run settings via RunSettingsFilePath MSBuild property.

This is by-design. If the scenario os different, I need repro steps to be able to investigate.

[OsirisTerje]

@Youssef1313

I tried to set up a repro using your MTPPLaygorund as a base, but that doesnt compile outside the adapter project., It also doesn't match the MTP documentation - OR - you have changed it again.

I think you can repro this simply by running

dotnet test  --logger "trx" 

and do that for both an ordinary test project and then for an MTP based project. If I got this right, the ordinary one will place the logger file in the TestResults folder, whereas the MTP based one doesnt place it anywhere.

@moh-hassan Comments ?

[OsirisTerje]

@Youssef1313

I do have a repro now at https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue1265

The repro has two projects, one ordinary, and one with MTP.

The one ordinary works as it should.

The one with MTP crash

And produce actually a log file, but that file is placed in the bin folder, not in the project folder as for the ordinary project.

The log file indicates it is being run the wrong way.

mtp_net9.0_x64.log

[Youssef1313]

• dotnet test --logger "trx" is invalid for MTP.
• Main of MTP isn't registering the MSBuild extension using testApplicationBuilder.AddMSBuild. So it's very expected that it doesn't recognize --internal-msbuild-node.

[OsirisTerje]

@Youssef1313 So what is the correct syntax ? And, looking at the response above, given that the logger syntax is wrong, is that a good response ?

[Youssef1313]

--logger trx will be completely ignored for MTP. We are changing that in MTP v2 so that it's an error instead of being ignored. The right way is:

1. If you are using the "VSTest" dotnet test with MTP, you need dotnet test -- --report-trx.
2. If you are using the new MTP dotnet test in .NET 10 SDK (opt-in via dotnet.config), you need dotnet test --report-trx

See https://learn.microsoft.com/dotnet/core/testing/unit-testing-with-dotnet-test for info.

[OsirisTerje]

Can these settings be added to a runsettings file ?

[Youssef1313]

I don't think it's possible today to configure TRX from runsettings. Runsettings isn't really a first-class citizen of MTP, and MTP core doesn't know about runsettings. It's only the VSTestBridge partially supporting it.

[OsirisTerje]

Are there any other way to configure this or is the command line parameter the only way ?

and

The reporter @moh-hassan said:

On the other side, Vs 2022 Test Explorere GUI, honour this property in both MTP and VsTest modes.

So does the Test Explorer do something special here ?

[Youssef1313]

@peterwald @drognanar for the Test Explorer question. It's likely reading the property from project evaluation and turning it into the env variable that holds runsettings. I don't think Test Explorer should do that for MTP actually.

The right thing to do is to adjust TestingPlatformCommandLineArguments MSBuild property, as @moh-hassan figured out already.