Tests results not aggregating result for each targetFrameworks

[livarcocc]

From @bhugot on June 20, 2017 9:50

Steps to reproduce

Create a test project with 2 targetFrameworks and 1 Test

run Dotnet Test --logger "trx;LogFileName=abc.trx"

Expected behavior

The file abc.trx should contains 2 lines, one for each target framework

Actual behavior

There is only one line for the last targetFramework used.

Environment data

dotnet --info output: .NET Command Line Tools (2.0.0-preview2-006127)

Product Information: Version: 2.0.0-preview2-006127 Commit SHA-1 hash: 946ea7980a

Runtime Environment: OS Name: Windows OS Version: 6.1.7601 OS Platform: Windows RID: win7-x64 Base Path: C:\Users\bhugot100316.dotnet\x64\sdk\2.0.0-preview2-006127\

Microsoft .NET Core Shared Framework Host

Version : 2.0.0-preview1-002111-00 Build : 1ff021936263d492539399688f46fd3827169983

Copied from original issue: dotnet/cli#6951

[codito]

dotnet test is overriding the previous trx log file. Merging the log files will break the schema since it supports a single test run today.

Can we suffix the target framework to trx filename for a MTFM project if user provides a log filename? E.g. .net46.trx and .netcoreapp1.0.trx

[jnm2]

Another issue I'm having today is that AppVeyor ignores all but the first trx file. Perhaps this is because the tests are identically named? Right now dotnet test's TRX can't be used in any form with AppVeyor if your tests target multiple frameworks. What would you do to solve that?

[mayankbansal018]

@pvlakshm Pratap, it seems to me that we need to spec it out,

• If users provides trx file name, then we will overwrite the 1st trx file,
• If no file name is created then we generate 2 separate trx file, each containing data of individual run based on target frameworks.

Unfortunately we cannot re-use trx file from 1st run, & append the data for 2nd. As proposed by Arun we can rename trx file by appending the targetframework name, or we can show user a clear warning that his trx file will be overwritten.

[mayankbansal018]

While overwriting the message we show is "WARNING: Overwriting results file: C:\Users\maban\source\repos\UnitTestProject32\UnitTestProject32\TestResults\abc.trx"

[pvlakshm]

Removing the bug tag, and marking it an enhancement. We will use this as input when we do planning for Q2 of next year.

[GraemeF]

Same problem occurs with multiple test projects. For example, I have an IntegrationTests.sln with two projects in it so that I can easily run them in a single command, however the second project gives the WARNING: Overwriting results file message and overwrites the first project's results.

[sherland]

@pvlakshm It's now more than two years later. Can you share some details about when we can expect a fix/ enhancement on this?

Possibly related: #2113

[singhsarab]

We tried to resolve this issue via https://github.com/microsoft/vstest/pull/2140 Please refer to description for the usage.

[Jsparham777]

@singhsarab Even using the prefix switch I still get the overwrite warning. This is due to the resolution of the timestamp. Is there a way to insert the test project name into the filename? This would stop it from overwriting or is there another way to overcome this?

[nohwnd]

@Jsparham777 Medeni was fixing this in https://github.com/microsoft/vstest/pull/2508 which is shipped in 16.8.

[jnm2]

AltCover is cool. When I run dotnet test /p:AltCover=true with AltCover in the test project, it outputs coverage.netcoreapp3.1.xml and coverage.net5.0.xml. (Automatically inserting each target framework before the file extension.) Why can't .trx output be more like AltCover?

[Prodigio]

I have a similair issue when running following command

dotnet test --filter "Name!~IntegrationTest&Name!~Base&Name~Test"
            --logger "trx;LogFileName=TestResults_Unit.xml"
            --results-directory "C:\output\TestResults"

This overrides the test file on each test run. Is there a way to specify that each test run should create a new test result file? Or can I specify the file type when using LogFilePrefix?

[Haplois]

@Prodigio, which version of the Test Platform are you using? This issue should be fixed by #2508 and shipped with v16.8 as mentioned by @nohwnd.

[Prodigio]

@Prodigio, which version of the Test Platform are you using? This issue should be fixed by #2508 and shipped with v16.8 as mentioned by @nohwnd.

@Haplois I was using v16.7.1 when I wrote the first post, but updated to v16.8.3 recently. Still the file get's overriden on every test execution. Is there a way to tell dotnet test to append a date/time at least?

To give a bit more insight: dotnet test is executed by running dotCover analysis on TeamCity.

In the end, this is the command which get's executed:

\.nuget\packages\jetbrains.dotcover.commandlinetools\2020.3.2\tools\dotCover.exe cover
--Output=D:\<application>\output\CodeCoverage\Report.dotCover.html
--ReportType=Html
--TargetExecutable="C:\Program Files\dotnet\dotnet.exe"
--TargetArguments="test --filter \"Name!~IntegrationTest&Name!~Base&Name~Test\"
                        --logger \"trx;LogFileName=TestResults_HTML.xml\"
                        --results-directory \"D:\<application>\output\TestResults\""
                        --TargetWorkingDir=D:\<application>
                        --InheritConsole=True
                        --AnalyseTargetArguments=True
                        --Filters="-:module=Test.*;-:module=IntegrationTest.*;-:module=Prism;-:module=NUnit3.TestAdapter;-:module=MoreLinq;-:module=*.Contract;-:module=*.Contracts;-:module=*.Bootstrapper;-:method=ToString;-:method=GetHashCode;-:method=MoveNext;"
                        --LogFile=D:\<application>\output\CodeCoverage\dotCoverLog_HTML.txt

[Haplois]

This overrides the test file on each test run. Is there a way to specify that each test run should create a new test result file? Or can I specify the file type when using LogFilePrefix?

It's possible not to customize the extension when using the LogFilePrefix option.

In the end, this is the command which get's executed

When you specify the exact filename using the LogFileName option, the TRX log gets overridden if it's already there. You need to use the LogFilePrefix option to keep the log file from getting overridden, unfortunately as I said, we don't support customizing the extension in this case.