Test categories aren't propagated to vstest trx logs

[andrey-noskov]

With the latest version of the nuni3-adapter unit test descriptions in trx files created with vstest.console (2017) don't have TestCategory elements.

Expected behaviour

All categories assigned to test cases are reflected in TestCategory elements in trx logs, for example:

    <UnitTest name="ThatEmptyContainsHeaders" storage="d:\work\nunit3-vs-adapter\src\nunittestadaptertests\bin\debug\netcoreapp1.0\nunit.visualstudio.testadapter.tests.dll" id="5d22f9c8-13f7-4be4-b020-438dbf715a39">
      <TestCategory>
        <TestCategoryItem TestCategory="DumpXml" />
      </TestCategory>
      <Execution id="cf9ffa57-0013-4c9a-877e-5104b828b8ec" />
      <TestMethod codeBase="D:\work\nunit3-vs-adapter\src\NUnitTestAdapterTests\bin\Debug\netcoreapp1.0\NUnit.VisualStudio.TestAdapter.Tests.dll" executorUriOfAdapter="executor://nunit3testexecutor/" className="NUnit.VisualStudio.TestAdapter.Tests.DumpXmlTests" name="ThatEmptyContainsHeaders" />
    </UnitTest>

Repro steps

1. build the nunit3-vs-adapter repo;
2. invoke vstest.console with the following arguments. note the logger argument

"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" "D:\work\nunit3-vs-adapter\src\NUnitTestAdapterTests\bin\Debug\netcoreapp1.0\NUnit.VisualStudio.TestAdapter.Tests.dll" /logger:"trx" /TestAdapterPath:"D:\work\nunit3-vs-adapter\bin\Debug"

3. the test will spit out a new result trx-file. Open it in a text editor and note that none of the test cases have TestCategory items, for example:

    <UnitTest name="ThatEmptyContainsHeaders" storage="d:\work\nunit3-vs-adapter\src\nunittestadaptertests\bin\debug\netcoreapp1.0\nunit.visualstudio.testadapter.tests.dll" id="c3d0232c-20e9-44be-a6e9-c48b0b4ef694">
      <Execution id="7a1b8f5c-23fd-4562-832f-3febbda631f9" />
      <TestMethod codeBase="D:\work\nunit3-vs-adapter\src\NUnitTestAdapterTests\bin\Debug\netcoreapp1.0\NUnit.VisualStudio.TestAdapter.Tests.dll" executorUriOfAdapter="executor://nunit3testexecutor/" className="NUnit.VisualStudio.TestAdapter.Tests.DumpXmlTests" name="ThatEmptyContainsHeaders" />
    </UnitTest>

Additional thoughts

I believe it is an issue in the TrxLogger itself: https://github.com/Microsoft/vstest/blob/0341da456744f78bba8ff2a25c1cddfc89c87b08/src/Microsoft.TestPlatform.Extensions.TrxLogger/Utility/Converter.cs#L51 but given that today vstest executable is distributed with a VS only and thus can not be update in reasonable timeframes, would you mind to accept a simple compatibility patch to the adapter to fix the issue?

And vstest still doesn't support custom traits: https://github.com/Microsoft/vstest/issues/98

[OsirisTerje]

I've contacted the MS PG about this, let's see what they answer before we do any workarounds in the nunit adapter code.

[andrey-noskov]

any updates?

[rprouse]

@OsirisTerje did you ever hear back from anyone on this? Should we consider a workaround?

[OsirisTerje]

I'll ping them again.

[Frannsoft]

Hello, any update from MS on this? Ran into this while seeing if our team could switch from mstest v2 to nunit. Thanks.

[pvlakshm]

I don't think the NUnit TestCategory was ever propagated to the trx file. It is not propagated for NUnit3, but if you have a repro where it was propagated for an earlier version of NUnit, can you please share that with us?

[OsirisTerje]

@pvlakshm I dont think it ever has been propagated, so it is more a question of what we need to do to get it propagated properly. We can do the workaround in #507 but it seems strange that is the only support that vstest has for custom properties, and that the only category item it accepts is the MSTest category type (See links (from above) : https://github.com/Microsoft/vstest/issues/98 and . https://github.com/Microsoft/vstest/blob/0341da456744f78bba8ff2a25c1cddfc89c87b08/src/Microsoft.TestPlatform.Extensions.TrxLogger/Utility/Converter.cs#L51

[andrey-noskov]

folks, can we split the efforts and a. merge the workaround and unblock the devs who need the categories; b. work with TrxLogger devs on a more generic long-term solution;

[OsirisTerje]

@andreyns Will merge the workaround first, and include it with the 3.11 release. It is overdue, but still waiting for a few more fixes to be included.
Nice if you verify it using the dev feed.

[OsirisTerje]

Since we have merged the workaround, I'll close this issue. The workaround will be out with 3.11. When MS has fixed it on their side, we can remove the workaround, but it still doesn't matter. We don't need keeping this issue open for that.

[OsirisTerje]

This workaround doesnt work. With the workaround as is, it will produce duplicate traits in the Test Explorer. Enabling only MsTest format causes Vstest not to recognice any category.

[jnm2]

@OsirisTerje It would be nice to figure out what the problem is so that we can provide a property with the label TestCategory to show things in the TRX and at the same time provide a property with the label Category in order to allow folks to use the word Category in their filters.

https://github.com/nunit/nunit3-vs-adapter/issues/568 contains the info we currently need to solve, right?

[esrahofstede]

Are there any updates on this?

[OsirisTerje]

@esrahofstede This is a VSTest issue, and I haven't heard anything more about this. Could be wise if you would be so kind to raise issues on the vstest github repo and on the DevCommunity site. If you do, please add the links there.

[klis8]

@OsirisTerje was there any solution for this released or workaround?

[OsirisTerje]

We never heard back from @esrahofstede , haven't checked the behaviour since, but if you can check, that would be appreciated.

[klis8]

@OsirisTerje , unfortunately I discovered this issue now and I am looking also for a solution.

[OsirisTerje]

Ok, then it is confirmed, that is sort of good though.

Can you raise an issue on the VSTest git repo, and add a link here to that issue?

See that @andreyn-msft has been involved here earlier, perhaps he can help.

[klis8]

@OsirisTerje I think this was the relevant ticket https://github.com/microsoft/vstest/issues/4142