Can't list full test names from a DLL.

[JunielKatarn]

Description

IMPORTANT: if the defect is reproduced only in a workflow from within the Visual Studio IDE then do not report the issue here - instead, please report it using Visual Studio's "Send Feedback" option that can be accessed from the Help menu OR using this link https://developercommunity.visualstudio.com.

For a defect reproducable from the vstest command line, describe the issue you've observed. There is no way to list all tests with their fully qualified name in the command line.

Steps to reproduce

What steps can reproduce the defect? Please share the setup, commandline for vstest.console, sample project, target framework etc. For a C++ test DLL (Microsoft Unit Test Framework for C++), run:

vstest.console.exe <target DLL> --ListFullyQualifiedTests

This will fail with message:

Target Path should be specified for listing FQDN tests!

Expected behavior

Share the expected output (List of FQDN tests in the DLL).

Actual behavior

What is the behavior observed? The command forces to print the output to a file using switch --ListTestsTargetPath. This seems enforced by this source line: https://github.com/Microsoft/vstest/blob/master/src/vstest.console/Processors/ListFullyQualifiedTestsArgumentProcessor.cs#L225.

Diagnostic logs

Please share test platform diagnostics logs. Instructions to collect logs are here.
The logs may contain test assembly paths, kindly review and mask those before sharing.

Environment

Please share additional details about the test environment. Operating system, Build version of vstest.console Windows 10, x64 Version 10.0.17134 Build 17134 VSTest, version 15.7.2

[singhsarab]

This was meant for internal consumption only and is not even documented anywhere. The current behavior is by design.

@pvlakshm Can you please triage this.

[pvlakshm]

Yes, this is by design, and I suppose we can live with it.

[JunielKatarn]

It's not a matter of "live with". It's a feature available in virtually every test framework, missing on this one.

Would it be too hard to implement for VSTest? Does this pose security issues? If it were truly by design, why does Visual Studio not have a problem displaying the tests with their fully qualified name?

[phatcher]

It also makes it non-obvious to locate test failures in VSTS - where I'm told the test "Add" has failed out a selection of 150 other methods called "Add" :-)

[pvlakshm]

The following would work: vstest.console abc.dll /ListFullyQualifiedTests /ListTestsTargetPath:abc.txt

[pvlakshm]

I will close this - let me know if there are any concerns.

[JunielKatarn]

@pvlakshm What channel should I reach if I want to address this at a development level?

I fail to understand how needing to create a file just to get the full test names is a "design" decision. Moreover, as @phatcher says, running tests (C++) forces the developer to give redundant and long names because the test class or name space just won't be output.

Is this also by design?

[cltshivash]

@JunielKatarn We show the display name of the tests when executed using vstest.console.exe. Visual Studio does show the same display name but under a hierarchy of namespace/classname that helps in distinguishing the test methods. ListFullyQualifiedName option is something that we want to remove at some point. Are you looking for an option to just display the FullyQualifiedName ? If you can let us know the scenario where you needed it, it would help us provide a more appropriate response.

@phatcher VSTS does have an option of grouping the tests under a container. I presume your classes with the same test methods are all in the same container ? What is the C++ test framework that you are using ?

[phatcher]

@cltshivash We are using both NUnit3 for dotnet and GoogleTest for C++ code.

I would like the ability to cluster the tests under assembly + namespace + class as per Visual Studio, I'm not sure what you mean by container, I can see options for Test Suite, but we are not using that so we get something like this...

I know this should be under the vsts-task but without the FQN being emitted from vstest I don't see how they would have enough information to be able to produce anything better

[JunielKatarn]

@cltshivash The main issue is not listing the tests, but running them. Even if the switch to list the existing tests is to be phased out, running C++ tests (Microsoft Unit Test Framework) also shows the test name without the containing test class or namespace prefix.

Not all workflows, especially automated ones can rely on Visual Studio to tell the fully qualified test name. We need to be able to differentiate two tests by namespace or test class at runtime.

Can this be currently achieved, or is this a bug?

[cltshivash]

@phatcher The UI for the test tab shows the test results grouped under the test file (in case of C++ it would be the dll). The name displayed is the DisplayName of the test case. Grouping based on namespace and class is not available in Azure Devops (VSTS). I would forward this to the concerned product owner and would recommend to open developer community ticket for this feature so that others can pitch in as well if they need it.

P.S: The FQN information is stored on the service today. It is the information used to identify the test case.

[cltshivash]

@JunielKatarn You can selectively run tests specifying the FQN as filter. https://github.com/Microsoft/vstest-docs/blob/master/docs/filter.md. Do let us know if this isn't sufficient.

[JunielKatarn]

(Retaking this issue).

Thanks, @cltshivash. This command works for me to run an individual test:

vstest.console.exe /TestCaseFilter:"FullyQualifiedName=Namespace1::TestClass2::TestMethod1" .\x64\Debug\SampleTest.dll

[guillaume86]

@phatcher I'm curently evaluating moving from TeamCity to Azure Pipelines. I'm hitting the same issue regarding the tests results tab. The UX is abysmal compared to TeamCity, but being able to group by and see namespaces would make it bearable.

Did you find a solution? Not sure if the root issue is here or if the Azure team has already the information but is not making it available in the tests tab (and the issue should be reported to them).

[Perrow]

I'm using this test-runner to verify versions of our product with a set of tests that runs against a dedicated environment. Like others have reported we have multiple tests with similar or identical names.

We run the test from the command line on a dedicated machine and the report is presented in a web interface.

Having the ability to have the fully qualified name reported to the command line seems like a logical option. The fact that most other test frameworks has this option verifies this in my opinion.

[doublehelix]

Hate to dredge up this old issue, but it's 2023.... We have more than 10K automated tests. Finding them absolutely requires knowing the full name of the test including namespace.. there are hundreds (approaching thousands) with identical or similar names of the function or value the test represents. Out of namespace context the names mean nothing.

When using MSTEST, we get the following output:

  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.Comment1
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.Comment2
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.Comment3
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.Comment4
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.CommentDoesntWork
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.CommentNotWork1
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.CommentNotWork2
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.CommentWhiteSpaces
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.Uncomment
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.Uncomment2
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.Uncomment3
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.Uncomment4
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.UncommentDoesntWork
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.UncommentShouldNotWotk
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.UncommentShouldNotWotk2
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSendGrid.UncommentWhiteSpaces
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.Comment1
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.Comment2
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.Comment3
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.Comment4
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.CommentComplex
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.CommentDoesntWork
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.CommentNotWork1
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.CommentNotWork2
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.CommentWhiteSpaces
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.Uncomment
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.Uncomment2
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.Uncomment3
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.Uncomment4
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.UncommentComplex
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.UncommentDoesntWork
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.UncommentShouldNotWotk
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.UncommentShouldNotWotk2
  Passed                Install.Test.Config.ConfigurationUpdaterTests.CommentUncommentSmtp.UncommentWhiteSpaces

However, when using vstest.console we get the arguable less useful: (tests running in parallel run in different order)

    Passed UncommentDoesntWork [< 1 ms]
    Passed Uncomment [< 1 ms]
    Passed Uncomment2 [< 1 ms]
    Passed Uncomment3 [< 1 ms]
    Passed Uncomment4 [< 1 ms]
    Passed UncommentShouldNotWotk [< 1 ms]
    Passed UncommentShouldNotWotk2 [< 1 ms]
    Passed UncommentWhiteSpaces [< 1 ms]
    Passed CommentDoesntWork [< 1 ms]
    Passed CommentNotWork1 [< 1 ms]
    Passed CommentNotWork2 [< 1 ms]
    Passed Comment1 [< 1 ms]
    Passed Comment2 [< 1 ms]
    Passed Comment3 [< 1 ms]
    Passed Comment4 [< 1 ms]
    Passed CommentWhiteSpaces [< 1 ms]
    Passed UncommentDoesntWork [< 1 ms]
    Passed Uncomment [< 1 ms]
    Passed Uncomment2 [< 1 ms]
    Passed Uncomment3 [< 1 ms]
    Passed Uncomment4 [< 1 ms]
    Passed UncommentComplex [< 1 ms]
    Passed UncommentShouldNotWotk [< 1 ms]
    Passed UncommentShouldNotWotk2 [< 1 ms]
    Passed UncommentWhiteSpaces [< 1 ms]
    Passed CommentDoesntWork [< 1 ms]
    Passed CommentNotWork1 [< 1 ms]
    Passed CommentNotWork2 [< 1 ms]
    Passed Comment1 [< 1 ms]
    Passed Comment2 [< 1 ms]
    Passed Comment3 [< 1 ms]
    Passed Comment4 [< 1 ms]
    Passed CommentWhiteSpaces [< 1 ms]
    Passed CommentComplex [< 1 ms]

I know we can dump the names of the tests to an output file using the switches intended for internal use, but they don't have the status/result so are even less useful.

[nohwnd]

When using MSTEST, we get the following output:

MSTEST meaning mstest.exe?

[nohwnd]

@doublehelix you should be able to do this in mstest using this workaround: https://github.com/microsoft/testfx/issues/410#issuecomment-623384833