vstest icon indicating copy to clipboard operation
vstest copied to clipboard

Test discovery problem when single quote is used in F# method name

Open VasilSirakov opened this issue 2 years ago • 1 comments

Description

When specifying a name for a test method with the double backtick syntax, adding a single quote is allowed by the compiler. This, however, causes some issue with the test discovery and it prevents not only the particular test from being discovered, but also all other tests in the test file defined below it.

Running dotnet test outputs the following warning:

[MSTest][Discovery][C:\path\to\solution\SingleQuoteTestName\bin\Debug\net7.0\SingleQuoteTestName.dll] MSTestAdapter failed to discover tests in class 'SingleQuoteTestName.TestClass' of assembly 'C:\path\to\solution\SingleQuoteTestName\bin\Debug\net7.0\SingleQuoteTestName.dll' because Whitespace is not valid in a ManagedName (pos: 19).

Given this warning it is hard to narrow it down immediately to the single quote being the problem. If you have many other tests, it is easy to miss this as the test run will not fail, rather it will just run the tests it managed to discover and succeed if those pass.

In my initial attempts to reproduce this issue, it seemed like having an odd number of single quotes was causing the issue whereas an even number was OK. After some further experiments, I unfortunately couldn't really narrow it down to some fixed pattern for when it happens and when it doesn't happen.

Steps to reproduce

Simply create a blank F# test project with no other dependencies and paste this in the Tests.fs file.

namespace SingleQuoteTestName

open System
open Microsoft.VisualStudio.TestTools.UnitTesting

[<TestClass>]
type TestClass () =

    [<TestMethod>]
    member _.``test that doesn't pass`` () =
        Assert.IsTrue(false);

    [<TestMethod>]
    member _.``test that passes`` () =
        Assert.IsTrue(true);

Then, run dotnet test and observe the output.

Expected behavior

All tests with names accepted by the compiler are discovered and executed.

Actual behavior

The single quote in the name of the first test method is preventing it from being discovered as well as all other tests in this file.

Environment

Replicated this both on .NET 6 (6.0.13) & .NET 7 (7.0.2).

AB#1819417

VasilSirakov avatar Feb 17 '23 12:02 VasilSirakov

I can repro, it is not just the presence of ', (e.g. ' works, 'hello' works. But 'a does not work, or ' , or ' '.

nohwnd avatar Feb 27 '23 13:02 nohwnd