Test Explorer doesn't support tests with `TestCaseAttribute`

[kojo12228]

Describe the bug

Test explorer doesn't work for NUnit tests that use TestCaseAttribute.

• Test builds and runs (.NET process running test ends)
• VSCode spinning icon does not go away until test is cancelled

Steps to reproduce

1. Create a new NUnit test project (dotnet new nunit -lang F#)
2. Open directory in Visual Studio Code
3. In UnitTest1, replace [<Test>] with [<TestCase("test")]
4. Select Run test in Test Explorer, or ▶️ in gutter

Expected behaviour

VSCode should correctly show the status of the test (whether succeeded or failed). Similarly, the Test Explorer should break down those test cases, rather than just showing one test.

Screenshots

Machine info

• OS: Windows
• .NET SDK version: 6.0.302
• Ionide version: 7.0.0

---

Frustratingly, I couldn't find anything in the logs that was logged as Debug in TestExplorer.fs, otherwise I might have found it easier to find out what parsing of tests/test results is failing. I would like to take a stab at trying to fix this, just depends on time (and whether I get distracted by other projects).

[kojo12228]

Ah, taking a look at Ionide.trx after running dotnet test <project>.fsproj --no-restore --logger:"trx;LogFileName=Ionide.trx" --noLogo gives me much better idea of what's going wrong.

[kojo12228]

Taking a deeper look, I don't fully understand where Ionide gets the tests method names, but for test cases, the .trx file has <method-name>("<test-case>") which doesn't sync up.

dotnet test --list-tests doesn't print out the namespace/class for the test for some reason. I see .NET Test Explorer uses vstest, but dotnet test is meant to replace it, and the output isn't exactly fun to parse.

The simplest solution would be to replace the tests in the Test Explorer tree view with the test cases once the .trx file has been parsed.

[kojo12228]

Turning out to be a lot more complicated than I thought:

TestCase takes any number of arguments. The following is example with 3:

[<TestCase("arg1", 2, 3L)>]
let Test1 (arg1, arg2, arg3) =

Leads to the following name in the .trx file, which will be fun to parse: Test1<String,Int32,Int64>("arg1",2,3L).

Looking in FsAutoComplete, it feels like this will require quite a bit of rejigging to support test cases (and I don't know how XUnit or Expecto handle similar scenarios).

[backerman]

I'm getting this behavior when using TestAttribute and not TestCaseAttribute; there's a rejected promise five stack frames into internal Fable stuff that appears to be from TestExplorer.fs:144:

let executionId = Map.find t.FullName testDefinitions

where e.g. t.FullName is PoudriereC2.Tests.MediaType.MediaTypeHandlerTests.TestGoodValues and the actual map key is PoudriereC2.Tests.MediaType+MediaTypeHandlerTests.TestGoodValues.

Is there a way to set breakpoints and such in the F# rather than the generated JavaScript?

[baronfel]

There is, and at one point it was working, but our sourcemaps are broken or something. I've been wanting to move us to esbuild and off of webpack partially because in my quick tests sourcemaps just worked, which is a HUGE step up from the current state.

[baronfel]

Oh! regarding the + in the computed name - that's what a type name for a type inside a module looks like - I believe the format is <NAMESPACE>+<MODULE>.<TYPE>. So we may have something off in our detection/calculation of module-embedded vs namespace-level tests?

[kojo12228]

The first three examples below match Chet's assessment (I wonder if there is a limit to module nesting), fourth example is a module with the same name as a class in a namespace:

module Outer =
    [<TestFixture>]
    module Inner =
        [<Test>]
        let Test1 () =
            Assert.Pass ()

        module Innermost =
            [<Test>]
            let Test2 () =
                Assert.Pass()

module Outer2 =
    [<TestFixture>]
    type InnerClass() =
        [<Test>]
        member this.Test1 () =
            Assert.Pass()

type SharedName() =
    do ()

[<TestFixture>]
module SharedName =
    [<Test>]
    let Test1 () =
        Assert.Pass()

Test Class Name	FullName in Code
TestNameSpace.Outer+Inner.Test1	TestNameSpace.Outer.Inner.Test1
TestNameSpace.Outer+Inner+Innermost.Test2	TestNameSpace.Outer.Inner.Innermost.Test2
TestNameSpace.Outer2+InnerClass.Test1	TestNameSpace.Outer2.InnerClass.Test1
TestNameSpace.SharedNameModule.Test1	TestNameSpace.SharedName

Last one seems more difficult to solve.

[baronfel]

Reopening to track the open case that @kojo12228 reported here.

[kojo12228]

I think this requires a change in FsAutoComplete, the logic (at least for NUnit) does not distinguish between a namespace and a module:

https://github.com/fsharp/FsAutoComplete/blob/f33887be9393087b05875f5c76334f557f0ed7be/src/FsAutoComplete.Core/TestAdapter.fs#L328-L347

The simplest change might just be adding SynModuleOrNamespaceKind or something like it onto the DTO. Then getFullName could be made aware of it.

https://github.com/ionide/ionide-vscode-fsharp/blob/1139128bb9a67fe8067b4930b044e3dccd821729/src/Components/TestExplorer.fs#L249-L252

That solves the first three cases. For the last case, I think TestAdapter needs to special case where SynTypeDefn and NestedModule are siblings in the AST with the same name, which shouldn't be too hard to do I would hope.

AST from Fantomas was incredibly helpful in trying to understand this:

https://fsprojects.github.io/fantomas-tools/#/ast?data=N4KABGBEAmCmBmBLAdrAzpAXFSAacUiaAYmolmPAIYA2as+EkAxgPZwWTJUC26ADlWawwAFXQAXAHK9YAZUHCAdHICuAIzQSATquYTV22AB1tx5KfM92qmiIDyqibG1gAvJeQQIAbQA84lrEiAAeBkYAfAC6nt5g1tC2IgCSyKiuHmZecd7+gRLRsTkQdhJikgCMYAAUAJTuRcVxAIJo9NoSSgAKVG01tZ6eCUlgjs7aAEwNWXF5ksFhhrCFM94SAJ78KWkuAMI0vWh10+ZNYHNaK6dNfDzqLmASABZESvlVx5nXZxCt7Z09Np1QZZDZbMByJ5UIzQGR8T6NMDQVj9EHmC4SBbhZYxLLDOwQqEwuEiL7ZXwBSRXclgUrlLQfepks5-FwAw51SAgAC+QA

[GarryPatchett]

This might, or might not, be useful but I have also come up against this issue in a slightly different way.

Sometimes, when I create and run some unit tests with Unquote on xUnit, the tests run successfully but the Testing Panel doesn’t always show the results.

In the examples below, “works” means that the tests run and complete as expected and the results are shown in the Testing Panel, and “doesn’t work” means that the spinners against each test keep spinning and the process keeps running until it’s manually cancelled even though the tests have completed according to the Ionide.trx file in the TestResults folder.

This works:

module Tests

open System
open Xunit
open Swensen.Unquote 

[<Fact>]
let ``My test`` () =
    test <@ true @>

But this doesn’t work:

module Tests

open System
open Xunit
open Swensen.Unquote 

module ``with my module`` = 

    [<Fact>]
    let ``My test`` () =
        test <@ true @>

After changing ‘module Tests’ to ‘namespace MyUnitTests’, this works:

namespace MyUnitTests

open System
open Xunit
open Swensen.Unquote 

module ``with my module`` = 

    [<Fact>]
    let ``My test`` () =
        test <@ true @>

But this doesn’t work:

namespace MyUnitTests

open System
open Xunit
open Swensen.Unquote 

module ``with my module`` = 

    module ``and this function`` = 

        [<Fact>]
        let ``My test`` () =
            test <@ true @>

I have been told that the same problem occurs when using Nunit also.

Ionide for F# version 7.5.1 VS Code version 1.76.0