The F# compiler takes a long time to compile each instance of xUnit's Assert.Equal

[mkb137]

The F# compiler takes a long time to compile xUnit assertions, e.g. Assert.Equal<int>(1,2), especially if untyped e.g. Assert.Equal(1,2)

To reproduce the problem:

1. Create an empty project.

2. Build the project.

3. Note the compile time, e.g.

  FSharpInferredTypeCompileTimes succeeded (0.9s)

4. Add a unit test file contaning 100 lines of random Assert.Equals, e.g.

    [<Fact>]
    member this.``Test Assert.Equal with no type parameter``() =
        Assert.Equal(42, generateRandomInt 1 100)
        Assert.Equal("hello", generateRandomString 10)
        Assert.Equal(3.14, generateRandomFloat 0.0 10.0)
        // 97 more lines

5. Build the project.
6. Note the compile time. On this machine, the compiler is adding 100ms for each Assert.Equal.

  FSharpInferredTypeCompileTimes succeeded (12.5s)

7. Add another 100 lines of Assert.Equals, so there are 200 total.
8. Build the project.
9. Note the compile time. The compiler continues to add about 100ms for each additional Assert.Equal.

  FSharpInferredTypeCompileTimes succeeded (22.9s)

10. Return to 100 lines of Assert.Equal and add strong typing to all the Assert.Equal calls, e.g.

    [<Fact>]
    member this.``Test Assert.Equal with strong typing``() =
        Assert.Equal<int>(42, generateRandomInt 1 100)
        Assert.Equal<string>("hello", generateRandomString 10)
        Assert.Equal<float>(3.14, generateRandomFloat 0.0 10.0)
        // 97 more lines

11. Build the project.
12. Note the compile time, and that it is much faster.

  FSharpInferredTypeCompileTimes succeeded (3.6s)

13. Add a wrapper function for Assert.Equal like so:

    let assertEqual (x: 'T, y: 'T) =
        Assert.Equal<'T>(x, y)

14. Modify all uses of Assert.Equal to call the wrapper function, e.g.

    [<Fact>]
    member this.``Test Assert.Equal with functional override``() =
        assertEqual(42, generateRandomInt 1 100)
        assertEqual("hello", generateRandomString 10)
        assertEqual(3.14, generateRandomFloat 0.0 10.0)

15. Build the project.
16. Note the compile time and that our single Assert.Equal has added 100ms and that each call to the wrapper has added no significant compile time.

  FSharpInferredTypeCompileTimes succeeded (1.0s)

I have only observed this problem with xUnit's Assertions. The problem exists with xUnit and xUnit.v3.

Expected behavior

Bulding an F# project containing xUnit Assert.Equals should be fast and should not take any longer than if called with a wrapper function.

Actual behavior

Compiling an F# project containing xUnit tests using Assert.Equals takes significantly longer than one with typed Assert.Equals or with a wrapper function, adding significant build time for each call to Assert.Equal.

Known workarounds

Add wrapper functions for xUnit assertions.

Related information

• Operating system Windows 11, Mac OS 15.5
• .NET Runtime kind (.NET Core, .NET Framework, Mono) .NET 9.0.200
• Editing Tools (e.g. Visual Studio Version, Visual Studio)

[mkb137]

Sample project. The various conditions can be controlled by defining constants, e.g.

dotnet build -p:DefineConstants="UNTYPED_ASSERT"
dotnet build -p:DefineConstants="TYPED_ASSERT"
dotnet build -p:DefineConstants="OVERRIDE_ASSERT"

Repro.zip

[T-Gro]

Thank you for the detailed analysis and perf issue reproduction.

Without generating actual traces, my hypothesis is method resolution for choosing one of the many Equal overloads available. If you extract that to a helper function, that overload selection only needs to happen once.

WHY overload resolution is so slow is a different story, and related to how F# approaches it w.r.t. to type inference (i.e. the case where generics are not specified and it is left to the compiler to resolve it). Typically, the constraint solver picks overloads by literally trying them and seeing how they play with inference of type variables, and then chooses the best candidates based on the results - so it is almost as slow as if it typechecked each of the available overloads.

This is definitely a curious case (100ms is massive amount of time) and I am sure there is potential by either caching common things or laying out data structures in a more convenient manner of the compiler.

A trace will be able to reveal the bottleneck here.

[T-Gro]

I like this issue being reported, because the nature of Assert.Equal overloads might reveal a deeper problem in the typechecking performance, one that likely already exists for other APIs incl. the .NET base class libraries.

[T-Gro]

I assume it will be something around this place https://github.com/dotnet/fsharp/blob/ecd8e2a5368fccef0c522b3803ff9aec1f8cf297/src/Compiler/Checking/ConstraintSolver.fs#L3486 , where candidates for the right method overload are collected and typechecked.