F# support

[KeterSCP]

Project description states that this is a

.NET testing framework

But .NET != C# Are there any plans to support F# projects?

[thomhurst]

Hey @KeterSCP . In all honesty I've never used F# but if it all compiles down to the same IL code, is there any reason this doesnt work with F#?

Thats a genuine question. If changes or tweaks need to be made please let me know.

A reproduction of any issues would be helpful, and if you want to help solve any challenges I'm open to that as like I said I'm not too versed in F#

[KeterSCP]

Hi @thomhurst! Thanks for the quick reply (and also nice job with this project!)

So here are the problems I encountered while trying minimal example in F# (it doesn't even assert anything, just returns 0):

module Tests

open TUnit.Core

[<Test>]
let ``My test`` () = 0

1. The test is not visible in the Tests tab in Rider (Testing platform support is enabled)
2. After running dotnet test from the CLI, build fails:

dotnet test
Restore complete (0.2s)
You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy
  TestProject2 succeeded (1.3s) → bin\Debug\net8.0\TestProject2.dll
  TestProject2 test failed with 1 error(s) (0.2s)
    ...\TestProject2\bin\Debug\net8.0\TestProject2.dll : error run failed: Tests failed: '...\TestProject2\bin\Debug\net8.0\TestResults\TestProject2_net8.0_x64.log' [net8.0|x64]

Test summary: total: 0, failed: 0, succeeded: 0, skipped: 0, duration: 0.2s
Build failed with 1 error(s) in 2.0s

Log file contains following output:

████████╗██╗   ██╗███╗   ██╗██╗████████╗
╚══██╔══╝██║   ██║████╗  ██║██║╚══██╔══╝
   ██║   ██║   ██║██╔██╗ ██║██║   ██║   
   ██║   ██║   ██║██║╚██╗██║██║   ██║   
   ██║   ╚██████╔╝██║ ╚████║██║   ██║   
   ╚═╝    ╚═════╝ ╚═╝  ╚═══╝╚═╝   ╚═╝   
   
   v0.1.783.0 | 64-bit | Microsoft Windows 10.0.22631 | win-x64 | .NET 8.0.8 | Microsoft Testing Platform v1.4.0
   

Test run summary: Zero tests ran - ...\TestProject2\bin\Debug\net8.0\TestProject2.dll (net8.0|x64)
  total: 0
  failed: 0
  succeeded: 0
  skipped: 0
  duration: 59ms

=== COMMAND LINE ===
C:\Program Files\dotnet\dotnet.exe exec ...\TestProject2\bin\Debug\net8.0\TestProject2.dll --internal-msbuild-node testingplatform.pipe.467200f8a0ac44d1a3488e611ee746e0 

[thomhurst]

Hmmm... Could you create a new repo with minimal repo please? Should help debug issues. I'll take a look but also might need your help if that's okay!

[KeterSCP]

Sure thing! Here is a separate repo, which demonstrates working example with XUnit and non-working example with TUnit:

https://github.com/KeterSCP/tunit-fsharp-repo

[bevanweiss]

My understanding is that TUnit relies on C# code generation. That won't work for a F# project, because the F# project will not compile the C# generated code... it will just be random files in the F# project.

What 'should' work however, is having a dedicated C# project just for the tests that references the F# project with the logic to be tested.

I don't think that F# has support for source generation in the same way that C# has, so I don't think it's an option at this stage. It does look like the F# source generator issue has been raised https://github.com/fsharp/fslang-suggestions/issues/864, but nothing advanced on it.

[thomhurst]

My understanding is that TUnit relies on C# code generation. That won't work for a F# project, because the F# project will not compile the C# generated code... it will just be random files in the F# project.

What 'should' work however, is having a dedicated C# project just for the tests that references the F# project with the logic to be tested.

I don't think that F# has support for source generation in the same way that C# has, so I don't think it's an option at this stage. It does look like the F# source generator issue has been raised fsharp/fslang-suggestions#864, but nothing advanced on it.

Ah interesting. Does this essentially mean it's not a TUnit bug? I just (maybe naively) assumed that the nugrt packages I published would work for any .net language because it all compiles down to the same thing. If not I just need to see if I can tweak the source generator depending on theangjage

[bevanweiss]

Ah interesting. Does this essentially mean it's not a TUnit bug? I just (maybe naively) assumed that the nugrt packages I published would work for any .net language because it all compiles down to the same thing. If not I just need to see if I can tweak the source generator depending on theangjage

You potentially know more in this space than me ;) My understanding was, the 'Core' stuff that you (@thomhurst) write in fixed C# is indeed compiled down on your side to .NET IL (unless you opt for AoT etc etc) during the nuget publishing, and is purely runnable under the .NET CLR, so is agnostic to C#/F#/VB.NET usage. However... for the source generated aspects, the parts which are spat out C# language code files, these need to be compiled at the 'end-users' side. If they have an F# / VB.NET project environment, they might not even have the Roslyn C# compiler AT ALL... I also think that the C# source generators are only enabled in C# projects, since they kind of inject themselves into the compilation process. Although they don't have to restrict themselves to .cs files (they can access other files also... but their output is stuffed into the C# compilation chain).

So I'd say it's not a TUnit 'bug' at all. It's just one of the limitations of using the (C#) Source Generator concept. There's potentially some ways to get around it, maybe having an MSBuild Task which separately launches the csc compiler against some boilerplate core code that then allows for the source generator to kick in and parse the F# files, to generate the final C# source which the csc invocation then builds... but it would also need to somehow link this csc result back into the F# project... so I reckon the 'bang for buck' on maintaining such a monstrosity wouldn't be there...

In an F# project, there is no C# compiler invocation, so I don't believe your source generator is even called.

https://learn.microsoft.com/en-us/dotnet/csharp/roslyn-sdk/source-generators-overview

[thomhurst]

Thanks for the detailed description that's really helpful!

So would the solution basically be to determine whether we're in an F# solution and if so generate F# code instead of C#?

I haven't actually done any experimentation on source generators with F# but I assume it's possible to check the language. The TUnit source generation isn't too complicated, so if F# needs it's own F# generated code, then let's make that happen :)

[bevanweiss]

Thanks for the detailed description that's really helpful!

So would the solution basically be to determine whether we're in an F# solution and if so generate F# code instead of C#?

I haven't actually done any experimentation on source generators with F# but I assume it's possible to check the language. The TUnit source generation isn't too complicated, so if F# needs it's own F# generated code, then let's add that as an issue and make that happen. Would you like to create an issue for that?

Unfortunately the F# compiler doesn't support this. In the C# Roslyn compiler they have broken the compilation process into two parts, an initial phase where it tokenizes and parses everything, and then the subsequent phase where it lowers and links everything to the IL. This is what allows the Source Generator concept to work. The first part runs, then it hands this information (and other 'project context') across to the Source Generator, which can make additional source objects etc, which are then also tokenized and parsed. Then it proceeds to do the 'real' backend compilation bit.

I'm not well versed in F#, but I think it's also missing core language support for things like 'partial classes' that were introduced to the C# language to help support source generation.

I think the things lacking from F# in the source generator space make it problematic to support at all for something like TUnit.

Two possible options might be:

1. Use runtime reflection as a fallback, like how the other Test Frameworks do... runtime reflection obviously doesn't care about which source language was used. It's all about the pure IL at that stage. But it means all the goodness that comes with Source Generation disappears.. and it would require a whole runtime reflection engine.
2. Go the route of IL weaving instead. So rather than integrating into the compile process like with C#, have a pre/post compilation process (post is probably 'easier') that directly manipulates / emits IL to do what the C# Source Generation currently does. Which sounds a bit painful. It would use the MSBuild stuff I mentioned previously, wouldn't get inbuilt compiler support for the parsing of the F#, would need to generate IL for the parts that are currently C# source gen'd (but couldn't rely on a C# compiler), and would need to work around various restrictions from the F# compilation (i.e. you couldn't just add extra parameters to test classes, since the F# IL might have done alignment / optimisations which make them difficult to change).

Of the two options, the first one is probably 'nicer' (easier, with less maintenance hassle).

[bevanweiss]

FYI: Because both VB.NET and C# are based within the Roslyn compiler, it does appear that Source Generation might be possible for VB... https://github.com/dotnet/vblang/issues/586

But F# isn't a Roslyn thing... so has some quite different interfaces.

I guess technically it would be possible to use something like these: https://fsharp.github.io/fsharp-compiler-docs/fcs/ https://fsprojects.github.io/fantomas/docs/end-users/GeneratingCode.html

To do a pre-build trigger, parse the F# and do like a source generator for it... and then the Microsoft F# compiler would kick in and compile the 'proper' F# project (which would include all the source generated files). I'm pretty sure it's a different source generator / parser interface than the Roslyn interface you've been using for C# though.

[thomhurst]

Oh damn this seems pretty complicated then. I guess for now I'll have to brand TUnit as C# only 😢

[thomhurst]

I think based on this, the answer to the original question is that there is no plans to support F#, mainly due to the lack of compiler APIs.

Very sorry about this, but it's out of my control really 😢

[thomhurst]

So the plan is now to support F#. The caveat is it'll use reflection as opposed to source generators.

[thomhurst]

This is now in v0.20.0

[thomhurst]

Given i don't / haven't really used F# can any of you try it out and feed back please? I added a smoke test project of just a simple happy test, but anything more involved I'll need some feedback please!

[licon4812]

Thanks for the awesome job @thomhurst . When making the F# template I stumble upon a potential issue.

In C# Generic attributes at compile time are automatically filled in if parameters are missing. For example in the code bellow you don't have to provide all arguments

[Test]
[ClassDataSource<DataClass>]
[ClassDataSource<DataClass>(Shared = SharedType.PerClass)]
[ClassDataSource<DataClass>(Shared = SharedType.PerAssembly)]
[ClassDataSource<DataClass>(Shared = SharedType.PerTestSession)]
public void ClassDataSource(DataClass dataClass)
{
    Console.WriteLine("This test can accept a class, which can also be pre-initialised before being injected in");

    Console.WriteLine("These can also be shared among other tests, or new'd up each time, by using the `Shared` property on the attribute");
}

In F# however I believe the compiler doesn't allow this, so the same declaration has the following error

[<Test>]
[<ClassDataSource(typeof<TestProject.DataClass>)>]
[<ClassDataSource(typeof<DataClass>, Shared = SharedType.PerClass)>]
[<ClassDataSource(typeof<DataClass>, Shared = SharedType.PerAssembly)>]
[<ClassDataSource(typeof<DataClass>, Shared = SharedType.PerTestSession)>]
member _.ClassDataSource(dataClass: DataClass) =
    Console.WriteLine("This test can accept a class, which can also be pre-initialised before being injected in")
    Console.WriteLine("These can also be shared among other tests, or new'd up each time, by using the `Shared` property on the attribute")

[xperiandri]

So the source generators are used for tests discovery and parameterized tests to avoid reflection, don't they? If so, can we specify the generators to look into the other referenced assembly instead of the current project and get tests from it?

[brmassa]

@KeterSCP I got this very error message in my C# project. The problem? My test suite had a

class Program {
void Main(){} 
}

, but TUnit generates its own entrypoint. dotnet run worked but dotnet test did not, showing the --internal-msbuild-node argument problem.

@thomhurst evaluate this scenario and, in case it's valid, considering including in the FAQ and/or deal with this possibility.

[thomhurst]

@KeterSCP I got this very error message in my C# project. The problem? My test suite had a

class Program {
void Main(){} 
}

, but TUnit generates its own entrypoint. dotnet run worked but dotnet test did not, showing the --internal-msbuild-node argument problem.

@thomhurst evaluate this scenario and, in case it's valid, considering including in the FAQ and/or deal with this possibility.

It's a small section but it is already noted here under installing TUnit: https://tunit.dev/docs/getting-started/installation/#manually

[thomhurst]

So the source generators are used for tests discovery and parameterized tests to avoid reflection, don't they? If so, can we specify the generators to look into the other referenced assembly instead of the current project and get tests from it?

It should look at all referenced projects by default.

This is for C# though. Not sure if that's what you mean since this ticket is about F#

[brmassa]

It's a small section but it is already noted here under installing TUnit: https://tunit.dev/docs/getting-started/installation/#manually

I was referring about the error, not the setup. In case one does create an entry point, a FAQ might point the mistake.