Benchmark several TFMs of a library on the same runtime

[ltrzesniewski]

Let's say I have a Library project that targets net5.0 and netstandard2.0, and a Benchmarks project that references it.

I could use the following configuration to benchmark the two TFMs:

[SimpleJob(RuntimeMoniker.Net48)]
[SimpleJob(RuntimeMoniker.NetCoreApp50)]

But the underlying runtime wouldn't be the same, and this can have a large impact on the results.

In my understanding, there's currently no straightforward way to benchmark the two TFMs on the same .NET 5 runtime.

---

Here's a solution I found:

internal class NetCoreAndStandardConfig : ManualConfig
{
    public NetCoreAndStandardConfig()
    {
        AddJob(
            Job.Default.WithId(".NET Core"),
            Job.Default.WithId(".NET Standard").WithArguments(new[] { new MsBuildArgument("/p:ForceNetStandard=true") })
        );
    }
}

But this requires adding the following conditional property to the Library project:

<TargetFramework Condition="'$(ForceNetStandard)' == 'true'">netstandard2.0</TargetFramework>

While it works, I don't like the fact that a change in the Library project is necessary. I'm wondering if BDN could provide an easier way to achieve this.

---

I initially thought that something like SetTargetFramework="TargetFramework=netstandard2.0" could be added to the ProjectReference item in the generated csproj file, but while this ends up referencing the netstandard2.0 version of the Benchmarks project, the resolved reference to Library will still target net5.0. This can be solved by adding another ProjectReference with SetTargetFramework to the Library project, but I'm not sure BDN should do this.

Also, while setting the TFM of the generated project to netstandard2.0 will end up referencing the correct assemblies, BDN will complain about the lack of an exe file.

So I'm not sure what's the best course of action here. While I think that having this feature would be very useful, unless I missed something it looks like implementing it won't be easy, and the workaround is not very straightforward.

[adamsitnik]

Hi @ltrzesniewski

.NET Standard is an abstraction. To run a console app, you need to specify the runtime, which provides the implementation of the abstraction. So there is no way to test .NET Standard performance, you can only test given runtime performance.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>net4.8;net5.0;netstandard2.0</TargetFrameworks>
  </PropertyGroup>
</Project>

using System;

namespace tfms
{
    class Program
    {
        static void Main(string[] args)
        {
#if NETSTANDARD2_0            
            Console.WriteLine("NETSTANDARD2_0");
#elif NETFRAMEWORK
            Console.WriteLine("NET48");
#elif NETCOREAPP
            Console.WriteLine("NET50");
#endif
        }
    }
}

PS C:\Projects\repros\tfms> dotnet run -c Release -f netstandard2.0
Unable to run your project.
Ensure you have a runnable project type and ensure 'dotnet run' supports this project.
A runnable project should target a runnable TFM (for instance, net5.0) and have OutputType 'Exe'.
The current OutputType is 'Exe'.
PS C:\Projects\repros\tfms> dotnet run -c Release -f net4.8
NET48
PS C:\Projects\repros\tfms> dotnet run -c Release -f net5.0
NET50
PS C:\Projects\repros\tfms>

[ltrzesniewski]

Hi @adamsitnik

Yes, I know that 🙂

I wanted to compare the performance of #if NETCOREAPP blocks of code against #else blocks in the Library project on the same runtime, as using a different runtime has a large performance impact.

I know that the .NET Standard version won't be used in .NET 5 apps, but I'd like to compare the two versions of my code only (I don't want to compare differences between my code and the runtime). Let's say I'd like to figure out if adding a net5.0 target to a netstandard2.0 library and using some .NET 5 features is worth it.

Maybe let's put some numbers to it:

Method	Job	Runtime	Arguments	Mean	Error	StdDev
CompiledAutoCallout	.NET Core	.NET Core 5.0	Default	509.3 ns	2.68 ns	2.37 ns
CompiledAutoCallout	.NET Framework	.NET 4.8	Default	684.8 ns	2.56 ns	2.14 ns
CompiledAutoCallout	.NET Standard	.NET Core 5.0	/p:ForceNetStandard=true	529.7 ns	4.47 ns	3.73 ns

".NET Standard" really means ".NET Standard version of Library running on .NET 5", but you can see that changing the runtime completely hides the difference between the two implementations in Library.

[DaveSkender]

I may have a similar use case that I'm trying to resolve regarding multi-targeting. Suppose:

<TargetFrameworks>net6.0;netstandard2.1;</TargetFrameworks>

And it produces two DLLs, one for .NET 6.0 and one for .NET Standard 2.1

And now I want to know how they comparatively perform with BenchmarkDotNet, with these attributes?

... I may be confusing the purpose of RuntimeMoniker, but I also don't see any other route to do this without creating multiple redundant Benchmark projects.

Background: the reason I want this is because I'm producing a multi-targeted public NuGet package and want to convince people to upgrade their chosen Frameworks so I can stop supporting older editions. My package, compiled for newer Frameworks, is measureably faster and I want to show that.

[ltrzesniewski]

Well, we have the same use case actually 😄

Here's my solution, for reference: the benchmark config, and the library project settings. It's the same thing as in the original post, but you can take a look at the real implmentation if you want to.

I suppose you could replace $(ForceNetStandard) with something like $(BenchmarkTargetFramework) to make it more generic if you need more than two target frameworks.

[timcassell]

It looks like we can support this by setting the SetTargetFramework property of the project reference.

<ProjectReference Include="$CSPROJPATH$">
    <SetTargetFramework>TargetFramework=netstandard2.0</SetTargetFramework>
</ProjectReference>

Of course, it would still require you to include the framework in your csproj.

[adamsitnik]

I think this feature is too niche to add it to BDN and make our config even more complex.

We can just document that the users need to use the right MsBuildArgument to get it working:

.WithArguments(new[] { new MsBuildArgument("/p:ForceNetStandard=true") })

[ltrzesniewski]

I think I've tried the SetTargetFramework property IIRC, and it wasn't sufficient to make it work, that's why I added a custom ForceNetStandard property which overrides the TargetFramework.

If you want to document the MSBuild argument, don't forget it requires the following line in the library project, since the property is custom:

<TargetFramework Condition="'$(ForceNetStandard)' == 'true'">netstandard2.0</TargetFramework>