BenchmarkDotNet icon indicating copy to clipboard operation
BenchmarkDotNet copied to clipboard
verified publisher icon dotnet

When benchmarking against multiple TFMs including a Native AOT one, BenchmarkDotNet gets confused about framework identifiers

Open akamsteeg opened this issue 1 year ago • 4 comments

(I'm using BenchmarkDotNet v0.14. The issue occurs on Windows, I haven't tested it on Linux or Mac OS.)

I have a Program.cs inside a benchmark project that looks like this:

public static class Program
{
  public static void Main(string[] args)
  {
    var config = GetConfig();
    BenchmarkSwitcher
      .FromAssembly(Assembly.GetExecutingAssembly())
      .Run(args, config);
  }

  private static ManualConfig GetConfig()
  {
    var job = Job.Default;

    var config = ManualConfig.Create(DefaultConfig.Instance)
      .AddJob(
        job.WithToolchain(CsProjCoreToolchain.NetCoreApp80).AsBaseline(),
        Job.Default.WithRuntime(NativeAotRuntime.Net80),
        job.WithToolchain(CsProjCoreToolchain.NetCoreApp60)
        )
      .AddDiagnoser(MemoryDiagnoser.Default);

    if (Environment.OSVersion.Platform == PlatformID.Win32NT)
    {
      config.AddJob(job.WithToolchain(CsProjClassicNetToolchain.Net481));
    }

    config.SummaryStyle = SummaryStyle.Default
      .WithRatioStyle(RatioStyle.Percentage);

    config.AddValidator(JitOptimizationsValidator.FailOnError); // Fail when any of the referenced assemblies are not optimized

    config.WithOrderer(new DefaultOrderer(SummaryOrderPolicy.FastestToSlowest));

    return config;
  }
}

So I'm running benchmarks against .NET 8 as my baseline, .NET 8 with Native AOT, .NET 6 and .NET 4.8.1 when on Windows. When running this using dotnet build -c Release && dotnet run -c Release -f net8.0 the final report in the console shows this:

// * Summary *

BenchmarkDotNet v0.14.0, Windows 11 (10.0.22631.3958/23H2/2023Update/SunValley3)
AMD Ryzen 7 7800X3D, 1 CPU, 16 logical and 8 physical cores
.NET SDK 9.0.100-preview.6.24328.19
  [Host]     : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-IRRDUC : .NET 8.0.7 (8.0.724.31311), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-BCZWNO : .NET 8.0.5, X64 NativeAOT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-EMODDW : .NET 6.0.32 (6.0.3224.31407), X64 RyuJIT AVX2
  Job-LCVGYL : .NET Framework 4.8.1 (4.8.9256.0), X64 RyuJIT VectorSize=256


| Method                        | Runtime       | password             | Mean       | Error    | StdDev   | Ratio    | RatioSD | Gen0   | Gen1   | Allocated | Alloc Ratio |
|------------------------------ |-------------- |--------------------- |-----------:|---------:|---------:|---------:|--------:|-------:|-------:|----------:|------------:|
| GetKAnonimityPartsForPassword | .NET 8.0      | -&HxcB_d             |   277.4 ns |  4.09 ns |  3.83 ns | baseline |         | 0.0043 |      - |     232 B |             |
| GetKAnonimityPartsForPassword | NativeAOT 8.0 | -&HxcB_d             |   284.4 ns |  5.69 ns |  7.20 ns |      +3% |    2.8% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&HxcB_d             |   288.1 ns |  4.49 ns |  4.20 ns |      +4% |    1.9% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&HxcB_d             | 3,249.2 ns | 54.70 ns | 51.16 ns |  +1,072% |    2.0% | 0.7477 | 0.0038 |    4710 B |     +1,930% |
|                               |               |                      |            |          |          |          |         |        |        |           |             |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&Hxc(...)QbuAz [64] |   361.9 ns |  7.22 ns |  6.75 ns | baseline |         | 0.0043 |      - |     232 B |             |
| GetKAnonimityPartsForPassword | NativeAOT 8.0 | -&Hxc(...)QbuAz [64] |   362.2 ns |  4.28 ns |  4.00 ns |      +0% |    2.1% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&Hxc(...)QbuAz [64] |   369.0 ns |  3.93 ns |  3.28 ns |      +2% |    2.0% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | -&Hxc(...)QbuAz [64] | 3,274.8 ns | 62.43 ns | 61.32 ns |    +805% |    2.6% | 0.7553 | 0.0038 |    4766 B |     +1,954% |
|                               |               |                      |            |          |          |          |         |        |        |           |             |
| GetKAnonimityPartsForPassword | .NET 8.0      | これはパスワードです           |   305.4 ns |  3.88 ns |  3.44 ns | baseline |         | 0.0043 |      - |     232 B |             |
| GetKAnonimityPartsForPassword | NativeAOT 8.0 | これはパスワードです           |   329.3 ns |  6.50 ns |  7.49 ns |      +8% |    2.5% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | これはパスワードです           |   330.8 ns |  3.53 ns |  3.30 ns |      +8% |    1.4% | 0.0043 |      - |     232 B |         +0% |
| GetKAnonimityPartsForPassword | .NET 8.0      | これはパスワードです           | 3,248.5 ns | 42.48 ns | 39.73 ns |    +964% |    1.6% | 0.7515 | 0.0038 |    4734 B |     +1,941% |

In the first part of the summary, where the jobs are listed, it correctly identifies the runtimes used. However, in the overview of the metrics per benchmark/runtime combination, it gets confused about it and only lists .NET 8.0 or NativeAOT 8.0 and not .NET 6 and .NET 4.8.1. The metrics are okay though, so it looks like just a display issue for the runtime names.

⚠️ Removing the AOT job (Job.Default.WithRuntime(NativeAotRuntime.Net80)) solves this issue so it seems to be related to having Native AOT benchmarks mixed with 'normal' ones.

For reproduction the code I ran is on Github: https://github.com/akamsteeg/AtleX.HaveIBeenPwned/commit/c6c6cd483a58044db104b94ca280db8612b20f03. Running it with dotnet build -c Release && dotnet run -c Release -f net8.0 and picking any benchmark should reproduce this. It's consistent for me on two Windows 11 machines.

akamsteeg avatar Aug 07 '24 15:08 akamsteeg

in the overview of the metrics per benchmark/runtime combination, it gets confused about it and only lists .NET 8.0 or NativeAOT 8.0 and not .NET 6 and .NET 4.8.1.

It seems occurred when using both WithToolchain and WithRuntime methods are used for jobs configuration.

WithToolchain don't set Runtime characterics to the Job. If Runtime value is not set. It display [Host] runtime value instead. And when Runtime column is shown. Toolchain column is automatically hidden.

As a work around. It can be resolved by explicitly setting runtime by using .WithRuntime(CoreRuntime.CoreNN).

Alternatively it can explicitly specify JobId by using WithId(id) method. and hide Runtime column.

filzrev avatar Jun 11 '25 07:06 filzrev

@filzrev , thanks for your reply. I've tested your suggested solution and that seems to work.

The configuration was changed too:

var config = DefaultConfig.Instance
      .AddDiagnoser(MemoryDiagnoser.Default)
      .AddColumn(StatisticColumn.Median, StatisticColumn.Min, StatisticColumn.Max)
      .AddJob(
 job.WithRuntime(CoreRuntime.Core80).AsBaseline(),
        job.WithRuntime(CoreRuntime.Core60),
        job.WithRuntime(NativeAotRuntime.Net80)
      );

    if (Environment.OSVersion.Platform == PlatformID.Win32NT)
    {
      config.AddJob(job.WithRuntime(ClrRuntime.Net481));
    }

That resulted in the correct output:


BenchmarkDotNet v0.14.0, Windows 11 (10.0.26100.4351)
AMD Ryzen 7 7800X3D, 1 CPU, 16 logical and 8 physical cores
.NET SDK 10.0.100-preview.3.25201.16
  [Host]     : .NET 8.0.17 (8.0.1725.26602), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-DAISQD : .NET 8.0.14, X64 NativeAOT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-SZYMSL : .NET 8.0.17 (8.0.1725.26602), X64 RyuJIT AVX-512F+CD+BW+DQ+VL+VBMI
  Job-WNOWAU : .NET 6.0.36 (6.0.3624.51421), X64 RyuJIT AVX2
  Job-AJMMOP : .NET Framework 4.8.1 (4.8.9310.0), X64 RyuJIT VectorSize=256

WarmupCount=16
Method Runtime password Mean Error StdDev Median Min Max Ratio RatioSD Gen0 Gen1 Allocated Alloc Ratio
GetKAnonimityPartsForPassword NativeAOT 8.0 -&HxcB_d 262.2 ns 5.15 ns 7.70 ns 259.7 ns 254.2 ns 283.2 ns -3% 3.8% 0.0043 - 232 B +0%
GetKAnonimityPartsForPassword .NET 8.0 -&HxcB_d 271.7 ns 5.26 ns 6.84 ns 270.4 ns 263.3 ns 284.1 ns baseline 0.0043 - 232 B
GetKAnonimityPartsForPassword .NET 6.0 -&HxcB_d 294.3 ns 5.75 ns 5.38 ns 293.2 ns 285.8 ns 304.0 ns +8% 3.0% 0.0043 - 232 B +0%
GetKAnonimityPartsForPassword .NET Framework 4.8.1 -&HxcB_d 2,970.3 ns 58.72 ns 80.38 ns 2,928.6 ns 2,887.5 ns 3,100.3 ns +994% 3.6% 0.7477 0.0038 4710 B +1,930%
GetKAnonimityPartsForPassword NativeAOT 8.0 -&Hxc(...)QbuAz [64] 340.1 ns 5.50 ns 4.88 ns 337.9 ns 335.3 ns 351.9 ns -0% 2.2% 0.0043 - 232 B +0%
GetKAnonimityPartsForPassword .NET 8.0 -&Hxc(...)QbuAz [64] 341.8 ns 6.63 ns 5.88 ns 342.5 ns 332.6 ns 352.4 ns baseline 0.0043 - 232 B
GetKAnonimityPartsForPassword .NET 6.0 -&Hxc(...)QbuAz [64] 363.3 ns 7.25 ns 8.63 ns 364.6 ns 353.4 ns 381.2 ns +6% 2.9% 0.0043 - 232 B +0%
GetKAnonimityPartsForPassword .NET Framework 4.8.1 -&Hxc(...)QbuAz [64] 3,095.0 ns 59.48 ns 70.80 ns 3,075.1 ns 3,022.9 ns 3,232.2 ns +806% 2.8% 0.7553 0.0038 4766 B +1,954%
GetKAnonimityPartsForPassword NativeAOT 8.0 これはパスワードです 289.4 ns 5.77 ns 5.40 ns 286.5 ns 284.4 ns 300.8 ns -2% 2.5% 0.0043 - 232 B +0%
GetKAnonimityPartsForPassword .NET 8.0 これはパスワードです 293.9 ns 5.57 ns 5.21 ns 291.2 ns 287.6 ns 302.4 ns baseline 0.0043 - 232 B
GetKAnonimityPartsForPassword .NET 6.0 これはパスワードです 302.3 ns 5.86 ns 5.75 ns 302.3 ns 295.1 ns 313.5 ns +3% 2.5% 0.0043 - 232 B +0%
GetKAnonimityPartsForPassword .NET Framework 4.8.1 これはパスワードです 2,987.2 ns 47.60 ns 44.53 ns 2,966.0 ns 2,946.5 ns 3,070.6 ns +917% 2.2% 0.7515 0.0038 4734 B +1,941%

akamsteeg avatar Jun 16 '25 15:06 akamsteeg

BenchmarkDotNet v0.15.2 add runtime validator to raise warning when Runtime column shows incorrect information.

As commented at https://github.com/dotnet/BenchmarkDotNet/pull/2771#issuecomment-2968409612. This validator is temporary workaround and it need to be fixed in future.

I think the UX needs to be improved around this, but this is ok as a stop-gap for now.

The root cause of problem is method returns Host runtime if runtime characteristic is not set. So it need to add additional logics to lookup Toolchain characteristic value. As far as I've confirmed following methods are relating to this issue.

  1. EnvironmentMode::GetRuntime()
  2. BenchmarkCase::GetRuntime()
  3. BenchmarkPartitioner::GetRuntime()

filzrev avatar Jun 17 '25 02:06 filzrev

I think the proper fix is to make Runtime a property of IToolchain. They are fundamentally coupled concepts, but the job system pretends like they are not (technically a user could use a toolchain with a non-matching runtime, a case that we have not tested). It's a breaking change, but will clean things up quite a bit. Internal code has a lot of places that check both the runtime and the toolchain because of this, and it's brittle (even some places that should do it but don't).

timcassell avatar Oct 16 '25 04:10 timcassell