BenchmarkDotNet icon indicating copy to clipboard operation
BenchmarkDotNet copied to clipboard

Can't override `DefaultConfig.Instance` settings when using custom configs with `IConfigSource` attribute.

Open filzrev opened this issue 4 months ago • 0 comments

I've created benchmark config by using ManualConfig derived class. And using this config with [Config(typeof(CustomConfig))] attribute.

When running benchmarks. It seems config is always merged with DefaultConfig.Instance by BenchmarkConverter::GetFullTypeConfig And it causes unintended settings to be used when running benchmarks.

Minimum code to reproduce problems

internal class Program
{
    static void Main(string[] args)
    {
        BenchmarkRunner.Run<SampleBenchmarks>(config: null, args: args);
    }
}

public class CustomBenchmarkConfig : ManualConfig
{
    public CustomBenchmarkConfig()
    {
        AddExporter(MarkdownExporter.GitHub);
    }
}

[Config(typeof(CustomBenchmarkConfig))]
public class SampleBenchmarks
{
    [Benchmark]
    public void Benchmark01()
    {
    }
}

Warnings Following warnings occurred when running above code.

// * Warnings *
Configuration
  Summary -> The exporter MarkdownExporter-github is already present in configuration. There may be unexpected results.

It seems default exporters are configured that are defined by DefaultConfig .

Temporary workaround This problem can be avoided by explicitly specifying the config or pass config that is created by ManualConfig.CreateEmpty().

filzrev avatar Apr 05 '24 23:04 filzrev