[Bug] Not unique exporter for exporter type

[workgroupengineering]

trafficstars

Description

If the same exporter is defined in several levels (assembly and class) with different parameters, they are not merged.

Steps to Reproduce

1. Download attachment
2. Decompress
3. Run
4. At the end of benchmark you can see in Console output like this:

   // * Export *
   BenchmarkDotNet.Artifacts\results\TestMultipleExportIssue.SimpleBenchmark-report.csv
   BenchmarkDotNet.Artifacts\results\TestMultipleExportIssue.SimpleBenchmark-report-github.md
   BenchmarkDotNet.Artifacts\results\TestMultipleExportIssue.SimpleBenchmark-report.html
   CustomExporter - Benchmark
   CustomExporter - Assembly
   

Expected Behavior

Console output like this:

// * Export *
BenchmarkDotNet.Artifacts\results\TestMultipleExportIssue.SimpleBenchmark-report.csv
BenchmarkDotNet.Artifacts\results\TestMultipleExportIssue.SimpleBenchmark-report-github.md
BenchmarkDotNet.Artifacts\results\TestMultipleExportIssue.SimpleBenchmark-report.html
CustomExporter - Benchmark

Version: 0.12.1

Attachment

TestMultipleExportIssue.zip

[workgroupengineering]

Hi, I have a custom exporter that queues the results in a benchmark file, but it is stuck with this issue. I have proposed two possible solutions PR #1702 and #1796 , but the review of both is blocked. Please, can someone do the review so that I can change what you think is wrong?

Thanks and sorry if I'm stalking you.

[workgroupengineering]

I recheck this issue on ed53a3fefaddecca293e52add11321548fd26a07, but it is still present.

Simple test:

        [Fact]
        public void ExporterDependencyOverlapping()
        {
            var mutable = ManualConfig.CreateEmpty();
            mutable.AddExporter(new BenchmarkDotNet.Exporters.Csv.CsvMeasurementsExporter(BenchmarkDotNet.Exporters.Csv.CsvSeparator.Comma));
            mutable.AddExporter(RPlotExporter.Default);

            var final = ImmutableConfigBuilder.Create(mutable);
            Assert.Equal(2, final.GetExporters().Count());
        }

[AndreyAkinshin]

Hey @workgroupengineering, sorry for such a delay. #1796 looks pretty good to me. I have left a few minor comments there. Once you resolve them, I will merge the PR.

[YegorStepanov]

We need to revert it because it breaks exporters.

Starting with this PR, only one exporter per type can be.

This means that the following cases stopped working in 0.13.2:

// DefaultConfig has MarkdownExporterAttribute.Github already
// It prints a warning and Atlassian file is not created.
[MarkdownExporterAttribute.Atlassian] 
public class Klass
{
    [Benchmark] public void M() { }
}

//it doesn't work for ManualConfig:
[XmlExporterAttribute.Full] // it uses XmlExporter.Full under the hood
[XmlExporterAttribute.Brief] // it uses XmlExporter.Brief under the hood
public class Klass
{
    [Benchmark] public void M() { }
}

What I suggest:

• revert it
• compare and filter exporters by existing IExporter.Name (to cover this issue)
• add RPlotValidator that checks we do not override CsvMeasurementsExporter.Default

@AndreyAkinshin do you agree with me?