Custom sinks

[JohanLarsson]

It is nice to put the results from benchmark runs in version control so that perf can be tracked over time. Currently I hack it like this:

    using System.Collections.Generic;
    using System.IO;
    using BenchmarkDotNet.Reports;
    using BenchmarkDotNet.Running;

    public class Program
    {
        private static readonly string DesinationDirectory = System.IO.Path.Combine(new DirectoryInfo(Directory.GetCurrentDirectory()).Parent.Parent.FullName, "Results");

        public static void Main()
        {
            foreach (var summary in RunAll())
            {
                CopyResult(summary.Title);
            }
        }

        private static IEnumerable<Summary> RunAll()
        {
            ClearAllResults();
            var switcher = new BenchmarkSwitcher(typeof(Program).Assembly);
            var summaries = switcher.Run(new[] { "*" });
            return summaries;
        }

        private static IEnumerable<Summary> RunSingle<T>()
        {
            var summaries = new[] { BenchmarkRunner.Run<T>() };
            return summaries;
        }

        private static void CopyResult(string name)
        {
#if DEBUG
#else
            var sourceFileName = System.IO.Path.Combine(Directory.GetCurrentDirectory(), "BenchmarkDotNet.Artifacts", "results", name + "-report-github.md");
            System.IO.Directory.CreateDirectory(DesinationDirectory);
            var destinationFileName = System.IO.Path.Combine(DesinationDirectory, name + ".md");
            File.Copy(sourceFileName, destinationFileName, true);
#endif
        }

        private static void ClearAllResults()
        {
            if (Directory.Exists(DesinationDirectory))
            {
                foreach (var resultFile in Directory.EnumerateFiles(DesinationDirectory, "*.md"))
                {
                    File.Delete(resultFile);
                }
            }
        }
    }

[veikkoeeva]

It would nice if there were a general purpose sink for this. I would like to incorporate the results to a DB too for further analysis (those analytic functions in relational engines and all that).

Also, looks like having common issues with https://github.com/PerfDotNet/BenchmarkDotNet/issues/155.

[AndreyAkinshin]

We could add an additional string property in the config (like ArtifactFolderName, can be absolute or relative) with BenchmarkDotNet.Artifacts as a default value. Probably, we also have to add some option to disable creating of the artifact folder. In most cases, I don't need the artifacts, I just look at the summary table.

It would nice if there were a general purpose sink for this.

I'm not sure about it. If you want to send result in a custom format to a custom sink, you always could take the summary and implement export manually. Do we really need an additional layer of abstraction for this?

@adamsitnik, what do you think?

[adamsitnik]

We could add an additional string property in the config (like ArtifactFolderName, can be absolute or relative) with BenchmarkDotNet.Artifacts as a default value.

I like this idea and I believe it should not break anything an be easy to implement.

As for the sinks: it's a great feature, but IMO we (BenchmarkDotNet) should only provide an api for that, which could be implemented by the community. Example: serilog. Currently it could be achieved by implementing custom IExporter?

[AndreyAkinshin]

@adamsitnik, do you want to implement it? =)

[adamsitnik]

@AndreyAkinshin sure. When are you releasing 0.10.0?

[AndreyAkinshin]

Probably it would be early October. I did a lot of changes in the measurement engine (some additional changes are coming) and I need some time to carefully check all my benchmarks. =)