Automate benchmark execution

[antonfirsov]

The parameter space became large and complex, running benchmarks is real chore now, and it would be even harder on Citrine, where we need to test with bigger ranges of t. To improve our productivity, I'm planning to implement some automation.

Goals:

• Automate BenchmarkDriver(2) execution for a given set of parameters.
• Produce output that is easy to deal with. Suggestion: .csv
  • Conversion to .md should be trivial (in case of excel: just copy paste the table to GitHub)
  • Making charts would be easy

Possible solutions:

• A PowerShell script to control BenchmarkDriver(2) for us. Disadvantage: it's PowerShell ;)
• A small C# app to control BenchmarkDriver(2) for us. Disadvantage: not self-contained enough, needs to be compiled

Example usage:

./RunBenchmarks.ps1 --server http://asp-perf-lin:5001 --client "http://asp-perf-load:5002" -e=epoll,uring -o=iothread,inline -t=4..8

This will run 2 * 2 * 5 = 20 benchmarks in total, and save there results in a CSV having the scheme:

e	o	t	RPS	CPU%	latency
.	.	.	..	....	......
.	.	.	..	....	......
.	.	.	..	....	......

@adamsitnik any concerns/expectations from your side? Preference for C# vs PowerShell?

[adamsitnik]

I am a n00b and I always go 100% C# 🗡

I've written a very small app that parses commands from a file and runs the benchmarks. Everything is hardcoded, but it gives me what I want, you might find it usefull:

using System;
using System.Diagnostics;
using System.IO;
using System.Linq;

namespace BenchmarkScheduler
{
    class Program
    {
        static void Main(string[] args)
        {
            const string workingDirectory = @"C:\Projects\aspnet_benchmarks\src\BenchmarksDriver";
            const string commandPrefix = @"run -c Release -- --display-output --server http://asp-perf-lin:5001 --client http://asp-perf-load:5002 --repository https://github.com/tmds/Tmds.LinuxAsync.git --project-file test/web/web.csproj --collect-trace";

            foreach (var commandSuffix in File.ReadAllLines(@"C:\Users\adsitnik\source\repos\BenchmarkScheduler\BenchmarkScheduler\commands.txt").Where(x => !string.IsNullOrEmpty(x)))
            {
                Console.WriteLine("```cmd");
                Console.WriteLine(commandSuffix);
                Console.WriteLine("```");

                Console.WriteLine("```log");

                using (var process = Process.Start(new ProcessStartInfo()
                {
                    FileName = "dotnet",
                    Arguments = $"{commandPrefix} {commandSuffix}",
                    WorkingDirectory = workingDirectory,
                    RedirectStandardOutput = true,
                    UseShellExecute = false,
                    CreateNoWindow = true
                }))
                {
                    string line = null;
                    while ((line = process.StandardOutput.ReadLine()) != null)
                    {
                        if (!string.IsNullOrEmpty(line) && !line.StartsWith('[') && !line.StartsWith("failed"))
                        {
                            Console.WriteLine(line);
                        }
                    }

                    if (process.HasExited)
                    {
                        process.WaitForExit();
                    }
                }

                Console.WriteLine("```");
            }
        }
    }
}