Unhandled Exception Thrown When Running an invalid Type-Based Benchmark with Arguments

[AvishaiDotan]

When attempting to run a type-based benchmark with arguments passed as parameters, if the benchmark is invalid, the system throws an unhandled exception.

Example:

public class InvalidBenchmark
{
}

BenchmarkRunner.Run<InvalidBenchmark>(null, new[] { " " });
// Or
BenchmarkRunner.Run(typeof(InvalidBenchmark), null, new[] { " " });

Unhandled exception. System.InvalidOperationException: Sequence contains no elements
   at System.Linq.ThrowHelper.ThrowNoElementsException()
   at System.Linq.Enumerable.Single[TSource](IEnumerable`1 source)
   at BenchmarkDotNet.Running.BenchmarkRunner.RunWithDirtyAssemblyResolveHelper(Type type, IConfig config, String[] args) in ...BenchmarkRunnerDirty.cs:line 89
   at BenchmarkDotNet.Running.BenchmarkRunner.<>c__DisplayClass0_0`1.<Run>b__0() in ...BenchmarkRunnerDirty.cs:line 24
   at BenchmarkDotNet.Running.BenchmarkRunner.RunWithExceptionHandling(Func`1 run) in ...BenchmarkRunnerDirty.cs:line 132
   at BenchmarkDotNet.Running.BenchmarkRunner.Run[T](IConfig config, String[] args) in ...BenchmarkRunnerDirty.cs:line 24
   at BenchmarkDotNet.Samples.Program.Main(String[] args) in ...Program.cs:line 10

With more specific details: When executing a type-based benchmark with arguments, the system invokes the BenchmarkSwitcher with .Single() method afterwards.

//BenchmarkDotNet\Running\BenchmarkRunnerDirty.cs
[MethodImpl(MethodImplOptions.NoInlining)]
private static Summary RunWithDirtyAssemblyResolveHelper(Type type, IConfig? config, string[]? args)
    => (args == null
        ? BenchmarkRunnerClean.Run(new[] { BenchmarkConverter.TypeToBenchmarks(type, config) })
        : new BenchmarkSwitcher(new[] { type }).RunWithDirtyAssemblyResolveHelper(args, config, false))
        .Single();

The RunWithDirtyAssemblyResolveHelper performs its checks, and when the benchmark fails, it returns an empty array as the result. This type of response is implemented multiple times throughout the code, for example:

\\BenchmarkDotNet\Running\BenchmarkSwitcher.cs
if (!allTypesValid) // there were some invalid and TypeFilter printed errors
    return Array.Empty<Summary>();

When the benchmark fails, the system attempts to execute Array.Empty<Summary>().Single(), which ultimately results in an unhandled error.