Stop gracefully when benchmark fails with an exception that can not be caught by a catch block

[adamsitnik]

trafficstars

The original issue was reported in https://github.com/dotnet/performance/issues/1701:

// Benchmark Process Environment Information:
// Runtime=.NET 6.0.0 (6.0.21.11801), X64 RyuJIT
// GC=Concurrent Workstation
// Job: Job-AQTFSE(PowerPlanMode=00000000-0000-0000-0000-000000000000, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1)

OverheadJitting  1: 1 op, 537000.00 ns, 537.0000 us/op
WorkloadJitting  1: 1 op, 6767100.00 ns, 6.7671 ms/op

OverheadJitting  2: 16 op, 731100.00 ns, 45.6938 us/op
WorkloadJitting  2: 16 op, 2527400.00 ns, 157.9625 us/op

WorkloadPilot    1: 16 op, 7800.00 ns, 487.5000 ns/op
// Benchmark Process 3336 has exited with code -1073741819
Unhandled exception. System.InvalidOperationException: Sequence contains no matching element
   at System.Linq.ThrowHelper.ThrowNoMatchException()
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Execute(ILogger logger, BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, IToolchain toolchain, BuildResult buildResult, IResolver resolver)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.RunCore(BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, ILogger logger, IResolver resolver, BuildResult buildResult)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo benchmarkRunInfo, Dictionary`2 buildResults, IResolver resolver, ILogger logger, List`1 artifactsToCleanup, String resultsFolderPath, String logFilePath, StartedClock& runChronometer)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[] benchmarkRunInfos)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)

The benchmark process exited with -1073741819 which translates to 0xc0000005 which is Access Violation.

BenchmarkDotNet should:

• detect failures for exceptions that can not be caught (most probably this logic has to be modified)
• stop gracefully in such cases

Whoever is willing to work on that should most probably start by creating a benchmark that throws an exception that can not be caught by a catch block (AccessViolation would be the best), then create a test for it similar to this one and fix BenchmarkRunnerClean.Execute

[danmoseley]

I wonder whether Watson grabbed a dump.

[kevinsalimi]

I wanna take this if still available.

[adamsitnik]

@kevinsalimi I've assigned you. Please let me know if you need some help.

[kevinsalimi]

Hello @adamsitnik. Thank you for the assignment. I spent ample time on the issue and learned a lot during the code exploration yet I have some questions.

The AccessViolation exception is this, am I right?

After creating a benchmark, I couldn't simulate any exceptions that cannot be caught by a catch block. I tried to threw variant exceptions and scenarios but I got nothing. In addition, I wrote code below in the benchmark:

unsafe
{
    IntPtr ptr = new IntPtr(100);
    int value = Marshal.ReadInt32(ptr);
}

I caught the AccessViolationException and the Benchmark stop gracefully like below.

// Benchmark Process Environment Information:
// Runtime=.NET Framework 4.8 (4.8.4300.0), X64 RyuJIT
// GC=Concurrent Workstation
// Job: DefaultJob

OverheadJitting  1: 1 op, 510500.00 ns, 510.5000 us/op

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Runtime.InteropServices.Marshal.ReadInt32(IntPtr ptr, Int32 ofs)
   at BenchmarkDotNet.Samples.IntroArguments.Benchmark() in C:\Users\Lenovo\Documents\OpenSourceProjects\BenchmarkDotNet\BenchmarkDotNet\samples\BenchmarkDotNet.Samples\IntroArguments.cs:line 20
   at BenchmarkDotNet.Autogenerated.Runnable_0.WorkloadActionNoUnroll(Int64 invokeCount)
   at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
   at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
   at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
   at BenchmarkDotNet.Autogenerated.Runnable_0.Run(IHost host, String benchmarkName)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args)
// AfterAll
// Benchmark Process 8632 has exited with code -1
Parse error in the following line:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
.
.
.
// ***** BenchmarkRunner: Finish  *****
.
.
.

Any ideas would be appreciated.

[kevinsalimi]

@adamsitnik

[timcassell]

I'm not sure how to generate an uncatchable AccessViolationException, but you could throw an uncatchable StackOverflowException:

void Kill()
{
    Kill();
}

[twallace27603]

You can also generate an uncatchable StackOverflowException by throwing it, although I love the elegant recursion. I found a great summary of "uncatchable errors on StackOverflow (the irony): https://stackoverflow.com/questions/7392783/list-of-exceptions-that-cant-be-caught-in-net. Basically there are conditions under which AccessViolationExceptions are sort of uncatchable, but it seems that the StackOverflowException is an easier candidate for testing the behavior.

[timcassell]

I believe if you manually throw a StackOverflowException it can be caught. It will only be un-catchable when the runtime throws it. At least that's what I observed when testing that in Unity a while ago. I haven't tested that behavior on the latest .Net runtime.

[twallace27603]

@timcassell you are 100% correct. I did something embarrassingly stupid for my first comment on this project. Your approach is the best test.

[danmoseley]

another example, presumably same fix:

[2021/09/08 09:11:10][INFO] System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
[2021/09/08 09:11:10][INFO]  ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException (00000005, 0xFFFDFFFF): Name does not resolve
[2021/09/08 09:11:10][INFO]    at System.Net.Dns.GetHostEntryOrAddressesCore(IPAddress address, Boolean justAddresses)
[2021/09/08 09:11:10][INFO]    at System.Net.Dns.GetHostEntry(String hostNameOrAddress)
[2021/09/08 09:11:10][INFO]    at System.Net.Tests.DnsTests.GetHostEntry() in /root/git/performance/src/benchmarks/micro/libraries/System.Net.Primitives/DnsTests.cs:line 14
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Autogenerated.Runnable_2149.WorkloadActionNoUnroll(Int64 invokeCount) in /root/git/performance/artifacts/bin/MicroBenchmarks/Release/net5.0/0f3d7004-1148-46cd-b551-e8a2449b85ae/0f3d7004-1148-46cd-b551-e8a2449b85ae.notcs:line 1582163
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Engines.Engine.RunIteration(IterationData data)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Engines.EngineFactory.Jit(Engine engine, Int32 jitIndex, Int32 invokeCount, Int32 unrollFactor)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Engines.EngineFactory.CreateReadyToRun(EngineParameters engineParameters)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Autogenerated.Runnable_2149.Run(IHost host, String benchmarkName) in /root/git/performance/artifacts/bin/MicroBenchmarks/Release/net5.0/0f3d7004-1148-46cd-b551-e8a2449b85ae/0f3d7004-1148-46cd-b551-e8a2449b85ae.notcs:line 1581840
[2021/09/08 09:11:10][INFO]    --- End of inner exception stack trace ---
[2021/09/08 09:11:10][INFO]    at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
[2021/09/08 09:11:10][INFO]    at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
[2021/09/08 09:11:10][INFO]    at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in /root/git/performance/artifacts/bin/MicroBenchmarks/Release/net5.0/0f3d7004-1148-46cd-b551-e8a2449b85ae/0f3d7004-1148-46cd-b551-e8a2449b85ae.notcs:line 3583
[2021/09/08 09:11:10][INFO] // AfterAll
[2021/09/08 09:11:10][INFO] // Benchmark Process 23065 has exited with code 255.
[2021/09/08 09:11:10][INFO] Unhandled exception. System.InvalidOperationException: Sequence contains no matching element
[2021/09/08 09:11:10][INFO]    at System.Linq.ThrowHelper.ThrowNoMatchException()
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Running.BenchmarkRunnerClean.Execute(ILogger logger, BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, IToolchain toolchain, BuildResult buildResult, IResolver resolver)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Running.BenchmarkRunnerClean.RunCore(BenchmarkCase benchmarkCase, BenchmarkId benchmarkId, ILogger logger, IResolver resolver, BuildResult buildResult)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo benchmarkRunInfo, Dictionary`2 buildResults, IResolver resolver, ILogger logger, List`1 artifactsToCleanup, String resultsFolderPath, String logFilePath, StartedClock& runChronometer)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[] benchmarkRunInfos)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config, Boolean askUserForInput)
[2021/09/08 09:11:10][INFO]    at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config)
[2021/09/08 09:11:10][INFO]    at MicroBenchmarks.Program.Main(String[] args) in /root/git/performance/src/benchmarks/micro/Program.cs:line 42
[2021/09/08 09:11:10][INFO] $ popd
[2021/09/08 09:11:10][ERROR] Process exited with status 134

cc @adamsitnik this was the issue I hit this morning

[YegorStepanov]

Summary, possibly a complete list of unhandled exceptions:

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

*(byte*)65536 = 13; // addresses up to 2^16 are zero (NullReferenceException)

Fatal error. Internal CLR error.

Marshal.StructureToPtr(13, (IntPtr)100, true);

Fatal error. Internal CLR error.

FormatterServices.GetUninitializedObject(typeof(Type).GetType()).ToString(); // deprecated in net7

Stack overflow.

R();
void R() => R();

[timcassell]

I'm unable to repro on Windows, even with 0.12.1 (the latest version at the time this issue was opened). Is it a Linux issue?