getsentry
Add support for catching native crashes in WinUI3
Package
Sentry
.NET Flavor
.NET
.NET Version
8.0.2
OS
Windows
SDK Version
4.2.1
Self-Hosted Sentry Version
No response
Steps to Reproduce
- Create WinUI3 / WinAppSDK app
Sentry.SentrySdk.CauseCrash(Sentry.CrashType.Native)
Expected Result
Crash is being reported to sentry.io
Actual Result
No crash is being reported (but app crashes with System.AccessViolationException: 'Attempted to read or write protected memory. This is often an indication that other memory is corrupt).
Happens both in debug as well as in Release mode
Hey @tipa, thanks for raising this. Could you provide us with some debug logs from the SDK so we can see what's going on?
Sure, I see these logs before the crash, during initialization:
Debug: Initializing Hub for Dsn: '...'.
Debug: This looks like a standard JIT/AOT application build.
Debug: Starting BackgroundWorker.
Debug: Registering integration: 'AutoSessionTrackingIntegration'.
Debug: BackgroundWorker Started.
Debug: Registering integration: 'AppDomainUnhandledExceptionIntegration'.
Debug: Registering integration: 'AppDomainProcessExitIntegration'.
Debug: Registering integration: 'SentryDiagnosticListenerIntegration'.
Info: DiagnosticSource Integration is disabled because tracing is disabled.
Debug: Registering integration: 'SystemDiagnosticsMetricsIntegration'.
Info: System.Diagnostics.Metrics Integration is disabled because no listeners are configured.
And then when I trigger the crash:
Debug: Triggering a deliberate exception because SentrySdk.CauseCrash(CrashType.Native) was called
An unhandled exception of type 'System.AccessViolationException' occurred in Sentry.dll
Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
You can reproduce it easily using this example: TestException.zip
I can definitely reproduce this... managed exceptions get caught by the global exception handler that Sentry wires up. This particularly native exception crashes the app, bypassing the global exception handler (and any other exception handling I try to setup).
I've tried surrounding the code in a simple try..catch, but the catch never executes. I've tried setting up a custom SynchronizationContext but the exception never gets posted. I'm not sure if it's possible to catch these kinds of exceptions in WinUI.
Sorry for the delay and the misleading CauseCrash utils. Unfortunately, the SDK does not support native crashes outside of AOT. We're going to guard the "demo" by a check and log instead of doing nothing.
@bruno-garcia tried adding and providing native support for regular JIT compiled applications over here https://github.com/getsentry/sentry-dotnet-minidump/issues/1 but that ran into a whole lot of issues.
@bruno-garcia tried adding and providing native support for regular JIT compiled applications over here https://github.com/getsentry/sentry-dotnet-minidump/issues/1 but that ran into a whole lot of issues.
To be clear it worked, but not the way you'd take the most value out of it. For example, there was no C# function names. I only saw C++ code, seemed like JIT generated code and less user-written code and no C#->C stack traces. SO it would require more work to get good support
I'm reopening this because we need to support capturing crashes in any sort of app. So this is a valid feature request imo
@bruno-garcia Do you have an update on this? We have a production application that is relying on Sentry to report all crashes. This could be a deal breaker if it's not fixed in the near future. We see reports on AppCenter so it seems like it should be possible.
Do you have an update on this? We have a production application that is relying on Sentry to report all crashes. This could be a deal breaker if it's not fixed in the near future. We see reports on AppCenter so it seems like it should be possible.
Unfortunately only a small fraction of our .NET users are running WinUI at this time. So while our goal is to have support for all platforms .NET can run, we have not been able to prioritize this yet because we have literally hundreds of orgs coming in every week into Sentry running their apps on iOS and Android. And the issue tracker for those platforms has been taking over all our bandwidth.
The good news is that next month we have a new team member joining the team. While I expect it'll take some time for him to ramp up, hopefully this is something we can deliver at some point after that.
It looks like some first tests are being done in #4297.
An alternative to an unhandled exception filter (which doesn't seem to work) could be using the WerRegisterRuntimeExceptionModule API.
It runs out of process and is being invoked by the OS after the process crashes, and it able to catch everything including fatal access violations etc. Actually crashpad is also using this, but only as a "last resort" for limited exception types. As found out in https://github.com/getsentry/sentry-dotnet/issues/2076 https://github.com/getsentry/symbolicator/issues/1731 Crashpad does not work well with .NET however, since the minidump created by Crashpad is missing information.
Instead, Sentry could implement such a WER DLL on its own and use MiniDumpWriteDump from Windows to create the minidump, which supports .NET fine.
Two problems with this approach arise in combination with MSIX unfortunately - but they can be worked around.
First, due to registry virtualization with MSIX the DLL is only registered in the MSIX container, not in the "real" registry, so the WER process cannot find it. (https://github.com/getsentry/sentry-native/issues/923) This can be worked around by starting reg.exe as an external process, ensuring it does run outside of the MSIX package identity and can write in the real registry.
The second issue is that the WER DLL cannot be loaded by the external WER process from the MSIX package itself. WER would try to load the DLL from the package, but fail with "access denied" as MSIX only grants DLL execute permissions to processes with the same MSIX identity. A workaround is to simply copy the WER DLL to another location, like tmp folder, which the WER process can access fine. The WER DLL must be compiled static, so no MSVC C++ redistributable is required at runtime.
Overall this approach seems to work, and it would be great if Sentry could do the work to bring all the pieces together so it can be easily used by everyone.
@BernhardMarconato thanks for the detailed research!
We have a tiny team working on the .NET SDK and so we have to pick our battles carefully. That solution sounds quite complex (high effort) and we don't have much demand for WinUI (low impact)... realistically then, we're very unlikely to prioritise building and maintaining this.
Ideally we'd find a solution that leverages the work we've already done for other platforms, as WinUI is just one of many technologies that we maintain integrations for. We can't afford to do a lot of bespoke engineering to support it.
I can see your point regarding maintenance effort. But to clarify, this approach would work with all Windows apps (C++, .NET, etc.) and would avoid dependency on Crashpad. The MSIX considerations apply to all MSIX packaged apps, not just WinUI apps (there are more and more of them, e.g. electron apps like ChatGPT or Slack from the Microsoft Store).
@BernhardMarconato any chance you'd be willing to contribute this to Sentry?
We're hiring for the team (Toronto or Vienna offices) but until we're able to grow that team. We're pretty overloaded with maintaining what we built and shipping Logs support right now.
re: electron apps, some of those you mentioned already use Sentry and electron bundles crashpad and our SDk for Electron leverages that ootb
@bruno-garcia Yes, I fully understand that resources are limited here. I'll see what can be done, maybe in a proof of concept state.
Crashpad works for the mentioned apps when MSIX packaged generally, but I'm pretty certain not the crashpad_wer handling for the two otherwise uncatchable exception types STATUS_FAIL_FAST_EXCEPTION STATUS_STACK_BUFFER_OVERRUN (which luckily do not happen that often though).
I see, that's interesting. @timfish does sentry electron capture those? Can we partner here on something done on sentry-native perhaps so we can capture these on sentry-electron, sentry-dotnet and stand alone sentry-native perhaps?
I see, that's interesting. @timfish does sentry electron capture those?
Not sure to be honest. Electron just uses Chromium with it's default bundled crashpad so we make do with what's available.
In its current configuration Electron crashpad does not capture minidumps for abort() on windows:
- https://github.com/electron/electron/issues/36862
Even if we could make changes and improve crashpad here, we'd either need to upstream our changes into Chromium or persuade Electron to apply the changes in a patch.
@supervacuus might know about this stuff too ^
Yes, I am fully aware of the option that @BernhardMarconato describes and raised it in multiple conversations as an alternative native backend for Windows.
There are numerous upsides for (direct) native users, like bypassing issues with dependencies (and sometimes even drivers) that abuse/overwrite the UEF at arbitrary points in application execution, at least as long as WerFault.exe gets to execute the module.
The main downside RE: integration with the remaining SDK is that the code runs inside WerFault.exe, an executable we do not control, so we need to create a new interface channel for that module (which is not a show-stopper). Similarly, on_crash/before_send hooks would require a specific solution if we added this backend, but I have already explored this approach a bit.
It boils down to prioritization, similar to Android Tombstone handling.
I've added a WER backend to Sentry Native in my fork, giving another option besides Crashpad.
As described in previous comments, it can handle any kind of exception, including fast-fail exceptions as it runs out-of-process; and does not rely on SetUnhandledExceptionFilter which doesn't properly work in WinUI apps. It uses a similar approach like in the Crashpad handler to save breadcrumbs/tags/etc. in files from the main app, and once the main app crashes the WER handler reads and uploads them together with a Minidump created using MiniDumpWriteDump. AI assisted in writing the code.
In my testing, it has worked well with the sentry_example.exe app and also with a MSIX packaged C++ app. I did not yet test with a C# .NET/WinUI 3 app yet because it's a bit tedious to reference native libraries from C#. But I expect it will work there as well, and will try once I have time. (In a C# app, sentry-native and sentry-dotnet would both need to be used to handle the entire bandwidth of C# exceptions to C++ exceptions to fatal access violation/fastfail exceptions).
Short steps to reproduce: Compile with -DSENTRY_BACKEND=wer -DSENTRY_BUILD_RUNTIMESTATIC=ON (static runtime recommended for MSIX, as we cannot rely on the MSIX C++ runtime dependency). Then set SENTRY_DSN env variable and run sentry_example.exe fastfail. Use tools like DebugView to verify the WER DLL was successfully executed on crash by the Windows WerFault.exe. Confirm the event is received on Sentry.
on_crash/before_send hooks do not work due to the out-of-process nature, but I personally wouldn't need them anyways.
Feel free to take a look and let me know your thoughts. Is this something that would be useful for you, and should I try to create a merge request into your repo? Alternatively, if this doesn't fit your coding guidelines or requirements, you're welcome to use or adapt the code from my fork however you see fit.
I tried it today from a C# WinUI 3 app, and it generally works (the dump is created and uploaded). I provoked a fastfail crash with Environment.FailFast() when clicking on a button. I initialized Sentry native using attached PInvoke code.
However, on the Sentry website the dump still does not show symbolicated function names for .NET code (same as described in https://github.com/getsentry/symbolicator/issues/1731). The minidump file, when opened from the cache directory on the PC, contains all the .NET information when opened in WinDbg or Visual Studio however. So there still seems to be a serverside problem with symbolicating .NET minidump files.
SentryNative PInvoke code C#
internal partial class SentryNative
{
private const string SentryLibrary = "sentry.dll";
// Opaque pointer types represented as IntPtr
public struct SentryOptions { }
public struct SentryValue
{
public ulong Bits;
}
// Enums
public enum SentryLevel
{
Debug = -1,
Info = 0,
Warning = 1,
Error = 2,
Fatal = 3
}
// Core options functions
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial IntPtr sentry_options_new();
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_options_free(IntPtr opts);
[LibraryImport(SentryLibrary, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller))]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_options_set_dsn(IntPtr opts, string dsn);
[LibraryImport(SentryLibrary, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller))]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_options_set_database_path(IntPtr opts, string path);
[LibraryImport(SentryLibrary, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller))]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_options_set_release(IntPtr opts, string release);
[LibraryImport(SentryLibrary, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller))]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_options_set_environment(IntPtr opts, string environment);
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_options_set_debug(IntPtr opts, int debug);
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_options_set_sample_rate(IntPtr opts, double sample_rate);
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_options_set_auto_session_tracking(IntPtr opts, int val);
// Core SDK functions
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial int sentry_init(IntPtr options);
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial int sentry_close();
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial int sentry_flush(ulong timeout);
// Event capture functions
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial SentryValue sentry_value_new_event();
[LibraryImport(SentryLibrary, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller))]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial SentryValue sentry_value_new_message_event(SentryLevel level, string logger, string text);
[LibraryImport(SentryLibrary, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller))]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial SentryValue sentry_value_new_string(string value);
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial SentryValue sentry_capture_event(SentryValue evt);
// Tag and context functions
[LibraryImport(SentryLibrary, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller))]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_set_tag(string key, string value);
[LibraryImport(SentryLibrary, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller))]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_set_extra(string key, SentryValue value);
[LibraryImport(SentryLibrary, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller))]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_set_context(string key, SentryValue value);
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_set_level(SentryLevel level);
// User functions
[LibraryImport(SentryLibrary, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller))]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial SentryValue sentry_value_new_user(string id, string username, string email, string ip_address);
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_set_user(SentryValue user);
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_remove_user();
// Breadcrumb functions
[LibraryImport(SentryLibrary, StringMarshalling = StringMarshalling.Custom, StringMarshallingCustomType = typeof(System.Runtime.InteropServices.Marshalling.AnsiStringMarshaller))]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial SentryValue sentry_value_new_breadcrumb(string type, string message);
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_add_breadcrumb(SentryValue breadcrumb);
// Session functions
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_start_session();
[LibraryImport(SentryLibrary)]
[UnmanagedCallConv(CallConvs = [typeof(System.Runtime.CompilerServices.CallConvCdecl)])]
public static partial void sentry_end_session();
// Helper method to initialize Sentry with basic configuration
public static bool Initialize(string dsn, string databasePath = ".sentry-native",
string release = null, string environment = null, bool debug = false)
{
try
{
var options = sentry_options_new();
if (options == IntPtr.Zero)
return false;
sentry_options_set_dsn(options, dsn);
sentry_options_set_database_path(options, databasePath);
if (!string.IsNullOrEmpty(release))
sentry_options_set_release(options, release);
if (!string.IsNullOrEmpty(environment))
sentry_options_set_environment(options, environment);
sentry_options_set_debug(options, debug ? 1 : 0);
sentry_options_set_auto_session_tracking(options, 1);
return sentry_init(options) == 0;
}
catch
{
return false;
}
}
// Helper method to capture a simple message
public static void CaptureMessage(string message, SentryLevel level = SentryLevel.Info)
{
try
{
var evt = sentry_value_new_message_event(level, null, message);
sentry_capture_event(evt);
}
catch
{
// Silently fail to avoid crashing the application
}
}
// Helper method to shutdown Sentry
public static void Shutdown()
{
try
{
sentry_close();
}
catch
{
// Silently fail
}
}
}
@BernhardMarconato. thank you so much for your efforts!
@supervacuus @JoshuaMoelans @vaind Is this something we could/should have in sentry-native,
or more something to add to a .NET WinUI 3 package (see getsentry/sentry-dotnet#3712, and considering getsentry/sentry-dotnet#4497)?
@supervacuus @JoshuaMoelans @vaind Is this something we could/should have in sentry-native
We should definitely provide this from the Native SDK, since it is helpful for direct and downstream usage. I already requested to spend time on this this fall.
However, it is entirely unclear to me whether adding the capability for a .NET WinUI 3 package should depend on such an implementation in the Native SDK, since there is currently only minimal integration in terms of enrichment.
I added a simple .NET test app repo where native crash handling can be reproduced, together with my "experimental" sentry-native fork. A readme with instructions can be found there.
The main issue, as mentioned earlier, is that Sentry on the backend doesn't appear to properly symbolicate .NET functions in the native crash stack traces; even if all PDB files are uploaded. I hope this repro makes it easier to get to the bottom of the problem.
Thanks for providing a test repro @BernhardMarconato, it's much appreciated. Now let's see what @Flash0ver can make of it :D