Stack Trace missing for NativeAOT App

[emrys90]

Package

Sentry

.NET Flavor

.NET Core

.NET Version

8.0

OS

Linux

SDK Version

4.7.0

Self-Hosted Sentry Version

24.6.0

Steps to Reproduce

I'm having an issue where the client is not exporting stack traces to the API for a NativeAOT compiled app. Works fine in a non-NativeAOT app.

Expected Result

Event is sent with stack traces.

Actual Result

Event is sent with no stack traces.

[jamescrosswell]

I played around to try to reproduce this with the following program. There is a stack trace, but it only has one frame, and that doesn't appear to be anything to do with Sentry (note the Console.WriteLine statements in the catch block):

SentrySdk.Init(options =>
{
    options.Dsn = "...YOUR DSN HERE...";
    options.IsGlobalModeEnabled = true;
    options.TracesSampleRate = 1.0;
});

Console.WriteLine("Starting transaction...");

var transaction = SentrySdk.StartTransaction("Program Main", "function");
SentrySdk.ConfigureScope(scope => scope.Transaction = transaction);

await ParentFunction();

transaction.Finish();
Console.WriteLine("Transaction Finished!");
return;

async Task ParentFunction()
{
    await Task.Delay(500);
    await RecursiveFunction(5);
}


async Task RecursiveFunction(int depth)
{
    var span = transaction.StartChild("function", nameof(RecursiveFunction));

    try
    {
        // Throw an exception
        if (depth <= 0)
        {
            throw new ApplicationException("Something happened!");
        }
        await RecursiveFunction(depth - 1);
        span.Finish();
    }
    catch (Exception exception)
    {
        // This is an example of capturing a handled exception.
        Console.WriteLine($"Capturing exception: {exception.Message}");
        Console.WriteLine(exception.StackTrace);
        SentrySdk.CaptureException(exception);
        span.Finish(exception);
    }
}

And what I see as output:

Starting transaction...
Capturing exception: Something happened!
   at Program.<>c__DisplayClass0_0.<<<Main>$>g__RecursiveFunction|3>d.MoveNext() in /Users/jamescrosswell/code/ConsoleAot/Program.cs:line 35
Transaction Finished!

This appears to be a limitation of AOT then.

[jamescrosswell]

Ah, interestingly if I remove the try...catch block and just let the exception surface, we capture the native exception and for that we get a full stack trace.

SentrySdk.Init(options =>
{
    options.Dsn = "...YOUR DSN HERE...";
    options.IsGlobalModeEnabled = true;
    options.TracesSampleRate = 1.0;
});

Console.WriteLine("Starting transaction...");

var transaction = SentrySdk.StartTransaction("Program Main", "function");
SentrySdk.ConfigureScope(scope => scope.Transaction = transaction);

ParentFunction();

transaction.Finish();
Console.WriteLine("Transaction Finished!");
return;

void ParentFunction()
{
    RecursiveFunction(5);
}


void RecursiveFunction(int depth)
{
    var span = transaction.StartChild("function", nameof(RecursiveFunction));

    // Throw an exception
    if (depth <= 0)
    {
        throw new ApplicationException("Something happened!");
    }
    RecursiveFunction(depth - 1);
    span.Finish();
}

And here's the output from that:

Starting transaction...
Unhandled exception. System.ApplicationException: Something happened!
   at Program.<>c__DisplayClass0_0.<<Main>$>g__RecursiveFunction|3(Int32 depth) in /Users/jamescrosswell/code/ConsoleAot/Program.cs:line 32
   at Program.<>c__DisplayClass0_0.<<Main>$>g__RecursiveFunction|3(Int32 depth) in /Users/jamescrosswell/code/ConsoleAot/Program.cs:line 32
   at Program.<>c__DisplayClass0_0.<<Main>$>g__RecursiveFunction|3(Int32 depth) in /Users/jamescrosswell/code/ConsoleAot/Program.cs:line 32
   at Program.<>c__DisplayClass0_0.<<Main>$>g__RecursiveFunction|3(Int32 depth) in /Users/jamescrosswell/code/ConsoleAot/Program.cs:line 32
   at Program.<>c__DisplayClass0_0.<<Main>$>g__RecursiveFunction|3(Int32 depth) in /Users/jamescrosswell/code/ConsoleAot/Program.cs:line 32
   at Program.<>c__DisplayClass0_0.<<Main>$>g__RecursiveFunction|3(Int32 depth) in /Users/jamescrosswell/code/ConsoleAot/Program.cs:line 32
   at Program.<>c__DisplayClass0_0.<<Main>$>g__ParentFunction|2() in /Users/jamescrosswell/code/ConsoleAot/Program.cs:line 21
   at Program.<Main>$(String[] args) in /Users/jamescrosswell/code/ConsoleAot/Program.cs:line 13
zsh: abort      ./bin/Release/net8.0/osx-arm64/ConsoleAot

[emrys90]

There are other work arounds as well for getting a stack trace on NativeAOT. If you do exception.StackTrace.ToString() it'll give a result that can be attempted to be parsed. Not sure if there's a better way to handle it or not.

I'm switching to Sentry from BugSnag, and they did have stack traces for my NativeAOT app.

[jamescrosswell]

If you do exception.StackTrace.ToString() it'll give a result that can be attempted to be parsed. Not sure if there's a better way to handle it or not.

We can't really use StackTrace.ToString(). That's missing loads of information that we need to do things like generating Enhanced Stack Frames. If you did want to capture this though, you could easily do this in the options... something like:

    options.SetBeforeSend(ev =>
    {
        if (ev.Exception is { } exception)
        {
            ev.SetExtra("Stack Trace", $"{exception.StackTrace}");
        }
        return ev;
    });

We did open a discussion with Microsoft about this and it looks like there might be something coming in net9.0 that will help.

I'm switching to Sentry from BugSnag, and they did have stack traces for my NativeAOT app.

I tried the sample applications a copied/pasted above with BugSnag and saw exactly the same results. If I do a try...catch followed by a bugsnag.Notify then I only see a single line in the stack trace on BugSnag. On the other hand, if I let the exception bubble up, a full native stack trace is captured (although the application obviously terminates in this case).

If you've got an example of something that BugSnag is capturing but Sentry is not, I'd love to learn more.

[emrys90]

Hmm weird, I was just doing a Bugsnag.Notify(exception, Severity.Error) and it would capture stack traces, just not line numbers.

[emrys90]

To work around this issue I'm trying out the ISentryStackTraceFactory interface set through: options.UseStackTraceFactory(new NativeAotStackTraceFactory());

Do you see any potential issues with how I'm doing this? Here's an example of the stack trace:

   at System.Exception.SetCurrentStackTrace() + 0x63
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.SetCurrentStackTrace(Exception) + 0x18
   at ProjectCards.Utility.ConsoleLogger.WriteLog(NetLogLevel, String) + 0x1b7
   at ProjectCards.Utility.LogExtensions.WriteLog(NetLogLevel, String, String, Int32, String) + 0x2f9
   at ProjectCards.Networking.NetworkAuthenticator.<AfterUserLoaded>d__2.MoveNext() + 0x3ab

    public class NativeAotStackTraceFactory : ISentryStackTraceFactory
    {
        public SentryStackTrace Create(Exception exception = null)
        {
            Console.WriteLine("Source Stack Trace: " + exception?.StackTrace);

            var trace = new SentryStackTrace();
            var frames = new List<SentryStackFrame>();
            
            var lines = exception?.StackTrace?.Split(System.Environment.NewLine) ?? Array.Empty<string>();
            foreach (var line in lines)
            {
                var match = Regex.Match(line, @"at (.+)\.(.+) \+");
                if (match.Success)
                {
                    frames.Add(new SentryStackFrame()
                    {
                        Module = match.Groups[1].Value,
                        Function = match.Groups[2].Value
                    });
                }
                else
                {
                    Console.WriteLine($"Regex match failed for: {line}");
                    frames.Add(new SentryStackFrame()
                    {
                        Function = line
                    });
                }
            }

            trace.Frames = frames;
            return trace;
        }
    }

[bitsandfoxes]

Do you see any potential issues with how I'm doing this?

I actually don't. @jamescrosswell would this be a valid workaround we could put into the SDK when we know we don't get anything better?

[jamescrosswell]

Do you see any potential issues with how I'm doing this?

I actually don't. @jamescrosswell would this be a valid workaround we could put into the SDK when we know we don't get anything better?

As a fallback, it might be great 🚀 We may or may not be able to get it to work with things like InApp frame detection and pretty sure we can't do fancy stuff like Enhanced stack traces and source mapping but it'd be a hell of a lot better than nothing right?

btw: Apologies for the delay - I've been out sick 😷.

[filipnavara]

The .NET 9 DiagnosticMethodInfo API has landed, btw. It would be nice to test it before it gets released later this year to be sure there's no bugs or functionality gaps.

[bitsandfoxes]

I'm adding this to the 5.0 milestone as it fits with the .NET9 support we're working on.

[jamescrosswell]

The .NET 9 DiagnosticMethodInfo API has landed, btw. It would be nice to test it before it gets released later this year to be sure there's no bugs or functionality gaps.

So I'm a year late, but I just did a bit of digging into that new class:

• https://github.com/getsentry/sentry-dotnet/issues/3691

From what I can tell then, I don't think it will help much with Sentry. There may be odd bits of reflection that we can replace, but not any of the core functionality that is currently unavailable for AOT compiled applications.

[jamescrosswell]

wrt this issue then, the only thing I can think that we could do is to parse the StackTrace.ToString similar to what @emrys90 put together above.

However, I'm not how we'd decide when to use this StackTrace.ToString method vs what we have now.

It looks like we already use this fallback (for frames, not entire stacks) in certain situations where it's clear we can't get anything better: https://github.com/getsentry/sentry-dotnet/blob/daaa5cad555ae586833e290ccb3cf866c26659d6/src/Sentry/Internal/DebugStackTrace.cs#L213-L218

If we had no frames, we could try to parse the StackTrace.ToString(). However I haven't yet encountered a scenario where we get no frames...

The only other options that I can think of is that we provide an implementation of ISentryStackTraceFactory and SDK users can wire this up manually if they want (i.e. if they're unhappy with the stack traces they're getting from the SentryStackTraceFactory).

[Flash0ver]

After a chat, we are thinking about

• expose a new public ISentryStackTraceFactory implementation
  • in namespace Sentry.Extensibility
  • mark it [Experimental] (so we can tinker with the name and means of initialization after gathering feedback)
  • naming ... something like "StringParsingStackTraceFactory" or "StringStackTraceFactory"
• document an example with SentryOptions.UseStackTraceFactory(ISentryStackTraceFactory) in "Troubleshooting"

[filipnavara]

This is pretty much what we ended up doing with .NET 8 (and it continues to work on .NET 9):

Detection of need for NativeAOT stack trace factory:

[SuppressMessage("Trimming", "IL2026:Members annotated with 'RequiresUnreferencedCodeAttribute' require dynamic access otherwise can break functionality when trimming application code", Justification = "<Pending>")]
private bool NeedNativeAotStackTraceFactory()
{
	// Try to detect NativeAOT
	if (!System.Runtime.CompilerServices.RuntimeFeature.IsDynamicCodeSupported)
	{
		var stackTrace = new StackTrace(false);
		if (stackTrace.GetFrame(0)?.GetMethod() is null)
		{
			Debug.WriteLine("Using NativeAotStackTraceFactory");
			return true;
		}
	}

	return false;
}

Initialization:

builder
	.UseSentry(optionsSentry =>
	{
		...

		// Try to detect NativeAOT and swap for simple stack trace parsing
		if (NeedNativeAotStackTraceFactory())
		{
			optionsSentry.UseStackTraceFactory(new NativeAotStackTraceFactory());
		}
	})

Implementation of the stack trace factory:

class NativeAotStackTraceFactory : ISentryStackTraceFactory
{
	[GeneratedRegex(@"at (.+)\.(.+) \+")]
	private static partial Regex StackTraceLineRegex();

	public SentryStackTrace Create(Exception exception = null)
	{
		Debug.WriteLine("Source Stack Trace: " + exception?.StackTrace);

		var trace = new SentryStackTrace();
		var frames = new List<SentryStackFrame>();
		
		var lines = exception?.StackTrace?.Split(System.Environment.NewLine) ?? Array.Empty<string>();
		foreach (var line in lines)
		{
			var match = StackTraceLineRegex().Match(line);
			if (match.Success)
			{
				frames.Add(new SentryStackFrame()
				{
					Module = match.Groups[1].Value,
					Function = match.Groups[2].Value
				});
			}
			else
			{
				Debug.WriteLine($"Regex match failed for: {line}");
				frames.Add(new SentryStackFrame()
				{
					Function = line
				});
			}
		}

		trace.Frames = frames;
		return trace;
	}
}

[jamescrosswell]

This is pretty much what we ended up doing with .NET 8 (and it continues to work on .NET 9):

Awesome, thanks @filipnavara !

btw is this just for performance reasons?

if (!System.Runtime.CompilerServices.RuntimeFeature.IsDynamicCodeSupported)

I assume if you removed that, stackTrace.GetFrame(0)?.GetMethod() would return null in a trimmed application regardless... but creating a stack trace is reasonably expensive.

We were using almost exactly this technique to detect whether trimming was enabled or not up until a recently. In the 5.12.0 release we've got a source generator that detects build properties and makes these available at runtime... I'm still a little bit nervous about our logic for inferring whether trimming is enabled or not based on the build properties though... It's per the docs but I'm worried there are edge cases. Potentially what you have (and what we had before) is more reliable.

[filipnavara]

I assume if you removed that, stackTrace.GetFrame(0)?.GetMethod() would return null in a trimmed application regardless... but creating a stack trace is reasonably expensive.

System.Runtime.CompilerServices.RuntimeFeature.IsDynamicCodeSupported is performance optimization. We know it's never going to be true for NativeAOT runtime. It can be true/false for other runtimes depending on various options.

There's an important difference between trimmed and NativeAOT. Trimming can be performed with MonoVM (mobile workloads) and CoreCLR too (PublishTrimmed=true). Both of these case will come with full reflection support for the code that didn't get trimmed and hence stackTrace.GetFrame(0)?.GetMethod() != null there.