Debug info for OpenTelemetry exceptions

[jamescrosswell]

We currently only have partial information for exceptions captured via OpenTelemetry's RecordException like this:

        using var activity = Telemetry.ActivitySource.StartActivity("Funny business");
        try
        {
            throw new Exception("test activity.RecordException");
        }
        catch (Exception e)
        {
            activity?.RecordException(e);
        }

PR #102905 introduces a change that allows us to capture this information properly. Specifically, it adds a NotifyActivityAddException callback that we could register to be notified of the original exception. It looks like this will only be available in net9.0.

Historical context

The Activity class in net8.0 and below doesn't provide anywhere explicitly to put the Exception and OpenTelemetry only stores the Exception Type, Message and a string representation of the Exception.

To indicate that there was some kind of error, we're currently creating a fairly poor/partial representation of the original exception that we send to Sentry. This Exception has the same type as the original exception and the same message, but doesn't have any stack trace or any of the other details (e.g. InnerException)... our middleware will provide a stack trace but this only gets as close as the entry point on the method where the exception was thrown and caught - it won't give the exact line where the exception was thrown.

We do log the OpenTelemetry "stack trace" (which is actually Exception.ToString() - not a real stack trace... and that contains a more accurate description of where the exception was raised. However we aren't able to associate that with debug info, source mapping etc. so, again, it's suboptimal.

Ultimately we would like to get resolution to either or both of:

• https://github.com/dotnet/runtime/issues/53641#issuecomment-966773529
• https://github.com/open-telemetry/opentelemetry-dotnet/issues/2439#issuecomment-1577314568

Possible Solutions for net8.0 and below

ASP.NET Core apps

One exception/loophole appears to be the Asp.NET Core instrumentation, which provides an EnrichWithException callback option that would allow us to get at the underlying exception (at least for ASP.NET Core applications).

Non-ASP.NET Core apps

Another possible solution, using the FirstChanceException, was suggested in a discussion in the OTel repo about this:

• https://github.com/dotnet/runtime/issues/53641#issuecomment-2102042043