Sentry Structured Logging for .NET

[AbhiPrasad]

https://develop.sentry.dev/sdk/telemetry/logs/

Sentry is adding support for structured logging. Let's add it to the .NET SDK!

1. Define the logs protocol and log envelope item in the SDK
2. Add the Public API (SDK options and methods) as per the docs
3. Make sure the the SDK follows the documented behavior
4. Attach default attributes to the SDK as per docs
5. Instrument popular logging libraries to send logs to Sentry. SDK maintainers can best decide what they should support, but we should definitely support the same integrations that we do in the SDK (NLog, Serilog, etc.)
6. Create a GH discussion that contains instructions for setting up the SDK in your repo. Eventually this will be moved into the primary docs. See the JS SDK's GH discussion for inspiration.

[bruno-garcia]

The idea is to:

1. Ship the barebone Sentry API into the core Sentry package
2. Make this work from Microsoft.Extensions.Logging
3. Make this work from Serilog
4. Make this work from NLog
5. Make this work from log4net

I'll dare to say that no one in .NET will use "Sentry's logging API". I know I wouldn't. But that's not the point. The point is to have the API in the main SDK, so that the logging libraries integrations only pipe data to it.

The order above allows us to try/test things incrementally. Once we get to n2, we can add it to nuget-trends and symbol-collector for dogfooding.

All our current logging integrations do:

1. Check level, if higher than MininumEventLevel it records an event
2. Check level, if higher than MinimumBreadcrumbLevel it adds a breadcrumb

e.g:

https://github.com/getsentry/sentry-dotnet/blob/084e3b4bf2c043b42ad6b11892a604e027b3d1a5/src/Sentry.Extensions.Logging/SentryLogger.cs#L39-L86

The crumb goes after, so that in a subsequent event, it will include the breadcrumb of the previous event.

This works well for error monitoring. But with the new logging integration, we'll send all logs to Sentry. The new logging could be a new logger class, that's added in of logging is enabled. Since little is shared from the current logger described above, and the new one.

[Flash0ver]

Sentry Structured Logging for .NET

• [ ] Add Sentry-Logger APIs
  • See https://develop.sentry.dev/sdk/telemetry/logs/
• [ ] Integration: Sentry.Extensions.Logging
  • public CaptureLog on IHub and ISentryClient
  • and/or internal Extension-Methods receiving IHub and ISentryClient
  • ensure SDK-Name/Version is set correctly, test via Sentry.AspNetCore and Sentry.Maui
  • see also https://github.com/getsentry/sentry-dotnet/pull/4158#discussion_r2089074276
• [ ] more buffering for Sentry.Extensions.Logging
  • add support for the new Microsoft.Extensions.Logging.Abstractions.IBufferedLogger interface
  • next to the original Microsoft.Extensions.Logging.ILogger
• [ ] Integration: Sentry.AspNetCore
• [ ] Integration: Sentry.AspNetCore.Blazor.WebAssembly
• [ ] Integration: Sentry.Azure.Functions.Worker
• [ ] Integration: Sentry.Maui
• [ ] Integration: Sentry.Serilog
• [ ] Integration: Sentry.NLog
• [ ] Integration: Sentry.Log4Net
• [ ] Internal APIs for Sentry.Unity
  • via internal Extension-Methods receiving IHub and ISentryClient
• [ ] Processing: Batching / Priority
  • see also https://github.com/getsentry/sentry-dotnet/pull/4158#discussion_r2069151972
  • see also https://github.com/getsentry/sentry-dotnet/pull/4158#discussion_r2089059281
• [ ] Sentry-Logger-API: Accelerator overloads for convenience
  • see also https://github.com/getsentry/sentry-dotnet/pull/4158#discussion_r2089046053
• [ ] Sentry-Logger-API: Generic overloads
  • 3 overloads (with 1, 2 and 3 generic arguments) per Log{Level} method
  • to avoid both boxing and array allocations when EnableLogs is false
  • see also https://github.com/getsentry/sentry-dotnet/pull/4158#discussion_r2069150349
• [ ] Sentry-Logger-API: [InterpolatedStringHandler]
• [ ] Documentation
  • see also https://github.com/getsentry/sentry-dotnet/pull/4158#discussion_r2071291517

[wondertalik]

Hey,

Thanks for this feature that are you working on. Any plans to adapt opentelemetry for logs as you did for tracing?

[AbhiPrasad]

Hey @wondertalik - thanks for your interest! At the current moment we are focusing on adding logs support via the Sentry SDKs. We'll evaluate OpenTelemetry support in the future. Our logs schema is a superset of the OpenTelemetry schema, so we are already compatible with OpenTelemetry.

[jamescrosswell]

There aren't yet/currently any docs for OpenTelemetry logging but it looks like even if you were using it, the instrumentation would still be done via ILogger<T> (i.e. the same way you'd instrument an app using Micosoft.Extensions.Logging). Anything logged to ILogger<T> can be captured by both Sentry (using the MEL integration) and OpenTelemetry.

I'm not sure if/why we'd ever want to add OpenTelemetry as a middle man in the process of piping logs from ILogger<T> to Sentry. About the only reason I can think of to do this would be so that any OpenTelemetry Resources could be shared by Logging and Tracing (assuming you're using OpenTelemetry for tracing as well)... I think Resources basically define a set of tags that would be applied to all traces and logs captured by an application.

@wondertalik was your main interest in ensuring logs you captured via ILogger<T> ended up in Sentry (something that can be done with the MEL integration already) or was there something else you needed from OpenTelemetry logging?

[wondertalik]

@jamescrosswell basically is unify setup when use sentry and opentelemetry together. Opentelemetry becomes a standart across all the clouds. The main benefit is posibility to send all telemetry to otel collector and via exporters resend where do i need trasparenty for all dev teams (services). I'm doing right now with tracing

For example

• for dev in my laptop i use otel collector + seq + aspire dashboard + jaeger + prometheus.
• for prod - otel collector, sentry, signoz

[jamescrosswell]

I see. That's not how Sentry's OTel integration for tracing works either then... For tracing we have a SpanProcessor - so we capture OTel spans in real time and send these to Sentry (there's no exporter involved).

So basically you can instrument your application with OTel for logging and tracing and have the logs/traces captured by Sentry, but OTel exporters aren't used at any point for either of those things.

[Syed-RI]

Is this going to count towards the quota? For reference, AppCenter didnt have cap or quota for the number log messages but had a retention policy of these logs for 30 days(IIRC). Apologies if its already documented somewhere and I have missed it.

[AbhiPrasad]

@Syed-RI check out this GH discussion for details about pricing/quota: https://github.com/getsentry/sentry/discussions/86804#discussioncomment-13163256. Feel free to comment on that thread if you have any follow up concerns.

[jason-kingsmill]

Is there any update for an ETA on this? It's the key requirement stopping us from fully moving to Sentry

[Flash0ver]

Thanks @jason-kingsmill for voicing interest! We expect to have an experimental pre-release ready next week, where we would appreciate your early feedback! I'll get back to you here once the package has been published.

[Syed-RI]

Same here, waiting!

[Syed-RI]

@Flash0ver has there been a pre-release or the team requires more time? :)

[Flash0ver]

Not yet. I'm afraid we need a bit more time. The chunks left are "Buffering/Batching" and the Microsoft.Extensions.Logging-integration (initially used by AspNetCore Maui). Both changes are currently under review, where I still have some improvements left to do. We'll keep you posted once we got the (experimental) Release published.

[Syed-RI]

@Flash0ver seeing https://github.com/getsentry/sentry-dotnet/pull/4158 last week, is this available now as an experimental feature? If so, please point me to MAUI specific docs (initialisation + example usage etc 🙏🏽

[Flash0ver]

Indeed. I was just about to write an update:

• Sentry Logs shipped in pre-release 5.12.0-alpha.0
  • for internal dogfooding ... see Symbol Collector:
    • getsentry/symbol-collector#236
    • getsentry/symbol-collector#238
  • you are most welcome to give the pre-release a try and provide early feedback
  • I'll open up a feedback discussion later this week
• we also aim to complete the open work on the API-shape, buffering/batching, and iron out the bugs found during initial dogfooding, for an upcoming non-pre-release (yet still marked "Experimental" until the Sentry Logs feature goes GA)
• docs are still TODO ... I intend to author them later this week
• for usage examples - until documentation is published - please see our Samples:
  • Console App
  • Sentry.Extensions.Logging
  • ASP.NET Core
  • .NET MAUI

[Syed-RI]

Apologies, as I like to ask dumb questions... just so we are clear.... doing options.Experimental.EnableLogs = true; will enable logging and, any ILogger operation (_logger.Information("Hello Sentry!!!")) will result in a log event captured in the portal? I'll wait for a more comprehensive docs.

Thanks for all your hard work!

[Flash0ver]

Exactly! Also ... no docs, no worries 😉

• SentryOptions.Experimental.EnableLogs = true;
  • enables SentrySdk.Experimental.Logger.Log..() to be batched and sent to Sentry
  • enables Sentry.Extensions.Logging integration
    • and with that integrates with usages of ILogger<TCategoryName>/ILogger
    • also in applications that use Sentry.AspNetCore or Sentry.Maui
• SentryOptions.Experimental.SetBeforeSendLog()
  • allows setting a callback that is invoked before logs are batched / scheduled to be sent to Sentry
    • when the delegate returns null, the log message is dropped
• SentryLoggingOptions.ExperimentalLogging.MinimumLogLevel
  • allows restricting the minimal Microsoft.Extensions.Logging.LogLevel of ILogger.Log..() messages to be batched and sent to Sentry
  • inherited by SentryAspNetCoreOptions and SentryMauiOptions as well

[KirylKvit-nxtec]

@Flash0ver Thank for providing us with the pre-release version! I've got one question regarding SentryExperimentalOptions.SetBeforeSendLog method. I wonder if it is possible to populate log object with additional properties that would be displayed when I inspect the log inside Sentry Logs panel? I'm interested in adding some properties that come from HttpContext. SentryAspNetCoreOptions.SetBeforeSend has version that accepts SentryEvent and SentryHint as args and I'm able to extract HttpContext from the SentryHint.Items property. Is there a way to do the same with SentryExperimentalOptions.SetBeforeSendLog? If not, will it be added in the future?

[Flash0ver]

Yes, via "Attributes" ... the API may change in an upcoming version, though.

Through the SentryLog instance, provided in the callback, you can call SentryLog.SetAttribute("key", value). "Attributes" are Key-Value-Pairs, with the following types supported:

• string
• bool
• 64-bit signed Integral numeric types
  • ulong will be converted to a string
• 64-bit floating-point numeric types
  • decimal will be converted to a string

I'll write documentation for the .NET SDK later this week. Until then, you may find more information on "Attributes" in our Dev-Docs: https://develop.sentry.dev/sdk/telemetry/logs/

[KirylKvit-nxtec]

Thanks for the quick response! What about the HttpContext part of the question? Here's what I can do with SentryAspNetCoreOptions.SetBeforeSend:

options.SetBeforeSend((sentryEvent, sentryhint) =>
{
        if (!sentryHint.Items.TryGetValue("httpContext", out var httpContextObj) || httpContextObj is not HttpContext httpContext)
        {
               return sentryEvent;
        }
        // Use httpContext
        return sentryEvent;
});

However, SentryExperimentalOptions.SetBeforeSendLog does not have and overload that has SentryHint in it. The question is: is there a plan to add SentryHint argument in the future? Is there a way to access HttpContext using current implementation? I understand that the current implementation is a subject to possibly change

[Flash0ver]

Ah ... I see. No, I'm afraid with the initial design, Sentry Logs don't support holding arbitrary data (apart from "Attributes"). Nor did we initially intend to integrate SentryHint.

However - now with your request - we could add a similar Items-Dictionary to SentryLog as well, which are not sent to Sentry but only used during the processing pipeline.

I'd first like to release Sentry Logs for .NET with the features required by the specification. I'll then later - probably end of this week - open up a feedback discussion, where I'd like to learn more about your use case, check demand for this and related scenarios, and discuss this feature further.

In the meantime, please take a peek at this sample: https://github.com/getsentry/sentry-dotnet/pull/4340/files#diff-24de9aa9dc673ac0931840d2d24adc7deb9a5a424d2114d4a27e0c8edbb1be5c Perhaps setting "Attributes" via SentrySdk.Experimental.Logger.Log..() and then trying to extract them in the SentryOptions.Experimental.SetBeforeSendLog callback could be a workaround ... to some degree.

[Flash0ver]

Sorry for the delay. And thank you for your patience. I am really excited about our plan to ship experimental Logs ([Experimental] in the SDK until Sentry Logs go GA) in the next minor release, v5.14.0. We have invested quite some time into a buffering/batching strategy with minimal thread locking, and are slaying the bugs and problems you have reported ... thank you for your feedback. Alongside the next minor release, we will invite you to more formal discussion threads, where we welcome more feedback and sharing experiences to pave the road for GA.

[RichardJECooke]

@Flash0ver I'm having trouble getting logs to run. Please let me know what I'm missing after trying to copy snippets from https://github.com/getsentry/sentry-dotnet/pull/4340/files.

I have in my .csproj file,

<PackageReference Include="Sentry.AspNetCore" Version="5.14.0-alpha.1" />

and in my Program.cs I have:

using System.Text.Json.Serialization;
using Microsoft.EntityFrameworkCore;
using TodoApi.Data;
using TodoApi.Middlewares;

DotNetEnv.Env.Load();

var builder = WebApplication.CreateBuilder(args);
builder.WebHost.UseSentry(options =>
{
    options.Environment = builder.Environment.EnvironmentName;
    options.TracesSampleRate = 1.0;
    options.ProfilesSampleRate = 1.0;
    options.Experimental.EnableLogs = true;
});

But I'm getting build error

/app/Program.cs(15,13): error CS1061: 'SentryAspNetCoreOptions' does not contain a definition for 'Experimental' and no accessible extension method 'Experimental' accepting a first argument of type 'SentryAspNetCoreOptions' could be found (are you missing a using directive or an assembly reference?)

[Flash0ver]

@RichardJECooke I was just about to post an update ... to cut to the chase: please update to 5.14.0 .. sorry for the confusion.

[Flash0ver]

We have now published 5.14.0, which includes Sentry Structured Logs for .NET. Thank you for your patience.

The APIs are still marked [Experimental] as we are genuinely excited to hearing your feedback. These include a set of SentrySdk.Experimental.Logger.Log.. APIs, and an integration into ILogger<T> for Sentry.Extensions.Logging in general, and both Sentry.AspNetCore and Sentry.Maui in particular.

We made a couple of improvements and bug fixes since the last release in 5.12.0-alpha.0, especially when it comes to performance and the Microsoft.Extensions.Logging.ILogger<TCategoryName> integration.

Documentation and an updated onboarding experience are currently in draft and will be published early next week, alongside starting a discussion thread on GitHub, where we are looking forward to your feedback about what we may improve (and what we should keep 😉).

[Dreamescaper]

Nice to have it published, thanks!

Feedback - it seems like exceptions, passed to ILogger, are not logged at all.

C#:

        catch (Exception e)
        {
            Logger.LogError(e, "Error during event #{MessageId} processing.", message.Id);
        }

JsonConsoleLogger:

{
    "EventId": 0,
    "LogLevel": "Error",
    "Category": "OnlineLessons.ApiService.Features.Integrations.Google.SyncSessionDeletedLambda",
    "Message": "Error during SNS message #df46f4f7-aa22-527b-bf57-95e2aabb8823 processing.",
    "Exception": "The service calendar has thrown an exception.\nHttpStatusCode is Gone.\nGoogle.Apis.Requests.RequestError\nResource has been deleted [410]\nErrors [\n\tMessage[Resource has been deleted] Location[ - ] Reason[deleted] Domain[global]\n]\n\nGoogle.GoogleApiException: The service calendar has thrown an exception. HttpStatusCode is Gone. Resource has been deleted\n   at Google.Apis.Requests.ClientServiceRequest`1.ParseResponse(HttpResponseMessage response)\n   at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync(CancellationToken cancellationToken)\n   at Google.Apis.Requests.ClientServiceRequest`1.ExecuteAsync()\n   at OnlineLessons.ApiService.Features.Integrations.Google.SyncSessionDeletedHandler.HandleAsync(SessionDeletedEvent request) in /home/runner/work/online-lessons-app/online-lessons-app/OnlineLessons.ApiService/Features/Integrations/Google/SyncSessionDeleted.cs:line 34\n   at OnlineLessons.ApiService.Common.Lambdas.SnsLambdaHandler`2.HandleAsync(SNSEvent snsEvent, ILambdaContext lambdaContext) in /home/runner/work/online-lessons-app/online-lessons-app/OnlineLessons.ApiService/Common/Lambdas/SnsLambdaHandler.cs:line 29\n   at OnlineLessons.ApiService.Common.Lambdas.SnsLambdaHandler`2.HandleAsync(SNSEvent snsEvent, ILambdaContext lambdaContext) in /home/runner/work/online-lessons-app/online-lessons-app/OnlineLessons.ApiService/Common/Lambdas/SnsLambdaHandler.cs:line 31",
    "State": {
        "Message": "Error during SNS message #df46f4f7-aa22-527b-bf57-95e2aabb8823 processing.",
        "MessageId": "df46f4f7-aa22-527b-bf57-95e2aabb8823",
        "{OriginalFormat}": "Error during SNS message #{MessageId} processing."
    }
}

Sentry:

[jamescrosswell]

Feedback - it seems like exceptions, passed to ILogger, are not logged at all.

@Dreamescaper that functionality should already exist with Sentry's MEL integration (i.e. doesn't require structured logging): https://github.com/getsentry/sentry-dotnet/blob/9ad2caa0c1e21296a90242c06e564d079a59020f/src/Sentry.Extensions.Logging/SentryLogger.cs#L55-L57

So exceptions should be captured and shown in the "Issues" feed.

@Flash0ver it might be nice if there was a link to related exceptions from Structured Logs though eh (same as we do for Trace Spans that have exceptions associated with them)?

I'm not sure if that's something the other SDKs have implemented or will implement. Perhaps the ability to pass an exception as an argument when logging is specific to .NET.

[Dreamescaper]

Link would be nice, yes, but also would be great to see error details "inline", without "jumping" through different pages to get the context (at least the exception message, as log entry could be pointless without it).

[AbhiPrasad]

For now errors (via captureException calls) and logs will go into separate UIs (errors in issues, and logs in the explore page), but eventually we'll have errors show up in the explore page so they can be queried and viewed side by side.