[FEATURE REQ] Custom Events in Application Insights using azure-monitor-opentelemetry

[juanmanuelug]

Library name and version

Azure.Monitor.OpenTelemetry.Exporter

Query/Question

I am implementing an SDK capable of registering telemetries in external services and as I want it to be vendor-neutral, I am using OpenTelemetry. One of the exporters that I am using is the Azure Monitor Exporter to send the telemetries to Application Insight, but all the telemetries(Span, metrics, logs) from OpenTelemetry are being registered as traces type or dependency type and I would like to be able to register telemetries as Custom Events(from Application Insights).

I saw this extension for Python but wasn't able to find a similar solution for .Net https://github.com/Azure/azure-sdk-for-python/issues/33472

So, I would like to know if as in Python, .net has or not a way to send Custom Events to Application Insights by using OpenTelemetry Azure Sdk

Environment

.Net Standard 2.0 Visual Studio 17.8

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist. //cc: @cijothomas @reyang @rajkumar-rangaraj @TimothyMothra @vishweshbankwar

[TimothyMothra]

Hello @juanmanuelug, I'm sorry to say that as of today, none of our Azure Monitor SDKs provide CustomEvents. We are strictly adhering to the OpenTelemetry specification which doesn't provide a mechanism to do this yet. There is a working group exploring this and we're providing input.

For the near term each language has to provide a workaround, described here: https://learn.microsoft.com/azure/azure-monitor/app/opentelemetry-add-modify#send-custom-telemetry-using-the-application-insights-classic-api

For .NET, we're asking you to use the classic Application Insights sdk's TelemetryClient.TrackEvent.

cc @mattmccleary

[juanmanuelug]

Hi again Thank you for the information! :D

[JarrodJ83]

OTEL does have a spec for Span Events. I was hoping these would be translated into "custom events" in App Insights but it appears the current SDK ignores these completely. Can you comment on what the team looking into this is thinking as far as if/how they'll handle "span events"? Will they be treated as "custom events" or will these be completely different things? If they are different will anything be done to deliver "span events" anywhere or will they continue to be ignore? Just curious because my short term idea was to just write my own exporter to export the span events into custom events using the class app insights sdk and was curious how future proof that solution might be.

[TimothyMothra]

Hi @JarrodJ83, there's some confusing vocabulary here so I want to first set some context...

• OTel Span == .NET Activity
• OTel Span Event == .NET Activity Event
• OTel Log == Application Insights "Trace"

Responding to your points...

1. We made the decision to map Span Events to Logs. From the link you shared:

   A Span Event can be thought of as a structured log message https://opentelemetry.io/docs/concepts/signals/traces/#span-events

   My understanding is that the concept of a "custom event" breaks the model of the 3 pillars of observability (Traces, Metrics, Logs). There was a working group exploring how to solve this. I haven't heard any recent updates from the working group. I'll follow up.

2. Your comment:

   it appears the current SDK ignores these completely.

   This should not be the case. Activity Events should be mapped to either Exceptions or Logs, unless you're encountering an exception at runtime. Can you please double check if these were ingested, they would be in Application Insights's "Traces" table. https://github.com/Azure/azure-sdk-for-net/blob/5b3a4040f9bf2039067ab69ebcd9141ecdbd3a93/sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/TraceHelper.cs#L229-L261

   If you need to collect internal logs, you can use OpenTelemetry's Self-Diagnostics feature: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter#troubleshooting

[JarrodJ83]

Thanks for the clarifications! This makes sense. Still learning all the OTEL concepts and how they are used so I appreciate the info.

On Thu, Apr 4, 2024, 4:17 PM Timothy Mothra @.***> wrote:

Hi @JarrodJ83 https://github.com/JarrodJ83, there's some confusing vocabulary here so I want to first set some context...

• OTel Span == .NET Activity
• OTel Span Event == .NET Activity Event
• OTel Log == Application Insights "Trace"

Responding to your points...

We made the decision to map Span Events to Logs. From the link you shared:

A Span Event can be thought of as a structured log message https://opentelemetry.io/docs/concepts/signals/traces/#span-events

My understanding is that the concept of a "custom event" breaks the model of the 3 pillars of observability (Traces, Metrics, Logs). There was a working group exploring how to solve this. I haven't heard any recent updates from the working group. I'll follow up. 2.

Your comment:

it appears the current SDK ignores these completely.

This should not be the case. Activity Events should be mapped to either Exceptions or Logs, unless you're encountering an exception at runtime. Can you please double check if these were ingested, they would be in Application Insights's "Traces" table.

https://github.com/Azure/azure-sdk-for-net/blob/5b3a4040f9bf2039067ab69ebcd9141ecdbd3a93/sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter/src/Internals/TraceHelper.cs#L229-L261

If you need to collect internal logs, you can use OpenTelemetry's Self-Diagnostics feature: https://github.com/Azure/azure-sdk-for-net/tree/main/sdk/monitor/Azure.Monitor.OpenTelemetry.Exporter#troubleshooting

— Reply to this email directly, view it on GitHub https://github.com/Azure/azure-sdk-for-net/issues/42157#issuecomment-2038131606, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB4SZSCU7EI3UHGAO7CLHLY3WYL7AVCNFSM6AAAAABDWLSUMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZYGEZTCNRQGY . You are receiving this because you were mentioned.Message ID: @.***>

[rossgrambo]

Hello @TimothyMothra ,

There was a working group exploring how to solve this. I haven't heard any recent updates from the working group. I'll follow up.

Have you heard any updates / made any plans to support Custom Events?

I'm from another Azure team working on a feature in our SDK. We have a preview sdk out today using a custom App Insights publisher but are trying to move to OpenTelemetry as much as possible. For our use case we require Custom Events in App Insights/Monitor as the result.

We are currently planning to build our own OpenTelemetry exporter that sends Custom Events using App Insights sdk. The exporter will be a project/class within our SDK. This isn't a bad state- but it will be wasted work if y'all plan to support Custom Events anytime soon.

We made the decision to map Span Events to Logs

My understanding of Events in OpenTelemetry is that they're committing to a separate API for emitting them- but the structure & exporters will continue to be built off of Logs. The schema is essentially a log with the Attribute "event.name". I'm curious if the existence of this Attribute would be enough to push it to Custom Events for y'all. It's likely what we will implement in our exporter.

[johncrim]

Late response to:

We made the decision to map Span Events to Logs. From the link you shared:

A Span Event can be thought of as a structured log message https://opentelemetry.io/docs/concepts/signals/traces/#span-events

My understanding is that the concept of a "custom event" breaks the model of the 3 pillars of observability (Traces, Metrics, Logs). There was a working group exploring how to solve this. I haven't heard any recent updates from the working group. I'll follow up.

At the risk of blowing this up into another topic (happy to take it elsewhere if someone can point me to the place), I don't think this logic works. For a couple reasons:

1. Where is it established that there are only 3 pillars of observability? I believe other constructs are under consideration, it's just that these 3 are well-defined and broadly used after many iterations from many experienced parties, and they represent an initial model for instrumenting and understanding distributed systems. But it's more or less acknowledged that these aren't sufficient for all problem areas - eg web apps. I haven't seen anyone claim that there are no more than 3 models needed for telemetry.
2. Events and Exceptions (as a thing that can be attached to an event) were added to spans, b/c it makes sense to attach such info to the span to understand what happened.
3. The App Insights schemas pre-date OpenTelemetry, and they work decently well for a large swath of problems. The Otel models are better for the domains they have solved, but the app insights schemas handle a larger set of app instrumentation problems. IMO the right way to map them together are to consider "what information do they represent" and iterate from there given the current starting point.

Some examples of useful schema mismatches between Otel and AppInsights: Exceptions are separate from log messages and events - that makes it easier to query for Exceptions. But they're not a separate pillar in Otel. PageViews and AvailabilityResults are both quite useful, and they don't have an analog in the current OpenTelemetry model. Requests and Dependencies are pretty distinct in the real world - I think there's real value in having them separate, even though they're both "Spans" in OpenTelemetry.

(I also would like a CustomSpan in the App Insights schema - there's a need for Span telemetry that is not a request and not a dependency).

Back to the issue at hand - IMO Span Events should be CustomEvents in the AppInsights model. Otel Span Events don't have a severity level or a category or a structured message. They are very simple events, with just the name/id and some tags, and a timestamp. I think that maps cleanly to customEvent, and NOT to traces.

[TimothyMothra]

Hi @johncrim,

• Regarding the "3 pillars" This was one of the original principals when OpenTelemetry was founded. This terminology can be found in several older blogs. It's since been removed from OpenTelemetry's documentation, which I think is an acknowledgement of your points that there are several of other meaningful types of telemetry. But these early decisions are still present in the OpenTelemetry API. OpenTelemetry only has three types of exporters (Traces, Metrics, Logs).

• Regarding Activity Events The Azure Monitor team made the decision to map Span/Activity Events to Logs (aka "Traces" in the Application Insights schema). This has already shipped in our stable SDK and there are no plans to change this.

• (I also would like a CustomSpan in the App Insights schema - there's a need for Span telemetry that is not a request and not a dependency).

  Please open a new issue to discuss this and I'll forward it to our PMs. I'd like to learn more about your needs here.

• Regarding CustomEvents To set expectations, this won't be delivered by our team in the near future. :( We're waiting for this community spec to be stabilized: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/event-api.md The Azure Monitor team is still monitoring this and it's at the very top of our backlog.
  Please continue upvoting this Feature Request, it is seen at all levels!