Data is being sampled even though sampling is disabled

[Stardox]

Environment

• AI SDK: Microsoft.ApplicationInsights.AspNetCore 2.13.1
• .NET version: .NET Core 3.1
• Type: ASP.NET Core web application
• The application is hosted within Azure as an App Service

The problem

Occasionally when reviewing App Insights logs of my Web App through Azure portal I notice that some of the entries within given telemetry operation are missing or that the whole telemetry operation is missing completely. By telemetry operation I mean all entries in the context of incoming HTTP request.

This behavior seems to be related to following message bubble which appears on top of App Insights window in Azure:

Data received from your application is being sampled to reduce the volume of telemetry data retained; only sampled documents will be returned. The sampling may be applied by the Application Insights SDK or on ingestion by Application Insights. Click to learn more.

The message points to following documentation which I have walked through already and disabled the sampling both in the SDK and also in portal.

SDK configuration

This is how I have configured the AI Telemetry Client within my Startup.cs class:

var aiOptions = new ApplicationInsightsServiceOptions
{
    EnableAdaptiveSampling = false,
    EnableQuickPulseMetricStream = false,
    InstrumentationKey = ikGoesHere
};
aiOptions.RequestCollectionOptions.TrackExceptions = true;
services.AddApplicationInsightsTelemetry(aiOptions);

AI configuration through portal

I need to retain all telemetry data (even though it is only for couple of days) for diagnostic purposes. How can I achieve that?

[rajkumar-rangaraj]

@Stardox , SDK and Server configuration looks fine. Could you please execute the following query and check if data is sampled.

union requests,dependencies,pageViews,browserTimings,exceptions,traces
| where timestamp > ago(1d)
| summarize RetainedPercentage = 100/avg(itemCount) by bin(timestamp, 1h), itemType

Ref: [Knowing whether sampling is in operation] (https://docs.microsoft.com/en-us/azure/azure-monitor/app/sampling#knowing-whether-sampling-is-in-operation)

[Stardox]

@rajkumar-rangaraj, for majority of results the RetainedPercentage is 100. However I also see entries where the percentage is ~91, ~68 or ~75 just to name few.

Not sure if this is relevant, but my App Insights resource is in South Central US region.

[rajkumar-rangaraj]

@Stardox , If RetainedPercentage is not equal to 100 in any of the row, then we consider data as sampled. This could also happen if multiple app logging to the same AI component. Could you please validate that only one app is logging data to this component.

[graham-saunders]

I'm having this same issue. Data sampling is set to 100% (all data) and with the above query I'm seeing retained percentage at ~53, ~48, ~60, etc.

[Stardox]

@rajkumar-rangaraj, there is only one application creating log entries in my AI component.

But even if there were multiple, then theoretically speaking, it would only affect this retained percentage report - It should not simply drop / ignore telemetry entries, right?

[Huss-Dev]

@rajkumar-rangaraj/@cijothomas , Getting the same issue as others here.

Have ensured that only one application is creating log entries in AppInsights component, but data keeps getting sampled. Any ideas on how to get all the data retained ?

[Huss-Dev]

@Stardox Did you by any chance find a solution to this issue. Would help if you could share your experience.

Thanks in advance

[DiscipleDV]

@Stardox @Huss-Dev @rajkumar-rangaraj Try disabling this setting. As I understand it, this can help as it disables the SDK on the Azure service side, which overrides the SDK on the application side (with its settings).

[chertpong]

We are facing the same issue, it would be great if anyone can share about the root cause of issue.

[cijothomas]

If you believe data is being sampled at the ingestion side, despite turning it off, please open support ticket with Azure.

If the SDK is sampling, despite configuring it to do otherwise, please reply here, and we can investigate. Please share a minimal repro for this behaviour.

[jsoumelidis]

Hello, we're also facing the same issue here regarding an azure function deployed. The percentage reported by the above query is 100%, though a lot of traces logged by our function are missing. Sampling is disabled in host.json and relevant sampling setting in portal is at 100%.

Any info/help is appreciated

[SavaNDragos]

Seeing same issue with Azure Functions v3, I have in the host.json. { "version": "2.0", "logging": { "fileLoggingMode": "always", "logLevel": { "default": "Trace" }, "applicationInsights": { "samplingSettings": { "isEnabled": false, "maxTelemetryItemsPerSecond": 20000 } } } } the telemetryclient is based on default: new TelemetryClient(TelemetryConfiguration.CreateDefault());

[cijothomas]

Please raise Azure functions issue in Azure functions runtime repo, as Functions uses a custom enablement of application insights sdk.

[github-actions[bot]]

This issue is stale because it has been open 300 days with no activity. Remove stale label or this will be closed in 7 days. Commenting will instruct the bot to automatically remove the label.

[Huss-Dev]

@Stardox @Huss-Dev @rajkumar-rangaraj Try disabling this setting. As I understand it, this can help as it disables the SDK on the Azure service side, which overrides the SDK on the application side (with its settings).

This solution worked in my case. Once the InterOP with Application Insights was disabled the correct settings got applied.

[andywarrens]

We also face this issue:

• in Azure AppInsights: Sampling: All Data 100%
• in code: no AppInsights code is added (ASP.NET Core 3.1)
• in App Service > App Insights blade: Interop is Off

still for lots of requests the itemCount = 1000, or 2000. Rounded off to this value, very strange.

[cijothomas]

We also face this issue:

• in Azure AppInsights: Sampling: All Data 100%
• in code: no AppInsights code is added (ASP.NET Core 3.1)
• in App Service > App Insights blade: Interop is Off

still for lots of requests the itemCount = 1000, or 2000. Rounded off to this value, very strange.

If you haven't added ApplicationInsights code in your application, but you have enabled ApplicationInsights from AppService portal (https://docs.microsoft.com/en-us/azure/azure-monitor/app/azure-web-apps-net-core?tabs=Windows%2Cwindows) then it will have sampling. This is independent of the Ingestion sampling.

[andywarrens]

@cijothomas but what kind of sampling is enabled? How does it work?

I have a feeling that for a particular request, the itemCount is way too high. I expect it to be called around 20-50 times a day. But in reality it's 10.000 (total records is 13, sum(itemCount) ~ 10.000)

[cijothomas]

https://docs.microsoft.com/en-us/azure/azure-monitor/app/sampling

[andywarrens]

Thanks! I've read that page before but I'm still not sure which sampling I have enabled. Because Adaptive and Fixed-rate are configured in code, it seems I have Ingestion Sampling enabled. But then the docs say:

It discards some of the telemetry that arrives from your app, at a sampling rate that you set.

But the only place where I can set it, it's set to All data 100%. So I wonder if my deduction is correct. These are the only settings I could find in my App Service related to Application Insights:

[cijothomas]

not sure which sampling I have enabled.

You haven't enabled anything explicitly. When you enable auto-instrumentation (from app service's Application Insights blade - https://docs.microsoft.com/en-us/azure/azure-monitor/app/azure-web-apps), it enables Application Insights with a default setting. (it includes sampling as well). There is not much control one has if you chose auto-instrumentation. If you want to have full control, then you need to use the SDK based instrumentation.

[andywarrens]

Aha. Thanks for that explanation. It would be great if the docs would add an overview of those default settings.

[cijothomas]

Aha. Thanks for that explanation. It would be great if the docs would add an overview of those default settings.

Please share the feedback in the docs page itself, so it won't get lost in this repo.

[andywarrens]

Do you mean a pull request on this repo (update this file), or the Github button on the bottom of the Sampling page you shared?

[cijothomas]

This ^, at the bottom of the doc

[github-actions[bot]]

This issue is stale because it has been open 300 days with no activity. Remove stale label or this will be closed in 7 days. Commenting will instruct the bot to automatically remove the label.