[QUERY] How To Disable Dependency Tracking Of HTTP 404 Requests

[Yashuaa]

Library name and version

Azure.Monitor.OpenTelemetry.Exporter v1.3.0

Query/Question

For some reason, you all (Microsoft) thought it was a good idea to log all HTTP requests that are made in our applications by default without allowing us to turn it off.

This causes 2 issues,

1. Unnecesary noise
2. You fill up our logs with garbage logging and cause our app insights to be capped which causes us to not see any failures or history of logs until the cap resets the next day

--

1. To reproduce the issue, install this NuGet

2. & produce a bunch of http requests that return 404s. (call some API that gives u a 404 back)

3. & go to app insights and click on Dependency Tracking - you will see them there

We have a lot of times when we hit a lot of partner apis that return 404s. These ARE NOT ERRORS, & we do not want Microsoft logging 4k of these per second. These just happen sometimes... When a 404 happens we catch it in code and suppress it and return a null.

In app insights prior, we could just do DependencyTrackingModuleEnabled = false or some weird thing like that, but you all have took away that property from the new app insights #thxMicrosoft..

So how can we turn off this ridiculous false positive logging?? We would rather not increase cap size and pay Microsoft more money because they took away a feature.

Environment

.NET 6 Azure Functions (latest)

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @cijothomas @rajkumar-rangaraj @reyang @TimothyMothra @vishweshbankwar.

[TimothyMothra]

Hi @Yashuaa ,

In OpenTelemetry, dependencies are collected by specific Instrumentation libraries. To filter or drop specific dependencies based, you'll need to configure that specific Instrumentation library.

Your HTTP dependencies are collected by the OpenTelemetry.Instrumentation.Http library. This library's config has a Filter delegate that will allow you to define what kind of telemetry to drop: https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.Http#filter-httpwebrequest-api

Next Steps

Are you using the Azure.Monitor.OpenTelemetry.AspNetCore package or the Azure.Monitor.OpenTelemetry.Exporter package? How you access the filtering delegate will be slightly different.

• If you're using the Azure.Monitor.OpenTelemetry.Exporter package, somewhere in your config you must have included .AddHttpClientInstrumentation(). You can add your filtering here or remove this call to exclude all HTTP dependencies.

• If you're using the Azure.Monitor.OpenTelemetry.AspNetCore package, we auto configure a few Instrumentation libraries to simplify onboarding. This sounds like the scenario you're describing which is why I asked. https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-dotnet-migrate?tabs=aspnetcore#customizing-httpclienttraceinstrumentationoptions

  Since this project obfuscates the Add Instrumentation method, you can write your filter rules like this:

  builder.Services.Configure<HttpClientTraceInstrumentationOptions>(options =>
  {
      options.FilterHttpRequestMessage = (httpRequestMessage) =>
      {
          /// write your filter rules here
  
          /// or return false exclude all HTTP dependencies.
          /// return false;
      };
  });
  

If you're still having issues, please share your full OpenTelemetry configuration so I can review.

[af4jm]

What about for Azure Functions in isolated process? AddHttpClientInstrumentation() is hidden inside UseAzureMonitor(), which can't be removed because it loads a bunch of useful stuff that's internal sealed so can't be loaded individually...

Related question: is the comment in this code block correct? Spelunking through y'all's source led me to that conclusion

Program.cs, inside HostBuilder.ConfigureServices(services):

        services.AddSingleton<MyResourceDetector>();
        services.ConfigureOpenTelemetryMeterProvider(builder =>
        {
            builder.AddMeter(
                // snip; list of our app's custom meters
            );
        });

        var telemetryBuilder = services.AddOpenTelemetry();
        telemetryBuilder.ConfigureResource(builder =>
            builder.AddDetector(sp => sp.GetRequiredService<MyResourceDetector>()));
        telemetryBuilder.UseFunctionsWorkerDefaults();

        telemetryBuilder.UseAzureMonitor(options =>
        {
            options.EnableLiveMetrics = false;
            options.Diagnostics.IsDistributedTracingEnabled = false;

            // these are specific to HTTP pipeline logging, despite their generic names
            options.Diagnostics.IsLoggingEnabled = false;
            options.Diagnostics.IsLoggingContentEnabled = false;
        });

public class MyResourceDetector : IResourceDetector
{
    private readonly IHostEnvironment hostEnvironment;

    public MyResourceDetector(IHostEnvironment hostEnvironment)
    {
        this.hostEnvironment = hostEnvironment;
    }

    public Resource Detect()
    {
        return ResourceBuilder.CreateEmpty()
            .AddService(serviceName: hostEnvironment.ApplicationName)
            .AddAttributes(new Dictionary<string, object> { ["host.environment"] = hostEnvironment.EnvironmentName })
            .Build();
    }
}

[Yashuaa]

Hi

Thank you for the detailed response. I tried returning false and that seemed to work as in I can't see the DETAILS of the 404 logs in dependency tracing from the app insights Dependency Tracking tab..

BUT querying the logs I still see the 404s there :(... is there any way to NOT store these in our logs?

It fills up our log space and makes us cap out and we DO NOT care about 404s

[TimothyMothra]

HI @sf4jm, please let me know if I misunderstood any of your questions.

Re: Functions

According to your code sample, I see var telemetryBuilder = services.AddOpenTelemetry();. You should be able to add customizations to the same service collection: services.Configure<HttpClientTraceInstrumentationOptions>(options => ...)

I'll add that the Functions team usually does their own customizations in addition to telemetry SDKs. If you're having issues specifically with Functions (ie: doesn't repro in any other app) please report those issues to the Functions team so they can track user feedback.

Re: comment

I'm not sure which comment you're asking about. Are you asking about this?

// these are specific to HTTP pipeline logging, despite their generic names options.Diagnostics.IsLoggingEnabled = false; options.Diagnostics.IsLoggingContentEnabled = false;

These are internal diagnostics generated by the HTTP Pipeline that we use to send a user's telemetry to your Application Insights resource. We explicitly set this off by default and you should be able to ignore this.

[TimothyMothra]

Hi @Yashuaa, I'm not able to reproduce the behavior you're describing. If you can share your full config, or ever better a sample app that demonstrates this behavior I would love to take a look.

Here's what I did:

Test 1: Sample app that filters ALL Http dependencies:

namespace WebApplication14
{
    using Azure.Monitor.OpenTelemetry.AspNetCore;
    using OpenTelemetry.Instrumentation.Http;

    public class Program
    {
        public static void Main(string[] args)
        {
            var builder = WebApplication.CreateBuilder(args);
            builder.Services.AddOpenTelemetry().UseAzureMonitor(x => x.ConnectionString = "...");
            builder.Services.Configure<HttpClientTraceInstrumentationOptions>(options =>
            {
                options.FilterHttpRequestMessage = (httpRequestMessage) => { return false; };
            });

            var app = builder.Build();

            app.MapGet("/", async () =>
            {
                var client = new HttpClient();
                var response = await client.GetAsync("https://httpstat.us/404");

                return "Hello World!";
            });

            app.Run();
        }
    }
}

When I query my logs, I see zero dependencies:

Test 2: filter only dependencies with 404 response

I was mistaken in my earlier response. the HTTP Instrumentation library doesn't provide an option to filter based on the response object. For this you'll need to use a custom FilteringProcessor:

namespace WebApplication14
{
    using System.Diagnostics;
    using Azure.Monitor.OpenTelemetry.AspNetCore;
    using OpenTelemetry;
    using OpenTelemetry.Trace;

    public class Program
    {
        public static void Main(string[] args)
        {
            var builder = WebApplication.CreateBuilder(args);
            builder.Services.AddOpenTelemetry()
                .WithTracing(tracerProviderBuilder =>
                {
                    tracerProviderBuilder.AddProcessor(new MyFilteringProcessor());
                })
                .UseAzureMonitor(x => x.ConnectionString = "...");
            var app = builder.Build();

            app.MapGet("/", async () =>
            {
                var client = new HttpClient();
                var response1 = await client.GetAsync("https://www.bing.com");
                var response2 = await client.GetAsync("https://httpstat.us/404");

                return "Hello World!";
            });

            app.Run();
        }

        private class MyFilteringProcessor : BaseProcessor<Activity>
        {
            public override void OnEnd(Activity activity)
            {
                if (activity.Status == ActivityStatusCode.Error)
                {
                    foreach (var tag in activity.TagObjects)
                    {
                        if (tag.Key == "http.response.status_code" && tag.Value?.ToString() == "404")
                        {
                            // Do not record the activity if the response status code is 404.
                            activity.ActivityTraceFlags &= ~ActivityTraceFlags.Recorded;
                        }
                    }
                }
            }
        }
    }
}

My logs:

[TimothyMothra]

It's been a few weeks without any response, so I'm closing this issue. If you have any follow up questions, please feel free to reopen the issue or open a new issue to continue the conversation.