Application Insights: Logs not streaming when deployed to App Service But streams in local debug environment

[sirsendu-mukherjee]

What I am trying to achieve Hello, I have dotnetcore web Api which is running in Azure App Service server farm. For logging I have used both ILogger and Log4net. I am trying to send the logs of service to application-insights but the logs are not streaming when deployed to Azure but when tested locally it works fine and I can see logs from both the library ILogger/Log4net getting streamed to Application Insights.

What I have tried so far Below configuration I have added in my service for application insights.

• Runtime version: netcoreapp3.1 version-2.31.0.1

• Hosting environment: Azure App Service

app-service.csproj

<ItemGroup>
    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.16.0" />
    <PackageReference Include="Microsoft.ApplicationInsights.Log4NetAppender" Version="2.16.0" />
    ...
<ItemGroup>

log4net.config

<appender name="aiAppender" type="Microsoft.ApplicationInsights.Log4NetAppender.ApplicationInsightsAppender, Microsoft.ApplicationInsights.Log4NetAppender">
  <InstrumentationKey name="AppInsightsKey" value="abcdefgh-abcd-abcd-abcd-abcdefghijkl" />
  <layout type="log4net.Layout.PatternLayout">
    <conversionPattern value="%message%newline"/>
  </layout>
</appender>

startup.cs

 public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
            Logger.ConfigureLog4Net("./logs/app.log", Configuration)
            ...
        }
 public void ConfigureServices(IServiceCollection services)
        {
            // The following line enables Application Insights telemetry collection.
            services.AddApplicationInsightsTelemetry(
                new Microsoft.ApplicationInsights.AspNetCore.Extensions.ApplicationInsightsServiceOptions
                {
                    EnableActiveTelemetryConfigurationSetup = true,
                    InstrumentationKey = Configuration["ApplicationInsights:InstrumentationKey"] 
                }) ;
            services.AddMvc();
            ...
        }

appsettings.json

  "ApplicationInsights": {
    "InstrumentationKey": "xxxxx"
  }

I have also added ApplicationInsights.config file as suggested in elsewhere for Log4net to work as below

<?xml version="1.0" encoding="utf-8"?>
<ApplicationInsights xmlns="http://schemas.microsoft.com/ApplicationInsights/2013/Settings">
	<InstrumentationKey>xxxxx</InstrumentationKey>
</ApplicationInsights>

I have also added the key ApplicationInsights:InstrumentationKey in App Service Configuration App Settings. As well tried with default app settings key APPINSIGHTS_INSTRUMENTATIONKEY.

I have made sure all the required App settings keys are present in App service configuration too.

Followed this MS document on common issues Troubleshooting Application Insights Agent (formerly named Status Monitor v2) and as it quotes below

If any of these DLLs are present in the bin directory, monitoring might fail:

Microsoft.ApplicationInsights.dll Microsoft.AspNet.TelemetryCorrelation.dll System.Diagnostics.DiagnosticSource.dll

But it's really confusing cause the required Nugets like "Microsoft.ApplicationInsights.AspNetCore" has "Microsoft.ApplicationInsights.dll" as dependency which in-turn has "System.Diagnostics.DiagnosticSource.dll" as dependency; and without the required AspNetCore nuget Application Insight won't work.

Essentially, this is a strange issue am facing where-in all logs and exceptions getting streamed perfectly fine in local run but once deployed to Azure App service no logs are streaming at all. Can someone shed some light here? Am I missing something obvious??

[TimothyMothra]

Hi @sirsendu-mukherjee, Thanks for the detailed explanation! There's a lot of information to break down, forgive me if I miss something.

You referenced:

1. AspNetCore application with the 'Microsoft.ApplicationInsights.AspNetCore' SDK.

   This configuration looks correct, (startup.cs and appsettings.json). I wouldn't expect to see the applicationinsights.config with our AspNetCore SDK. You might consider testing without the applicationinsights.config and seeing if it's necessary.

2. Log4Net

   I'm not personally familiar with log4net so I cannot comment on best practices.

   I'm surprised to see someone using both ILogger and Log4Net. FYI, we do have support for ILogger + Application Insights. https://docs.microsoft.com/en-us/azure/azure-monitor/app/ilogger

3. Deploying this to Azure's App Service

   You have two options with App Service; either manually instrument using the SDK or using the Codeless Attach. Note that these are mutually exclusive options. If the Codeless product discovers any SDK dependencies already in your application, it should back off to avoid conflicting with your application. You can learn more about codeless attach here: https://docs.microsoft.com/azure/azure-monitor/app/azure-web-apps-net-core

4. You referenced a doc page for Status Monitor v2.

   Note that this is a separate product intended for on-premise VMs. I think you can ignore this doc page and refer to the link I shared above.

5. Regarding DLL incompatibilities

   These incompatibilities are only relevant if you're trying to use any of our Codeless Attach products. Because you've already instrumented your application using the SDK, I think you can safely ignore this part.

6. Regarding Environment Variables

   You mentioned that you set both ApplicationInsights:InstrumentationKey and APPINSIGHTS_INSTRUMENTATIONKEY I'm not familiar with the first, but if you set the second that would take precedence over any other configuration. Because you've already instrumented your application using the SDK, I would remove both of these to avoid conflicts.

Next Steps

It sounds like you have everything setup in your application correctly, based on your local testing! My best guess right now is that you're having some kind of environmental issue.

My recommendation is to collect the SDK logs, this would help identify what issue our SDK is experiencing. We have instructions to collect logs here: https://github.com/microsoft/ApplicationInsights-dotnet/tree/main/troubleshooting/ETW#self-diagnostics I also noticed that you're on an older version of our SDK (v2.16). If you can upgrade to the latest (v2.20), you can take advantage of the Self-Diagnostics feature which would be easier to collect logs from the serverless environment. IMPORTANT: It's NOT recommended to share those logs here on GitHub because they may contain PII or OII.

If you get blocked on any of these steps, feel free to open an Azure Support Request. https://github.com/microsoft/ApplicationInsights-dotnet#support Our support staff would be better able to review your App Service instance and can help review your SDK logs. You can reference this github issue in your support ticket.

[sirsendu-mukherjee]

Hello @TimothyMothra

Appreciate for your suggestions on this. To respond on the points you have referred (as below)

1. applicationinsights.config this config file is for Log4net library to work. ILogger works perfectly fine always but initially I found Log4net not streaming logs at all neither locally not in Azure. Investigating through this issue and found from others faced this same issue that this file may be required. See Application Insights integration with Log4Net

2. Yes true but we have used Log4Net initially and thus few logs are still lying around in application code. it's a future cleanup task but nit a priority currently. BUT, essentially for both the above points, this same setup is working in my local machine (Windows 10 Enterprise) and App Service Plan is also for Windows machine (Microsoft Windows NT 10.0.14393.0 - From Kudu). Thus I feel, it should work just fine when deployed to Azure App Service as well.

My mistake, it's a typo actually while posting the question. I am actually using the latest SDK version 2.20.0 for all the packages.

As per the referenced document, I have enabled self diagnostic for SDK but strangely I am getting a empty file. Below are the steps have taken ..

1. Create a file named “ApplicationInsightsDiagnostics.json” in the project root folder
2. File content would be as below

    {
        "LogDirectory": ".",
        "FileSize": 1024,
        "LogLevel": "Error"
    }

Ran it locally, as well deployed to Azure App Service and collected the generated logs. Below is the content of the log files. Strangely in either case of Local/Azure run the generated Log is empty file .. pretty much.

Please let me know.

[TimothyMothra]

Regarding the self-diagnostics, change the log level to Warning. You're looking for the internal logs from this class, but specifically the "TransmissionSendingFailed" message. https://github.com/microsoft/ApplicationInsights-dotnet/blob/2f10802f443cb6cdd2cafe8af26ad060eaeee822/BASE/src/ServerTelemetryChannel/Implementation/TelemetryChannelEventSource.cs#L172-L173

Regarding your comment; this same setup is working in my local machine This is typically a clue that it's an environmental issue. Our SDK isn't aware of where it runs and we expect it to "just work" when your application is deployed.

[sirsendu-mukherjee]

Thanks much @TimothyMothra

Yes, I see what you are pointing to now. I have the SDK logs generated and could see the "TransmissionSendingFailed" message in them, specifically like below. Please let me know how can I share those files with you, so that you can probably look into them once.

2022-01-28T21:50:13.6209370Z:TransmissionSendingFailed. TransmissionId: {0}. Message: {1}.{2BFKAROBhpE=}{Type: 'System.Net.Http.HttpRequestException' Message: 'No such host is known. | No such host is known.'}{some namespace} 2022-01-28T21:50:13.6209511Z:BufferEnqueueNoCapacity. Size: {0}. Capacity: {1}.{0}{0}{some namespace} 2022-01-28T21:50:13.6209533Z:BufferEnqueueNoCapacity. Size: {0}. Capacity: {1}.{0}{0}{some namespace}

[TimothyMothra]

"No such host is known" - this may be a DNS related issue in the App Service environment. If that's the case, I won't be able to help you with this.

One last thing you can try, we have a PowerShell script that you can execute within your environment: https://github.com/microsoft/ApplicationInsights-dotnet/tree/main/troubleshooting/Ingestion

This would allow you to test your network connection to our ingestion service. If this doesn't work, you will need to open an Azure Support Request to investigate. https://github.com/microsoft/ApplicationInsights-dotnet#support You can reference this github issue in your support ticket.

[sirsendu-mukherjee]

@TimothyMothra Tim - I have already created a Azure support case for this and had a working session with few members from App insight and App Service Networking team. We found that there is nothing wrong with the network connection to ingestion service and rest of all is fine from my end. To that matter, I am getting the live metrics absolutely fine but Logs/Transaction Search are not getting streamed.

This is one of strange case. I have uploaded all the SDK log files to the file transfer they given. Azure support team collected all the required information and they would reach out to "Product Group" for further investigation (Hopefully probably coming to your plate :) ). I hope, to get a resolution at the earliest as it's bit priority for me.

Thank you.

[sirsendu-mukherjee]

Not sure but I would keep this thread open until the resolution received. Thus can post the resolution here too which would help others in future. Please let me know if that's acceptable.

[arturohernandez10]

I have a similar issue happening in our application. The difference is that I have the application show data in Visual Studio as well as in an Application Insights instance in Azure. About half of them appear in both, and the other half only appear locally. I suspect there is some "hidden" or undocumented extra processing that happens with the messages, which is causing it to get lost. I am still troubleshooting. My reasoning is based on how the dependency name gets processed. I can see there is some paring being made. And for some reason, the Data field is showing up as the Name only locally. I have shown the code to a colleague just to verify I am not missing something. I suspect both the Data and Name get parsed and transformed. And there is certain format that I am not following when setting those two fields. This is a god example of a leaky abstraction. I'll update here when I find the solution, for my case.

[TimothyMothra]

Hi @arturohernandez10, your issue sounds different than this one. I recommend creating a new Issue with some examples so we don't confuse these two.

[arturohernandez10]

I was starting the TelemetryClient from DI in one of the two. In fact it is a different issue. Why did it even show in VS?? But it doesn't matter, It's resolved. Thanks.

[github-actions[bot]]

This issue is stale because it has been open 300 days with no activity. Remove stale label or this will be closed in 7 days. Commenting will instruct the bot to automatically remove the label.

[matzuki-hub]

@arturohernandez10 any update on this?