System.NullReferenceException while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer

[LawrenceSmith1437]

Have just redeployed an Azure App Service application that was previously full-framework. It was retooled to .Net 3.1 Core, including the latest versions of all sdk's and dependencies. Upon deploying the app and watching the live stream metrics in App Insights, I started seeing these null ref trace message pop up. I don't know if this is causing failures for my clients or if this is innocuous.

It appears to be a regression of the issue from the former repository, See: https://github.com/microsoft/ApplicationInsights-aspnetcore/issues/903

Repro Steps

1. Deploy an Asp.Core 3.1 app to an azure app service
2. Observe in the SampleTelemetry of the Live Metrics Stream, an occasional Trace message with content like this:

" Time 3:48:43 PM Message AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer, exception message - System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.AspNetCore.Http.DefaultHttpRequest.get_Cookies() at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.UpdateRequestTelemetryFromPlatformContext(RequestTelemetry requestTelemetry, HttpContext platformContext) at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry) at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry) at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)] AspNetCoreEnvironment Production"

Actual Behavior

See repro, included error trace message

Expected Behavior

No Null Reference errors

Version Info

SDK Version : Microsoft .Net Core SDK 3.1.100 .NET Version : Microsoft .Net Core Core 3.1 How Application was onboarded with SDK(VisualStudio/StatusMonitor/Azure Extension) : OS : None Hosting Info (IIS/Azure WebApps/ etc) : Azure App Services (current as of 1/2/2020)

[TimothyMothra]

I confirmed that this bug was fixed in our SDK 2.10. https://github.com/microsoft/ApplicationInsights-dotnet/blob/a26a43f012ef6afca563b43b16ab019698a3f062/NETCORE/src/Microsoft.ApplicationInsights.AspNetCore/TelemetryInitializers/WebSessionTelemetryInitializer.cs#L60-L64

This will get fixed when Azure App Services updates to our latest SDK.

@Mikhail-msft can you confirm which version of our SDK is in production?

[melborp]

Hi @TimothyMothra ,

I just upgraded from 2.13.1 to latest 2.14 and i can see this error also happening in trace. I got intrigued and checked 3 other environments and i can see it has happened a few times to about 10 times in the last 7 days.

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer, exception message - System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.AspNetCore.Http.DefaultHttpRequest.get_Cookies() at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.UpdateRequestTelemetryFromPlatformContext(RequestTelemetry requestTelemetry, HttpContext platformContext) at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry) at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry) at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

The application is running ASP.NET Core 3.1.6 and in those cases the AI SDK was 2.13.1. We are now looking to upgrade to 2.14.0.

On App Service if i navigate to "d:\Program Files (x86)\SiteExtensions\ApplicationInsightsAgent" there seems to be 2.8.36 version there.

Unsure if this trace error is serious or not, since it only happens a few times.

[TimothyMothra]

Hi @melborp, Thanks for reporting, I'm sorry you're having trouble with this.

I checked the code, here's the method that's throwing: https://github.com/microsoft/ApplicationInsights-dotnet/blob/743848ab2bf9ec465ac8b313a4b6626b60c7315e/NETCORE/src/Microsoft.ApplicationInsights.AspNetCore/TelemetryInitializers/WebSessionTelemetryInitializer.cs#L60-L76

As you can see, we're already performing a null check before accessing the Cookies at line 62. I confirmed this in both v2.13.1 and v2.14. If you're ever able to repro this I would love to get more details.

[Mikhail-msft]

@vishweshbankwar please share the current AI .netcore SDK version for 2.8.36 extension.

[TimothyMothra]

@Mikhail-msft which product is the 2.8.36 extension? is that WAD?

[vishweshbankwar]

@Mikhail-msft @TimothyMothra
2.8.36 extension version uses 2.8.0 SDK for netcore3.1 apps.

[TimothyMothra]

@vishweshbankwar SDK v2.8.0 was released in September 2019. The current version is v2.14.0, released in April 2020.

Are there any plans to upgrade?

[Mikhail-msft]

I believe you wanted the ApplicationInsights package version. it is 2.11:

    <PackageReference Include="Microsoft.ApplicationInsights.AspNetCore" Version="2.8.0" GeneratePathProperty="true" />
    <PackageReference Include="Microsoft.ApplicationInsights" Version="2.11.0" GeneratePathProperty="true" />

We do upgrade SDK versions regularly based on stability details

[TimothyMothra]

@Mikhail-msft The versions numbers are in sync now. NetCore 2.8 ~ AI SDK 2.11 NetCore 2.14 ~ AI SDK 2.14

[kimpenhaus]

@TimothyMothra we are facing this kind of exception as well, like a 4.4k times in the last 30 days on our production system:

I don't think that the null check will help here at UpdateRequestTelemetryFromPlatformContext The HttpDefaultRequest Cookies is kind of lazy property as it is using the underlying feature from inside the DefaultHttpContext:

https://github.com/dotnet/aspnetcore/blob/8efcca43ceaaf72281e17f25d759844f488914ac/src/Http/Http/src/Internal/DefaultHttpRequest.cs#L143

https://github.com/dotnet/aspnetcore/blob/8efcca43ceaaf72281e17f25d759844f488914ac/src/Http/Http/src/Internal/DefaultHttpRequest.cs#L63

and I believe this is why the NRE is occuring inside the get_Cookies and not the method UpdateRequestTelemetryFromPlatformContext itself. This might be the case if the DefaultHttpRequest is already uninitialized because all feature are set to default/null.

https://github.com/dotnet/aspnetcore/blob/8efcca43ceaaf72281e17f25d759844f488914ac/src/Http/Http/src/Internal/DefaultHttpRequest.cs#L49

Another exception occuring in this context on our production system is the IFeatureCollection has been disposed ObjectDisposedException

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer, exception message - System.ObjectDisposedException: IFeatureCollection has been disposed.
Object name: 'Collection'.
   at Microsoft.AspNetCore.Http.Features.FeatureReferences`1.ThrowContextDisposed()
   at Microsoft.AspNetCore.Http.DefaultHttpRequest.get_Cookies()
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.UpdateRequestTelemetryFromPlatformContext(RequestTelemetry requestTelemetry, HttpContext platformContext)
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry)
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry)
   at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

based on the discussion on an issue inside aspnetcore and the comment of @davidfowl https://github.com/dotnet/aspnetcore/issues/11069#issuecomment-501761731 I think this might be an multi-threading issue as well. (bullets 2-4) I think the DefaultHttpRequest.get_Cookies() accesses are performed in a time where either the Unitialize() method is already called on the main thread (case NullReferenceException) or the underlying middleware has already disposed its FeatureCollection (case ObjectDisposedException) as the request has ended on the main thread.

[melborp]

I can see that this is still happening but 2 types.

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer, exception message - System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.AspNetCore.Http.Features.FeatureReferences`1.UpdateCached[TFeature,TState](TFeature& cached, TState state, Func`2 factory, Int32 revision, Boolean flush)
   at Microsoft.AspNetCore.Http.DefaultHttpRequest.get_Cookies()
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.UpdateRequestTelemetryFromPlatformContext(RequestTelemetry requestTelemetry, HttpContext platformContext)
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry)
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry)
   at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

Second (as reported by @kimpenhaus)

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer, exception message - System.ObjectDisposedException: IFeatureCollection has been disposed.
Object name: 'Collection'.
   at Microsoft.AspNetCore.Http.Features.FeatureReferences`1.ThrowContextDisposed()
   at Microsoft.AspNetCore.Http.DefaultHttpRequest.get_Cookies()
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.UpdateRequestTelemetryFromPlatformContext(RequestTelemetry requestTelemetry, HttpContext platformContext)
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry)
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry)
   at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

[cwe1ss]

We are also experiencing this issue. We're on sdkVersion "dotnetc:2.15.0-44797".

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer, exception message - System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.AspNetCore.Http.DefaultHttpRequest.get_Cookies()
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.UpdateRequestTelemetryFromPlatformContext(RequestTelemetry requestTelemetry, HttpContext platformContext)
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry)
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry)
   at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

[cijothomas]

@TimothyMothra Please take a look at this when you get a chance. This seem to be an issue even in latest release, so we need to do a fix.

[LeanderVanEck]

Is there any update on this? It seems we are having this same issue on "dotnetc:2.16.0-18277"

[github-actions[bot]]

This issue is stale because it has been open 300 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[kimpenhaus]

asking for updates so github-actions is not closing this issue

[kwaazaar]

Seeing this now:

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer, exception message - System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.AspNetCore.Http.DefaultHttpRequest.get_Cookies()
at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.UpdateRequestTelemetryFromPlatformContext(RequestTelemetry requestTelemetry, HttpContext platformContext)
at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry)
at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry)
at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

The field sdkVersion contains dotnetc:2.20.0-103. The runtime is .net 6 and running in a Linux container.

[okolvik-avento]

I'm also seeing this error:

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer, exception message System.NullReferenceException: Object reference not set to an instance of an object. at
Microsoft.AspNetCore.Http.DefaultHttpRequest.get_Cookies()
at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.UpdateRequestTelemetryFromPlatformContext(RequestTelemetry requestTelemetry, HttpContext platformContext)
at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry)
at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry) at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

.NET 6.0.5, running on Windows Server 2019, IIS 10. Application Insights 2.20.0

[davidfowl]

Can the users of this confirm if they are using SQL Server when seeing this issue?

[kwaazaar]

Yes we do indeed. Through Entity Framework CV ore and 6.4.

Op za 4 jun. 2022 18:27 schreef David Fowler @.***>:

Can the users of this confirm if they are using SQL Server when seeing this issue?

— Reply to this email directly, view it on GitHub https://github.com/microsoft/ApplicationInsights-dotnet/issues/1524#issuecomment-1146645091, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACEGA47ZAT6I47KXNJ577QTVNN7YTANCNFSM4KCH4XAQ . You are receiving this because you commented.Message ID: @.***>

[davidfowl]

This PR https://github.com/dotnet/SqlClient/pull/1637 will fix a source of these issues. I want to make sure we can find 90% of these issues that happen with ASP.NET Core.

@okolvik-avento Helped here by providing dumps that helped me identify this problem pretty quickly. I'm hoping he can share his experience with others on this thread to help do the same so we can pin point the other problematic patterns/libraries that cause this to happen.

[okolvik-avento]

@davidfowl made a quick and dirty guide https://okolvik-avento.github.io/pages/2022-06-05-creating-iis-userdumps

[hasandogu]

We are also seeing this on a .net6 application on a Linux app service. Any insights would be appreciated

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.ClientIpHeaderTelemetryInitializer, exception message - System.ObjectDisposedException: Request has finished and HttpContext disposed.
Object name: 'HttpContext'.
at Microsoft.AspNetCore.Http.DefaultHttpContext.ThrowContextDisposed()
at Microsoft.AspNetCore.Http.DefaultHttpContext.get_Features()
at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry)
at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

[Kev8144]

Experiencing the same using .net 6 on windows OS using IIS.

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer, exception message - System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.UpdateRequestTelemetryFromPlatformContext(RequestTelemetry requestTelemetry, HttpContext platformContext)
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry)
   at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry)
   at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

[mrmstewart]

Experiancing this exception on our prod app service but not dev or staging even though its the same code, same nugets and same config as far as we can tell.

App Insights version 2.2.1

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer, exception message - System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.AspNetCore.Http.DefaultHttpRequest.get_Cookies() at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.UpdateRequestTelemetryFromPlatformContext(RequestTelemetry requestTelemetry, HttpContext platformContext) at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer.OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry) at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry) at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

AppVersion 6.0.12

SDKVersion dotnetc:2.21.0-429

[hellomrsun]

we have the following errors on prod

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.ClientIpHeaderTelemetryInitializer, exception message - System.ObjectDisposedException: Request has finished and HttpContext disposed. Object name: 'HttpContext'. at Microsoft.AspNetCore.Http.DefaultHttpContext.ThrowContextDisposed() at Microsoft.AspNetCore.Http.DefaultHttpContext.get_Features() at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry) at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.SyntheticTelemetryInitializer, exception message - System.ObjectDisposedException: IFeatureCollection has been disposed. Object name: 'Collection'. at Microsoft.AspNetCore.Http.Features.FeatureReferences`1.ThrowContextDisposed() at Microsoft.AspNetCore.Http.DefaultHttpRequest.get_Headers() at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.SyntheticTelemetryInitializer.OnInitializeTelemetry(HttpContext platformContext, RequestTelemetry requestTelemetry, ITelemetry telemetry) at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry) at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

AI (Internal): [Microsoft-ApplicationInsights-Core] [msg=Log Error];[msg=Exception while initializing Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.WebSessionTelemetryInitializer, exception message - System.ObjectDisposedException: Request has finished and HttpContext disposed. Object name: 'HttpContext'. at Microsoft.AspNetCore.Http.DefaultHttpContext.ThrowContextDisposed() at Microsoft.AspNetCore.Http.DefaultHttpContext.get_Features() at Microsoft.ApplicationInsights.AspNetCore.TelemetryInitializers.TelemetryInitializerBase.Initialize(ITelemetry telemetry) at Microsoft.ApplicationInsights.TelemetryClient.Initialize(ITelemetry telemetry)]

solution target framework: net6.0 .net sdk: 6.0.302 Microsoft.Extensions.Logging.ApplicationInsights: 2.20.0

[rgrace-puck]

We are still seeing this in .NET 8 (Isolated) function apps. All packages are up-to-date.