Severe performance issues with applicationinsights 2.1.9

[guy-microsoft]

trafficstars

While we tried to migrate from applicationinsights 1.8.10 to applicationinsights 2.1.9, we noticed some severe performance issues regarding both CPU usage and RAM usage. We noticed the issues when we did testing to compare the performance of the 2 versions. **The only difference in the test setup is the version of applicationinsights package. ** The observed issues:

1. Performance of our entire application decreases at a rate of ~50% (CPU usage 50% increase). Although in version 2.1.9 more dependencies are tracked, the CPU usage increase is not reasonable and cannot be explained merely by the increase in dependencies tracking. In addition, RAM usage has increased in about 200MB.
2. When we do several identical tests one after another on the same NodeJS application, with 5 minutes breaks between them, we see that the later tests perform worse than the previous tests (the later the test, the worse the application performs). It feels as if some resources are not freed despite the breaks between the tests.

Are those issues known and going to be fixed soon? They are very noticeable when compared side by side with version 1.8.10, and they prevent us from migrating from 1.8.10 to 2.X.X.

[hectorhdzg]

Thanks for reporting this @guy-microsoft this is definitely really interesting, we are not aware of any performance issues with latest version of the SDK, is there any reproducible environment you can share?, more info about the implementation of the SDK and other external dependencies would be useful as well for us to investigate further.

[guy-microsoft]

Hi @hectorhdzg , we encountered this issue with NodeJS versions 14.18.2 and 16.13.1 in a K8S cluster. My application is performance driven, handling a few dozens of requests every second. Each request results in making several requests to our dependencies from the application. We use the Application insights SDK for tracking dependencies and for logging traces.

[hectorhdzg]

@guy-microsoft we fixed some issues with memory leaks when live metrics is enabled, and we did a lot of perf profiling to understand impact in normal circumstances, we found a few perf impacted areas, like regex and JSON parsing as expected but these will never cause a 50% CPU increase, please give it a try using newer version of the SDK, also if you enable live metrics you could see performance data live, just be sure Application Insights can reach the internet so data is not consistently being retried and written in disk because of retry logic.

[guy-microsoft]

Thanks @hectorhdzg , I will test my scenarios with the newest version of the SDK.

[nonoxp]

Hi @guy-microsoft , Did you have a chance to test it with the newest version yet? I think our team is in the same situation, at least we share your concerns about the 2.X.X versions. After fiddling with the package versions and the telemetry collection settings, I also suspect something wrong happening with the dependency tracking, leading to nasty CPU increases and impact on the service eventually. (applicationinsights 2.1.4 / nodejs 14.17.4 on top of an Azure App Service)

[guy-microsoft]

Hi @nonoxp I made some quick testing and it seemed to me that the issues are not resolved, but I would like to test it more deeply to make sure they indeed haven't.

[hectorhdzg]

@nonoxp, @guy-microsoft let us know how it goes, this is not a wide issue affecting everyone, we tested using Azure Storage, Azure Datatable SDKs and Redis calls, and performance was fine, if you can do some perf profiling and share the results we may understand better what exactly is causing the problem on your side.

[guy-microsoft]

@hectorhdzg I will try to make CPU profiling later this month and I will share the results with you.

[cjwilsontech]

We had this issue running 2.1.8 in GKE. Downgrading to 1.8.10 as mentioned in the first post improved the issue significantly. Our containers were running out of memory within a day beforehand.

[guy-microsoft]

In version 2.2.2 the situation seems to be better.

[PhantomRay]

I'm not sure if this is related. I recently upgrade my project from 1.x to 2.3.5. I see these lines

MaxListenerExceededWarning: Possible EventEmitter memory leak detected. 11 unpipe listeners added to [AzureApplicationInsightLogger]
MaxListenerExceededWarning: Possible EventEmitter memory leak detected. 11 error listeners added to [AzureApplicationInsightLogger]
MaxListenerExceededWarning: Possible EventEmitter memory leak detected. 11 drain listeners added to [AzureApplicationInsightLogger]
MaxListenerExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [AzureApplicationInsightLogger]

[hectorhdzg]

@PhantomRay can you double check that class "AzureApplicationInsightLogger"? I don't believe that one is coming from this SDK