ECONNRESET - Does not gracefully throw error, but crashes web app

[PJBCoetzer]

ECONNRESET - Does not gracefully throw error, but crashes web app

We have a NodeJS app running as a Azure Web App Service on a Linux based App Service Plan. (configured to be running as always on).

Setup:

• NodeJS 16
• App Service Plan (Linux)
• Redis (Azure managed hosted service)
• Application Insights (Azure managed hosted service)

Packages:

• express 4.17.2
• dotenv 14.2.0
• redis 4.0.2
• applicationinsights 2.2.0

The web service does basic data calculations returning a result as a REST API service. Redis is used to store previously calculated results.

Application Insights has been enabled on the App Service level in the portal. For additional fault monitoring, we added the NPM Package applicationinsights version 2.2.0 in code.

Application insights is configured at start up of the app using:

const appInsights = require("applicationinsights");
appInsights.setup(process.env.APPLICATIONINSIGHTS_CONNECTION_STRING)
appInsights.start()

The app service runs for some time but then crashes unexpectedly with the following in the KUDU logs:

2022-01-20T00:41:19.028838008Z events.js:377
2022-01-20T00:41:19.029056811Z       throw er; // Unhandled 'error' event
2022-01-20T00:41:19.029073211Z       ^
2022-01-20T00:41:19.029079111Z 
2022-01-20T00:41:19.029084211Z SocketClosedUnexpectedlyError: Socket closed unexpectedly
2022-01-20T00:41:19.029089512Z     at TLSSocket.<anonymous> (/home/site/wwwroot/node_modules/@node-redis/client/dist/lib/client/socket.js:184:118)
2022-01-20T00:41:19.029095412Z     at Object.onceWrapper (events.js:520:26)
2022-01-20T00:41:19.029100512Z     at TLSSocket.emit (events.js:412:35)
2022-01-20T00:41:19.029105412Z     at net.js:675:12
2022-01-20T00:41:19.029110212Z     at TCP.done (_tls_wrap.js:563:7)
2022-01-20T00:41:19.029115112Z Emitted 'error' event on Commander instance at:
2022-01-20T00:41:19.029128012Z     at RedisSocket.<anonymous> (/home/site/wwwroot/node_modules/@node-redis/client/dist/lib/client/index.js:338:14)
2022-01-20T00:41:19.029149012Z     at RedisSocket.emit (events.js:400:28)
2022-01-20T00:41:19.029154512Z     at RedisSocket._RedisSocket_onSocketError (/home/site/wwwroot/node_modules/@node-redis/client/dist/lib/client/socket.js:207:10)
2022-01-20T00:41:19.029159212Z     at TLSSocket.<anonymous> (/home/site/wwwroot/node_modules/@node-redis/client/dist/lib/client/socket.js:184:107)
2022-01-20T00:41:19.029164013Z     at Object.onceWrapper (events.js:520:26)
2022-01-20T00:41:19.029168413Z     [... lines matching original stack trace ...]
2022-01-20T00:41:19.029172813Z     at TCP.done (_tls_wrap.js:563:7)

I then removed the use of Redis to test the scenario without a external connection, but after some time running, the application still crashes without triggering try/catch code.

I was able to trace the following debug information:

arg0:OperationalError {cause: Error: read ECONNRESET
    at TCP.onStreamRead…nternal/stream_base_commons:220:20)
    at TC…, isOperational: true, errno: -4077, code: 'ECONNRESET', syscall: 'read', …}
cause:Error: read ECONNRESET\n    at TCP.onStreamRead (node:internal/stream_base_commons:220:20)\n    at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {errno: -4077, code: 'ECONNRESET', syscall: 'read', stack: 'Error: read ECONNRESET\n    at TCP.onStreamRea…Trampoline (node:internal/async_hooks:130:17)', message: 'read ECONNRESET'}
code:'ECONNRESET'
errno:-4077
isOperational:true
syscall:'read'
message:'read ECONNRESET'
name:'Error'
stack:'Error: read ECONNRESET\n    at TCP.onStreamRead (node:internal/stream_base_commons:220:20)\n    at TCP.callbackTrampoline (node:internal/async_hooks:130:17)'

My local debug console points me to the file: /node_modules\diagnostic-channel-publishers\dist\src\console.pub.js:43:39 which as I understand is used to log console log events to Application Insights*.

I then removed Application Insights and the Web App has been running stable without any crashes. I re-enabled the use of Redis and no issues traced thus far. This points me to the issue being Application Insights not being able to gracefully handle a break in TCP Socket connection to the Application Insights Service.

Any way to confirm this or prevent the app to crash?

[hectorhdzg]

@PJBCoetzer this could be prevented disabling console patching using environment variable APPLICATION_INSIGHTS_NO_PATCH_MODULES more info here, if you have the actual error we can fix this on node-diagnostic-channel package, not seeing how the code could throw an error so any extra info would be really useful, we are already have exception handling in problematic areas

[hectorhdzg]

@PJBCoetzer is this still an issue for you?, the SDK should not crash under any circumstances, can you try to update to latest?

[JacksonWeber]

Closing as there's no feedback here. Please test again with the new version of the SDK as we've upgrade to OpenTelemetry under the hood. Thank you.