aws-sdk-js [AWS-Lambda] - NetworkingError: Client network socket disconnected before secure TLS connection was established

We are having problems with several of our services to access DynamoDb. This problem is intermittent, but occurs randomly several times a day at each of the services on (AWS LAMBDA).

Any suggestion?

{
    "errorType": "Runtime.UnhandledPromiseRejection",
    "errorMessage": "NetworkingError: Client network socket disconnected before secure TLS connection was established",
    "reason": {
        "errorType": "NetworkingError",
        "errorMessage": "Client network socket disconnected before secure TLS connection was established",
        "code": "NetworkingError",
        "message": "Client network socket disconnected before secure TLS connection was established",
        "path": null,
        "host": "dynamodb.us-east-1.amazonaws.com",
        "port": 443,
        "region": "us-east-1",
        "hostname": "dynamodb.us-east-1.amazonaws.com",
        "retryable": true,
        "time": "2021-01-06T14:25:06.323Z",
        "stack": [
            "Error: Client network socket disconnected before secure TLS connection was established",
            "    at connResetException (internal/errors.js:609:14)",
            "    at TLSSocket.onConnectEnd (_tls_wrap.js:1547:19)",
            "    at Object.onceWrapper (events.js:420:28)",
            "    at TLSSocket.emit (events.js:326:22)",
            "    at TLSSocket.EventEmitter.emit (domain.js:483:12)",
            "    at endReadableNT (_stream_readable.js:1223:12)",
            "    at processTicksAndRejections (internal/process/task_queues.js:84:21)"
        ]
    },
    "promise": {},
    "stack": [
        "Runtime.UnhandledPromiseRejection: NetworkingError: Client network socket disconnected before secure TLS connection was established",
        "    at process.<anonymous> (/var/runtime/index.js:35:15)",
        "    at process.emit (events.js:326:22)",
        "    at process.emit (/var/task/node_modules/source-map-support/source-map-support.js:495:21)",
        "    at processPromiseRejections (internal/process/promises.js:209:33)",
        "    at processTicksAndRejections (internal/process/task_queues.js:98:32)"
    ]
}

Jan 07 '21 11:01 robsonkades

I got something very similar: intermittent, similar error, different service (S3):

Error: Client network socket disconnected before secure TLS connection was established
    at connResetException (internal/errors.js:609:14)
    at TLSSocket.onConnectEnd (_tls_wrap.js:1547:19)
    at Object.onceWrapper (events.js:420:28)
    at TLSSocket.emit (events.js:326:22)
    at TLSSocket.EventEmitter.emit (domain.js:483:12)
    at endReadableNT (_stream_readable.js:1223:12)
    at processTicksAndRejections (internal/process/task_queues.js:84:21) {
  code: 'ECONNRESET',
  path: null,
  host: 's3.eu-west-1.amazonaws.com',
  port: 443,
  localAddress: undefined
}

Jan 07 '21 14:01 gyula-s

It seems to be a problem at Lambda.

Our support from aws: I also want to confirm that we are already aware of this issue in Nodejs Lambda function and can confirm that it is affecting a very small set of invocations for Nodejs runtime. There are some mitigation steps that we suggest for it, meanwhile the Lambda internal team is doing the investigation for the root cause.

Jan 07 '21 15:01 robsonkades

@robsonkades, thanks for providing the update, can you share the ticket number that you have with AWS and I can look into it, would like to close the issue if the issue is not related to the SDK.

Jan 19 '21 23:01 ajredniwja

@ajredniwja, Thank you for your help!

Case 7831443621

Jan 20 '21 11:01 robsonkades

@robsonkades, thanks for providing that, I can reach out to the team related to that and can update once I hear back from them.

Jan 26 '21 08:01 ajredniwja

Hi - we're also seeing some instances of this in a NodeJs lambda, are there any updates?

Feb 02 '21 11:02 rcoundon

@rcoundon No solution so far

Feb 02 '21 15:02 robsonkades

Ok, thanks for the update. Any ETA?

Feb 02 '21 16:02 rcoundon

I was not able to get any updates as of yet, I have contacted the engineers associated and will update you guys once I have some more insight from Lambda team.

Feb 02 '21 20:02 ajredniwja

2021-02-05T01:27:25.910Z	1f0f65f6-4194-4948-b322-981bb95c4466	ERROR	Unhandled Promise Rejection 	{
    "errorType": "Runtime.UnhandledPromiseRejection",
    "errorMessage": "NetworkingError: Client network socket disconnected before secure TLS connection was established",
    "reason": {
        "errorType": "NetworkingError",
        "errorMessage": "Client network socket disconnected before secure TLS connection was established",
        "code": "NetworkingError",
        "message": "Client network socket disconnected before secure TLS connection was established",
        "path": null,
        "host": "dynamodb.ap-southeast-1.amazonaws.com",
        "port": 443,
        "region": "ap-southeast-1",
        "hostname": "dynamodb.ap-southeast-1.amazonaws.com",
        "retryable": true,
        "time": "2021-02-05T01:27:25.908Z",
        "stack": [
            "Error: Client network socket disconnected before secure TLS connection was established",
            "    at connResetException (internal/errors.js:609:14)",
            "    at TLSSocket.onConnectEnd (_tls_wrap.js:1549:19)",
            "    at Object.onceWrapper (events.js:420:28)",
            "    at TLSSocket.emit (events.js:326:22)",
            "    at TLSSocket.EventEmitter.emit (domain.js:483:12)",
            "    at endReadableNT (_stream_readable.js:1241:12)",
            "    at processTicksAndRejections (internal/process/task_queues.js:84:21)"
        ]
    },
    "promise": {},
    "stack": [
        "Runtime.UnhandledPromiseRejection: NetworkingError: Client network socket disconnected before secure TLS connection was established",
        "    at process.<anonymous> (/var/runtime/index.js:35:15)",
        "    at process.emit (events.js:314:20)",
        "    at process.EventEmitter.emit (domain.js:483:12)",
        "    at process.emit (/var/task/node_modules/source-map-support/source-map-support.js:495:21)",
        "    at processPromiseRejections (internal/process/promises.js:209:33)",
        "    at processTicksAndRejections (internal/process/task_queues.js:98:32)"
    ]
}

I face same error in aws lambda, I am using nodejs looks like aws-sdk lib problem?? using "aws-sdk": "^2.835.0", try upgrade version see how

Feb 05 '21 02:02 yeukfei02

In case it's helpful, our issue is related to writes to S3.

Feb 05 '21 10:02 rcoundon

Same for us. Seems to affect quite significantly our Lambdas. Thanks @ajredniwja for the update.

Feb 08 '21 10:02 Sohett

We've also got the same issue with AppSync - happening on a somewhat sensitive service.

Any idea how we can mitigate this until there is a proper fix?

Feb 16 '21 17:02 jonmifsud

We had the same issue reading from secretsmanager but also reading from s3 today. Our lambdas are running in a VPC so that is likely the issue in our case. https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html#vpc-samples

Feb 25 '21 00:02 brucemackenzie

@brucemackenzie the lambda functions I have do not run in a VPC so I doubt this is a VPC specific issue. The primary function where this is causing us a concern has the issue does run on a cron - every minute - so not sure if that has any impact.

However, I am possibly more concerned should this could happen on webhook requests. Especially as I may not be guaranteed a retry by a third-party service should the function fail.

Feb 25 '21 01:02 jonmifsud

We don't use VPC approximately 0.7% of requests fail.

Feb 25 '21 02:02 robsonkades

Seeing same issue appearing here.

Apr 06 '21 10:04 OliverBailey

+1 same issue. @robsonkades do you still have this issue? This has started to appear for us recently. ( lambda deployed in a VPC, but this seems do not matter )

Apr 13 '21 08:04 illia-sh

+1 same issue while calling a 3rd party service via https

node v14.16.0
aws-sdk 2.804.0

Apr 13 '21 12:04 birsi

+1 same issue node v14.x

2021-04-14T08:07:31.506Z 8636001a-b230-4580-828b-966334e116ee INFO came to callback Error: Client network socket disconnected before secure TLS connection was established at connResetException (internal/errors.js:607:14) at TLSSocket.onConnectEnd (_tls_wrap.js:1544:19) at TLSSocket.emit (events.js:327:22) at endReadableNT (internal/streams/readable.js:1327:12) at processTicksAndRejections (internal/process/task_queues.js:80:21) { code: 'ECONNRESET', path: null, host: 'cognito-idp.us-east-2.amazonaws.com', port: 443, localAddress: undefined } undefined

Apr 14 '21 08:04 paliwalgaurav

@illia-sh yes, still with the same problem

Apr 14 '21 13:04 robsonkades

We also started seeing this a couple of days ago. Before that there was absolutely no problem. The lambda does not run in a VPC but is invoked by API Gateway

Apr 16 '21 05:04 technicallyfeasible

I experience the same error When ts is compiled with js after putting axios in forEach, map environment : Connect Lambda to SQS and run Lambda

2021-04-16T06:00:24.537Z 1f630065-c993-5f4e-bb70-5896e0bbfe96 INFO Error: Client network socket disconnected before secure TLS connection was established at connResetException (internal/errors.js:609:14) at TLSSocket.onConnectEnd (_tls_wrap.js:1549:19) at Object.onceWrapper (events.js:420:28) at TLSSocket.emit (events.js:326:22) at endReadableNT (_stream_readable.js:1241:12) at processTicksAndRejections (internal/process/task_queues.js:84:21) { code: 'ECONNRESET', path: null, host: 'bot-api.kakao.com', port: 443, localAddress: undefined, }

Apr 16 '21 08:04 code-xhyun

Experiencing the same issue for AWS_Lambda while logging data into Sumologic, this is impacting us at a higher level. Requesting some ETA on the same

Client network socket disconnected before secure TLS connection was established at connResetException

Lambda issue

Apr 21 '21 11:04 aagamdoshi

+1 on this issue, we started experiencing this recently on a newly re-deployed micro service. AWS Lambda on Nodejs 12 runtime

{ "errorType": "Runtime.UnhandledPromiseRejection", "errorMessage": "Error: Client network socket disconnected before secure TLS connection was established", "reason": { "message": "Client network socket disconnected before secure TLS connection was established", "name": "Error", "stack": "Error: Client network socket disconnected before secure TLS connection was established\n at connResetException (internal/errors.js:609:14)\n at TLSSocket.onConnectEnd (_tls_wrap.js:1549:19)\n at Object.onceWrapper (events.js:420:28)\n at TLSSocket.emit (events.js:326:22)\n at TLSSocket.EventEmitter.emit (domain.js:483:12)\n at endReadableNT (_stream_readable.js:1241:12)\n at processTicksAndRejections (internal/process/task_queues.js:84:21)" }