HTTP socket keepalive with aws lambda container reuse issue

[SimonWoolf]

trafficstars

Customer reports that when ably-js is used from a lambda function with authUrl token auth, if the lambda container is reused for a subsequent invocation, requests to the authUrl on that invocation requests fail immediately with ECONNRESET.

My suspicion is that aws lambda is dropping all network traffic while the function is between invocations, which is leaving the tcp socket to the auth server apparently still open (as it was never properly closed) but actually not. To test this I've asked the customer to disable HTTP keepAlive in the restAgentOptions. If that works then we should figure out a better workaround that does not require the customer to do that. (eg maybe do a single unconditional retry on ECONNRESET even if we normally wouldn't, or something)

┆Issue is synchronized with this Jira Bug by Unito

[SimonWoolf]

Update: customer has confirmed that the keepAlive workaround was effective, so we need to figure out a more permanent fix.

[amcdnl]

I'm having a similar issue with the realtime client. Any suggestions for that?

[SimonWoolf]

If you mean just for making rest requests, the same workaround applies - add restAgentOptions: {keepAlive: false} to the clientOptions object. E.g. new Ably.Rest({ key: 'xxx', restAgentOptions: {keepAlive: false}});.

If you mean the realtime connection - mostly customers don't use the realtime client in lambda invocations as the common usecase for lambdas is for transient actions, e.g. firing off a publish, for which there's not much point opening a websocket only to make one publish on it and then close it again, it would be more efficient to just make a rest request. But if you really do need to open a realtime connection for whatever reason, you should probably explicitly close() it once you've finished using it in each function invocation.

[amcdnl]

Doesn't look like restAgentOptions is a option:

Also since lambdas reused warm containers, wouldn't it be more efficient to keep a open one as long as its hot?

[owenpearson]

@amcdnl Sorry this option is missing from our type definitions - I've opened an issue to get that fixed. The option will still work, however. See the snippet below for a temporary workaround:

import Ably, { Types } from 'ably';

const realtime = new Ably.Rest.Promise({
  key: 'xxx',
  restAgentOptions: { keepAlive: false },
} as Types.ClientOptions);