Running Gateway on AWS Lambda hits socket errors when loading managed schema

[sdemjanenko]

I run @apollo/gateway@^0.36.0 on AWS Lambda and it appears that the background Promise that fetches managed schema from uplink.api.apollographql.com can fail, putting the entire server on the lambda in a bad state. The only way this resolves itself is when the Lambda cools off and gets destroyed.

The errors that I observe in logs are:

This data graph is missing a valid configuration. More details may be available in the server logs.

An error occurred during Apollo Server startup. All GraphQL requests will now fail. The startup error was: An error occurred while fetching your schema from Apollo: request to https://uplink.api.apollographql.com/ failed, reason: Client network socket disconnected before secure TLS connection was established

Note: I've seen reasons

• Client network socket disconnected before secure TLS connection was established
• socket hang up

This mainly seems to happen when the lambda experiences an error (e.g. throwing AuthenticationError in context, an query result validation error or even a lambda timeout).

I was partially able to mitigate this providing a fetcher to ApolloGateway to force the connection: close header to prevent connection reuse. Despite this change, the error still happens. My current working theory is that the background Promise to load the managed schema is running when the Lambda is frozen (because callbackWaitsForEmptyEventLoop is set to false)

callbackWaitsForEmptyEventLoop – Set to false to send the response right away when the callback runs, instead of waiting for the Node.js event loop to be empty. If this is false, any outstanding events continue to run during the next invocation.

https://docs.aws.amazon.com/lambda/latest/dg/nodejs-context.html

It appears that this may be a bug with AWS Lambda itself: https://github.com/aws/aws-sdk-js/issues/3591

Given that Lambda can freeze when the client response is complete (and we shouldn't delay the client response to refresh the managed schema), does it make sense to turn off the refresh mechanism in Lambda or implement it as a CI/CD step?

[Nicoowr]

Same error here, we had to add schemaConfigDeliveryEndpoint: null to the gateway config to keep the old system.

[moraisp]

I also have the same error. I understand this is a common problem with lambda. Do we have a way to manually trigger the managed schema update in a way that allow us to wait for it to complete before exiting the lambda?

[lassesteffen]

Same error for me, running on Node 14

[MaLub]

Same here - it works well for a while and that it breaks down. But afterwards runs for days. Try to set the experimental_pollInterval do a few minutes - hopefully finishing the Lambda before fetch the managed federation config again.

[adikari]

any updates on this from the team? I am getting this error constantly

UplinkFetcher failed to update supergraph with the following error: An error occurred while fetching your schema from Apollo: request to https://uplink.api.apollographql.com/ failed, reason: socket hang up

[c0pp3rt0p]

Just to add my findings to this as well. MFH will not retry on post methods and that is what is being used when fetching the remote SDLs from the subgraphs with federation. Even with retries you could still find yourself in this strange state that occurs when the failure happens. So adding retries is not a silver bullet.