decorateRequest is not thread safe

[vdh]

Due to the way the code for decorateRequest is written, when multiple requests are executed concurrently the currentAwsArguments cache variable will clash and overwrite wrong event and context data between the simultaneous requests.

[mcollina]

How are multiple requests executed concurrently? AWS lambda only processes one request at a time.

[vdh]

@mcollina Well if someone more knowledgeable than me about AWS can guarantee it doesn't effect production that's at least a silver lining, but when running locally via serverless-offline it's been destroying my sanity for weeks now just managing to find out where and why multiple requests kept swapping event data seemingly at random in race conditions. 😰

A previous developer set up this plugin so I'm unable to check in with them about it, but they set it up as:

let HANDLER = null;
module.exports = async (event, context) => {
	if(!HANDLER) {
		HANDLER = awsLambdaFastify(await initialiseApp());
	}

	return HANDLER(event, context);
};

where initialiseApp returns the Fastify app instance. So the currentAwsArguments variable inside the plugin is being contaminated across events.

[mcollina]

Well if someone more knowledgeable than me about AWS can guarantee it doesn't effect production that's at least a silver lining

Yes. It's well documented: https://docs.aws.amazon.com/lambda/latest/dg/lambda-concurrency.html.

[adrai]

but when running locally via serverless-offline it's been destroying my sanity

which run mode are you using? https://www.serverless.com/plugins/serverless-offline#run-modes

[Eomm]

Spoiler: it is still a draft, but here is an alternative if you don't want this headache anymore: https://github.com/Eomm/blog-posts/pull/45/files#diff-2a221c0b97cb2a3269efba097ce7a8e75b28a42dd122da92b846c58b6bd39415

[vaunus]

We've seen weirdness locally where req.awsLambda is undefined sometimes when a lot of parallel requests are happening. I presume it is because of this issue as the way we simulate invoking the lambda handlers in our local dev environment does create multiple concurrent invocations in the same Node process. We've worked around it by using decorateRequest: false, serializeLambdaArguments: true and the x-apigateway-event header which seems to work well.