Question: Behavior of the Agent in AWS Lambda with `callbackWaitsForEmptyEventLoop` set to false?

[astorm]

The AWS Lambda Context object has an obscure setting, callbackWaitsForEmptyEventLoop. Per the docs

Set to false to send the response right away when the callback runs, instead of waiting for the Node.js event loop to be empty. If this is false, any outstanding events continue to run during the next invocation

It's unclear what the agent behavior will be with this set to false, or what effect (if any) this setting has on the freeze/thaw cycle.

This issue is intended as a place for folks who run into this issue to report problems, as well as a place for us to drop our own research into this setting as time allows.

[trentm]

Here we wrap the given Lambda handler callback, if any: https://github.com/elastic/apm-agent-nodejs/blob/v3.31.0/lib/lambda.js#L493

such that the APM agent finishes flushing its data to the Lambda extension before the given callback is called: https://github.com/elastic/apm-agent-nodejs/blob/v3.31.0/lib/lambda.js#L395-L398

There are timeouts involved in this flush handling -- separately in waiting for inflight spans to be serialized and for the extension to respond to the intake request. As well, there isn't a guarantee that the APM agent's first APM server version fetch request is complete by the time the first invocation is complete.

However, in general, the lambda handler wrapped is designed to have completed its work for a Lambda invocation before the callback is complete, so the value of callbackWaitsForEmptyEventLoop shouldn't impact its behaviour.