Missing documentation on callbackWaitsForEmptyEventLoop and run modes

[jer-sen]

It should be clear that callbackWaitsForEmptyEventLoop is not handled at all by serverless-offline with these impacts:

• in worker-threads mode: lambdas are terminated after an idle timeout without waiting for an empty event loop => ressources may not be cleared properly (connection to DB...) so they have to be cleared in a setTimeout juste before functionCleanupIdleTimeSeconds
• in in-process mode: lambdas are never terminated => ressources may never be cleared so they have to be cleared in a setTimeout juste before functionCleanupIdleTimeSeconds
• in child-processes mode: lambdas are terminated just after returning without waiting for an empty event loop => ressources may not be cleared properly so they have to be cleared before returning

And of course the lambda always returns immediately even if event loop is not empty.

[dnalborczyk]

thank you for filing the issue @jer-sen

you are right, callbackWaitsForEmptyEventLoop is entirely being ignored right now, and I'm not sure if it's even possible to support with the current node.js api or javascript api.

I have personally never used callbackWaitsForEmptyEventLoop, and just looking at the AWS documentation and others, generally speaking usage doesn't seem to be advised.

we should definitely add it to the docs. would you mind creating a PR for this?

[jer-sen]

I don't think it can be supported by node.js.

It is the recommended way to use MongoDB with handlers using a callback cf https://www.mongodb.com/docs/atlas/manage-connections-aws-lambda/

But you are right, with async handlers, callbackWaitsForEmptyEventLoop is also ignored by AWS so maybe it should be just stated in the doc that:

serverless-offline ignores callbackWaitsForEmptyEventLoop value and always act as if callbackWaitsForEmptyEventLoop was set to false (but depending on the running mode, and unlike what AWS does, some code can be executed after the handler returned a promise or called the callback)

You should also add documentation about lambda termination for each mode, I suggests that (some changes between my first message in this thread):

• in worker-threads mode: lambdas are terminated after an idle timeout => ressources may not be cleared properly (connection to DB...) so they have to be properly cleared in a setTimeout just before functionCleanupIdleTimeSeconds
• in in-process mode: lambdas are never terminated => ressources may never be cleared (until serverless-offline has exited) so they have to be properly cleared in a setTimeout juste before functionCleanupIdleTimeSeconds is reached
• in child-processes mode: lambdas are terminated just after returning => ressources will not be cleared properly so they have to be cleared before returning or in a SIGTERM handler (which will always be triggered just after returning)

[dnalborczyk]

serverless-offline ignores callbackWaitsForEmptyEventLoop value and always act as if callbackWaitsForEmptyEventLoop was set to false

not quite sure if false covers the case tho, because as you mentioned the code keeps running (does not freeze), which would be the behavior for true. I would argue it behaves more like true (the default) than false, minus the freezing. (to be sure I would have to check the code, including the behavior with the different modes.

(but depending on the running mode, and unlike what AWS does, some code can be executed after the handler returned a promise or called the callback)

I think this applies to callback only (not the "returned Promise")

You should also add documentation about lambda termination for each mode, I suggests that (some changes between my first message in this thread):

• in worker-threads mode: lambdas are terminated after an idle timeout => ressources may not be cleared properly (connection to DB...) so they have to be properly cleared in a setTimeout just before functionCleanupIdleTimeSeconds

I don't think anyone needs to do anything special regarding the serverless-offline cleanup for worker-threads? when it's terminated, everything else dies with it (I hope), similar as a lambda container? that said, if anyone wants to hook into any events, it should be possible with event listeners on the process (I haven't tried it yet) or other similar supported web api.

• in in-process mode: lambdas are never terminated => ressources may never be cleared (until serverless-offline has exited) so they have to be properly cleared in a setTimeout juste before functionCleanupIdleTimeSeconds is reached

same as above. a db connection could just remain open, or possibly auto close with reconnects. it wouldn't be any different than with async functions.

• in child-processes mode: lambdas are terminated just after returning => ressources will not be cleared properly so they have to be cleared before returning or in a SIGTERM handler (which will always be triggered just after returning)

same as above.

[jer-sen]

serverless-offline ignores callbackWaitsForEmptyEventLoop value and always act as if callbackWaitsForEmptyEventLoop was set to false

not quite sure if false covers the case tho, because as you mentioned the code keeps running (does not freeze), which would be the behavior for true. I would argue it behaves more like true (the default) than false, minus the freezing. (to be sure I would have to check the code, including the behavior with the different modes.

(but depending on the running mode, and unlike what AWS does, some code can be executed after the handler returned a promise or called the callback)

It's more like false because the response is immediately sent back to the client (in case of an http event) and the lambda timeout will not be triggered after returning. But, you are right, the lambda is not freezed just after returning, except with child-processes (it is killed immediately). Note that it is similar to what can happen on AWS when the lambda is immediately reused.

I think this applies to callback only (not the "returned Promise")

Yes, on AWS if the handler returns a Promise, the handler is freezed immediately (same as callbackWaitsForEmptyEventLoop === false with callback handler).

You should also add documentation about lambda termination for each mode, I suggests that (some changes between my first message in this thread):

• in worker-threads mode: lambdas are terminated after an idle timeout => ressources may not be cleared properly (connection to DB...) so they have to be properly cleared in a setTimeout just before functionCleanupIdleTimeSeconds

I don't think anyone needs to do anything special regarding the serverless-offline cleanup for worker-threads? when it's terminated, everything else dies with it (I hope), similar as a lambda container? that said, if anyone wants to hook into any events, it should be possible with event listeners on the process (I haven't tried it yet) or other similar supported web api.

AFAIK connections to a DB server must be cleaned properly. As for now, workers are terminated after functionCleanupIdleTimeSeconds (if they are not reused in the meantime). Things have to be properly cleaned before they are terminated. AFAIK event listeners on the process does not work in workers and the only way to detect a termination would be use messages. Unlike on AWS, with --reloadHandler you may have a lot of workers (since they are not reused), and so cleanup is more important.

• in in-process mode: lambdas are never terminated => ressources may never be cleared (until serverless-offline has exited) so they have to be properly cleared in a setTimeout juste before functionCleanupIdleTimeSeconds is reached

same as above. a db connection could just remain open, or possibly auto close with reconnects. it wouldn't be any different than with async functions.

Since --reloadHandler is ignored (always false) there wont be lots of instances so nothing important to clean. Note that with this mode you can have 2 events handled simultaneously in the same runtime environment, which can never happen on AWS... So resources must not be cleaned before returning because another event may be using it.

• in child-processes mode: lambdas are terminated just after returning => ressources will not be cleared properly so they have to be cleared before returning or in a SIGTERM handler (which will always be triggered just after returning)

same as above.

Since --reloadHandler is ignored (always true) cleaning is important as in worker-threads mode.

[dnalborczyk]

It's more like false because the response is immediately sent back to the client (in case of an http event) and the lambda timeout will not be triggered after returning. But, you are right, the lambda is not freezed just after returning, except with child-processes (it is killed immediately). Note that it is similar to what can happen on AWS when the lambda is immediately reused.

I can see your point. speaking of which, child-processes only exists for historical reasons when worker threads weren't available yet. I think it's about time to remove it in the next major. I'm thinking of creating a survey in the next day for arguments for keeping or removing. I don't think worker threads does not cover any scenario child-processes does, and if there is, we can hopefully fix it. the same almost applies for in-process, the only possible reason I can see to use it is to save memory for local development, as libraries are shared across lambdas (incl. serverless and plugins).

AFAIK connections to a DB server must be cleaned properly. As for now, workers are terminated after functionCleanupIdleTimeSeconds (if they are not reused in the meantime). Things have to be properly cleaned before they are terminated.

you are right, things should ideally be properly cleaned. what I'm trying to say is that this applies to "real" lambdas as well. this is not a serverless-offline only issue?

AFAIK event listeners on the process does not work in workers and the only way to detect a termination would be use messages.

good question. I'll have to give it a try. I would imagine that those would work. That said, I don't really know how lambdas are being killed and if there's a time period (incl. CPU) given to take action (and run some cleanup code), or any api for that matter (e.g. postpone). I was always under the impression that there is not. the container would just be removed while frozen. if you have any information feel free to send it my way.

Unlike on AWS, with --reloadHandler you may have a lot of workers (since they are not reused), and so cleanup is more important.

Since --reloadHandler is ignored (always true) cleaning is important as in worker-threads mode.

good point. I have to check the code for that, I'm not sure what the cleanup timing procedure is for unused lambdas in serverless-offline. if --reloadHandler is being used those should probably be cleaned up right away, without any delay.

[jer-sen]

this is not a serverless-offline only issue?

With --reloadHandler you can have a lot of instances, and AFAIK there is no concurrency limit in serverless-offline. So there can be more to clean than on AWS.

if you have any information feel free to send it my way.

I think that frozen lambdas are killed without any notice.