lamby
lamby copied to clipboard
rails-lambda
SIGTERM SignalException: Init error when loading handler
Been seeing these a lot more lately on the latest cookiecutter with Ubuntu image. Normally I would only see these on shutdown but I am starting to see them more on first invoke/init. Here is a screenshot of the stack trace. May fork or PR here. https://github.com/aws/aws-lambda-ruby-runtime-interface-client
Questions because I'm seeing this elsewhere as well, have seen very few reports of this from others, and I'm trying to narrow down causes:
- Does this only happen during startups of newly deployed code, i.e., did you deploy new code and then first invocations caused this error?
- Do you have any additional extensions loaded?
I wonder whether or not lambda gets caught terminating new inits along side old instances of the runtime by broadcasting SIGTERMs differently than before...
Does this only happen during startups
Yes, I think so. I have updated the image above to show a bit more of the completeness. Any log that has this SIGTERM... it is the ONLY thing in the log. So my assumption is that it is only during startup and the Lambda service bails on this function and starts another.
Do you have any additional extensions loaded?
Yup, these two:
- https://github.com/rails-lambda/crypteia
- https://github.com/rails-lambda/lambda_punch
I wonder whether or not lambda gets caught terminating new inits along side old instances of the runtime by broadcasting SIGTERMs differently than before...
"along side" 🤔
I'm gonna turn on these two env vars and monitor things.
LAMBDA_PUNCH_LOG_LEVEL: debug
CRYPTEIA_DEBUG: true
"along side" meaning that a sigterm gets broadcast to terminate old lambdas running while the new lambdas are attempting to init and both old and new versions of the lambda get terminated and then the new version starts again. Just a theory. I'd need some kind of sha or other code version identifier to confirm that.
For us, we resolved by manually trapping the sigterm per the details suggested here. It feels like this is something the underlying RIC should handle, but at the same time, I can understand why signal handling may be something best left to the app code rather than the handler.
Thanks! Just learned about this today too:
- https://twitter.com/wm_matuszewski/status/1680071801156653056?s=43&t=az8mU_0mUrWwmcIF6DD2OA
- https://github.com/aws-samples/graceful-shutdown-with-aws-lambda/tree/main/nodejs-demo