sidorares
idleTimeout doenst work in lambda environment?
Hi! I'm building a Lambda function with Drizzle ORM (also tested with TypeORM) and I'm not sure if this is the "normal" operation. Here is the problem:
My application starts the DB connection outside the handler to reuse it between Lambda calls. To reuse the same connection, I do not kill the pool or connections. My idea is that the "idleTimeout" setting will handle this (if not reused again by other Lambda containers). Locally, this works great, but in the Lambda environment, the connections do not respect the timeouts specified in "idleTimeout," and the connection turns into a zombie connection. Am I missing something here?
My connection:
I know this has been quiet for a minute, but I am seeing a similar result. I don't believe it's an issue with idleTimeout, but more an issue with the way that lambdas work in general. I won't attempt to recreate the specifics but this blog has what I believe to be a relevant description of the problem in general. In short if the lambda returns and AWS freezes the lambda, there is no guarantee that any of the ongoing connection management code will be executed, or if it is executed that all connections will have reached the idleTimeout to be released. In the event that a subsequent request is served by a warm lambda and the idleTimeout has been met I do see the connections being cleaned up and no zombie connections result in MySQL. If that isn't the case, these connections persist until MySQL determines they are dead.
That said, if I am willing to accept that the risk of a few zombie connections is ok I am still running into a challenge regarding Unhandled Promise Rejections that are causing subsequent invocations to fail. I'm seeing the following types of errors that cause the Lambda runtime to exit.
"Error: connect ETIMEDOUT",
" at PoolConnection._handleTimeoutError (/opt/nodejs/node_modules/mysql2/lib/base/connection.js:200:17)",
" at listOnTimeout (node:internal/timers:569:17)",
" at process.processTimers (node:internal/timers:512:7)"
"Error: write ECONNRESET",
" at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)",
" at handleWriteReq (node:internal/stream_base_commons:51:26)",
" at writeGeneric (node:internal/stream_base_commons:149:15)",
" at Socket._writeGeneric (node:net:962:11)",
" at Socket._write (node:net:974:8)",
" at writeOrBuffer (node:internal/streams/writable:392:12)",
" at _write (node:internal/streams/writable:333:10)",
" at Writable.write (node:internal/streams/writable:337:10)",
" at PoolConnection.write (/opt/nodejs/node_modules/mysql2/lib/base/connection.js:395:43)",
" at PoolConnection.writePacket (/opt/nodejs/node_modules/mysql2/lib/base/connection.js:303:12)"
"Error: write ECONNRESET",
" at afterWriteDispatched (node:internal/stream_base_commons:160:15)",
" at writeGeneric (node:internal/stream_base_commons:151:3)",
" at Socket._writeGeneric (node:net:962:11)",
" at Socket._write (node:net:974:8)",
" at writeOrBuffer (node:internal/streams/writable:392:12)",
" at _write (node:internal/streams/writable:333:10)",
" at Writable.write (node:internal/streams/writable:337:10)",
" at PoolConnection.write (/opt/nodejs/node_modules/mysql2/lib/base/connection.js:256:32)",
" at PoolConnection.writePacket (/opt/nodejs/node_modules/mysql2/lib/base/connection.js:303:12)",
" at ClientHandshake.sendSSLRequest (/opt/nodejs/node_modules/mysql2/lib/commands/client_handshake.js:46:16)"
From a consumers perspective using the API, I don't see a way that I can add a rejection listener such that these could be handled (albeit likely best I could do is ignore them) so they do not cause the process to exit. I guess my question would be, are these low level errors bubbled to the pool and if so how can I subscribe to them so they are not considered unhandled rejections?
@wellwelwel , wanted to bump to see if I should create a separate issue or if this is related.
Hi, @jej2003 🙋🏻♂️
Your issue is related, but probably not the same. I think it would be easier to find for other users experiencing the same issue with your own issue for this (feel free to make it the way you like it).
As for solving the issues, it's complicated because we don't have any reproductions that we can replicate locally.