r2dbc-pool
r2dbc-pool copied to clipboard
r2dbc
R2dbcTimeoutException - pool runs out of connections
We have a high volume processing project using r2dbc-pool and r2dbc-postgresql modules. In some environments we started periodically experiencing issues where attempts to get a connection from pool end up with exception:
org.springframework.dao.DataAccessResourceFailureException, Message: Failed to obtain R2DBC Connection; nested exception is io.r2dbc.spi.R2dbcTimeoutException: Connection acquisition timed out after 120000ms
We captured a couple of heap dumps and observed:
- all io.rdbc.pool.PooledConnections have isClosed=true
- all io.r2dbc/postgresql.PostgresqlConnection objects they point to seem to be open and idle (client.isClosed = 0, client.transactionStatos = IDLE)
- Looking at io.r2dbc.pool.ConnectionPool and its reactor.pool.SimpleDequePool, it has a pendingSize = 7, acquired = 20 (we set the pool size to 20 per container and the idleResources deque seems empty
We suspect that in a period of a spike of error conditions with connections (e.g. lock timeout) there's a racing or timing condition that occurs that prevents connections from being released properly and returned to the pool.
Do you have suggestions for investigating and troubleshooting this issue to it's root cause? Thanks in advance!
@SimoneGiusso Thanks Simone! I've tried the repro from #198 but was not able to reproduce the problem. The issue seems very close to what we are experiencing. I've asked Alexei (the creator of #198) for details.
UPDATE: I was actually able to reproduce the problem after running the app for a while. I am investigating and will keep you posted.
@jivkodobrev Hi, do you have any update? Can you share some code example, so I can help you solve it? Thanks
@cyberluke Hi - unfortunately I got reassigned to something else and will not get the chance to work on this. BTW - check the #198 mentioned above, it seems to be the issue. HTH