HikariCP
HikariCP copied to clipboard
Published
20 hours ago •
brettwooldridge
brettwooldridge
JAVA app still uses broken connection after Connection reset by peer
Hello, After trying to figure out whats wrong with our app, I have decided to look here for some help.
We have a JAVA app which uses below packages for Hikari database pool
+--- com.custom.gwtext:gwtext-server
| +--- org.hibernate:hibernate-hikaricp:5.4.6.Final
| | +--- org.hibernate:hibernate-core:5.4.6.Final (*)
| | \--- com.zaxxer:HikariCP:3.2.0 -> 3.3.0
| | \--- org.slf4j:slf4j-api:1.7.25 -> 1.7.36
| +--- com.zaxxer:HikariCP:3.3.0 (*)
The issue is that we are currently trying to migrate to kubernetes and we started to see a lot of java.sql.SQLException: Connection is closed errors after application is running for several hours.
After enabling Hikari DEBUG log and adding leakDetectionThreshold we started to notice a few things -
This is the configuration for the Hikari pool
00:06:05.521 [main] DEBUG com.zaxxer.hikari.HikariConfig - HikariPool-2 - configuration:
00:06:05.522 [main] DEBUG com.zaxxer.hikari.HikariConfig - allowPoolSuspension.............false
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - autoCommit......................false
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - catalog.........................none
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionInitSql...............none
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionTestQuery.............none
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - connectionTimeout...............30000
00:06:05.523 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSource......................none
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceClassName.............none
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceJNDI..................none
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - dataSourceProperties............{password=<masked>}
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - driverClassName................."oracle.jdbc.OracleDriver"
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckProperties...........{}
00:06:05.524 [main] DEBUG com.zaxxer.hikari.HikariConfig - healthCheckRegistry.............none
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - idleTimeout.....................600000
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - initializationFailTimeout.......1
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - isolateInternalQueries..........false
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - jdbcUrl.........................jdbc:oracle:thin:@1.1.1.1:1521:test
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - leakDetectionThreshold..........200000
00:06:05.525 [main] DEBUG com.zaxxer.hikari.HikariConfig - maxLifetime.....................18000000
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - maximumPoolSize.................90
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - metricRegistry..................none
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - metricsTrackerFactory...........none
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - minimumIdle.....................15
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - password........................<masked>
00:06:05.526 [main] DEBUG com.zaxxer.hikari.HikariConfig - poolName........................"HikariPool-2"
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - readOnly........................false
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - registerMbeans..................false
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - scheduledExecutor...............none
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - schema..........................none
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - threadFactory...................internal
00:06:05.527 [main] DEBUG com.zaxxer.hikari.HikariConfig - transactionIsolation............default
00:06:05.528 [main] DEBUG com.zaxxer.hikari.HikariConfig - username........................"user"
00:06:05.528 [main] DEBUG com.zaxxer.hikari.HikariConfig - validationTimeout...............5000
- After starting up the java app and logging in with the user, after some minutes (probably the leakDetectionThreshold) we notice below log entry, which says that there is 2 leaks right away
[HikariPool-2 housekeeper] WARN com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for oracle.jdbc.driver.T4CConnection@5ae99303 on thread pool-10-thread-1, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:104)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:134)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:250)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:258)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:467)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.beginTransaction(Unknown Source)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransaction(HibernateUtilImpl.java:335)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransactionIfNeeded(HibernateUtilImpl.java:330)
at com.custom.gwtext.server.utils.HibernateUtilImpl.getSession(HibernateUtilImpl.java:70)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.reserveIdBatch(DatabaseSequenceIdPoolDataUtil.java:29)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.reserveIdsInDatabase(DatabaseSequenceIdPool.java:106)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.access$300(DatabaseSequenceIdPool.java:24)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool$ReserveIdsTask.run(DatabaseSequenceIdPool.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
and a few minutes later second one
[HikariPool-2 housekeeper] WARN com.zaxxer.hikari.pool.ProxyLeakTask - Connection leak detection triggered for oracle.jdbc.driver.T4CConnection@7b911a8f on thread pool-6-thread-1, stack trace follows
java.lang.Exception: Apparent connection leak detected
at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.getConnection(DatasourceConnectionProviderImpl.java:122)
at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:38)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:104)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:134)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:250)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:258)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:246)
at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:83)
at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:467)
at jdk.internal.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.beginTransaction(Unknown Source)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransaction(HibernateUtilImpl.java:335)
at com.custom.gwtext.server.utils.HibernateUtilImpl.startTransactionIfNeeded(HibernateUtilImpl.java:330)
at com.custom.gwtext.server.utils.HibernateUtilImpl.getSession(HibernateUtilImpl.java:70)
at com.custom.gwtext.server.utils.DbUtilImpl.doWork(DbUtilImpl.java:516)
at com.custom.gwtext.server.utils.DbUtilOracleImpl.executeQueryDBSpecific(DbUtilOracleImpl.java:117)
at com.custom.gwtext.server.utils.DbUtilOracleImpl.executeQueryDBSpecific(DbUtilOracleImpl.java:67)
at com.custom.gwtext.server.utils.DbUtilImpl.executeQueryInternal(DbUtilImpl.java:266)
at com.custom.gwtext.server.utils.DbUtilImpl.executeQuery(DbUtilImpl.java:173)
at com.custom.gwtext.server.query.QueriesDefinitionRegistryImpl.getData(QueriesDefinitionRegistryImpl.java:147)
at com.custom.gwtext.server.query.QueriesDefinitionRegistryImpl.getData(QueriesDefinitionRegistryImpl.java:74)
at com.custom.pins.server.data.helper.QueryHelper.executeSelectFirstRow(QueryHelper.java:54)
at com.custom.pins.server.data.helper.QueryHelper.selectFirstRow(QueryHelper.java:36)
at com.custom.pins.server.data.helper.QueryHelper.selectFirstValueRequired(QueryHelper.java:80)
at com.custom.pins.server.business.bdt.BusinessDataTableDataUtil.existsBusinessTable(BusinessDataTableDataUtil.java:20)
at com.custom.pins.server.business.bdt.BusinessDataTableBuilderImpl.existsTable(BusinessDataTableBuilderImpl.java:225)
at com.custom.pins.server.security.audit.ws.WebServiceLoggerConfigImpl.createWsRequestConfig(WebServiceLoggerConfigImpl.java:116)
at com.custom.pins.server.security.audit.ws.WebServiceLoggerConfigImpl.getWsRequestConfig(WebServiceLoggerConfigImpl.java:75)
at com.custom.pins.server.security.audit.ws.WebServiceAuditLogService.getWsRequestSpecificConfiguration(WebServiceAuditLogService.java:88)
at com.custom.pins.server.security.audit.ws.WebServiceAuditLogService.logIfConfigEnabled(WebServiceAuditLogService.java:71)
at com.custom.pins.server.security.audit.ws.WebServiceAuditLogService.lambda$getInternalWsLogConsumer$0(WebServiceAuditLogService.java:62)
at com.custom.gwtext.server.security.audit.AuditLogServiceImpl.lambda$createEventHandler$2(AuditLogServiceImpl.java:90)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:150)
at java.base/java.lang.Thread.run(Thread.java:834)
after which they both stay active for a long time
[HikariPool-2 housekeeper] DEBUG com.zaxxer.hikari.pool.HikariPool - HikariPool-2 - After cleanup stats (total=17, active=2, idle=15, waiting=0)
- Then after a few hours, this can be 3-10 hours, so no specific range, maybe depends on activity etc, no idea, this pops up in logs
[pool-10-thread-1] WARN c.zaxxer.hikari.pool.ProxyConnection - HikariPool-2 - Connection oracle.jdbc.driver.T4CConnection@5ae99303 marked as broken because of SQLSTATE(08006), ErrorCode(17002)
java.sql.SQLRecoverableException: IO Error: Connection reset by peer
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:929)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1119)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3780)
at oracle.jdbc.driver.T4CCallableStatement.executeInternal(T4CCallableStatement.java:1300)
at oracle.jdbc.driver.OraclePreparedStatement.execute(OraclePreparedStatement.java:3887)
at oracle.jdbc.driver.OracleCallableStatement.execute(OracleCallableStatement.java:4230)
at oracle.jdbc.driver.OraclePreparedStatementWrapper.execute(OraclePreparedStatementWrapper.java:1079)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.execute(ProxyPreparedStatement.java:44)
at com.zaxxer.hikari.pool.HikariProxyCallableStatement.execute(HikariProxyCallableStatement.java)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.lambda$reserveIdBatch$0(DatabaseSequenceIdPoolDataUtil.java:34)
at org.hibernate.jdbc.WorkExecutor.executeWork(WorkExecutor.java:37)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2306)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2303)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:306)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2325)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2310)
at jdk.internal.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.doWork(Unknown Source)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.reserveIdBatch(DatabaseSequenceIdPoolDataUtil.java:29)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.reserveIdsInDatabase(DatabaseSequenceIdPool.java:106)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.access$300(DatabaseSequenceIdPool.java:24)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool$ReserveIdsTask.run(DatabaseSequenceIdPool.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Connection reset by peer
at java.base/sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245)
at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:355)
at oracle.net.nt.TimeoutSocketChannel.read(TimeoutSocketChannel.java:144)
at oracle.net.ns.NIOHeader.readHeaderBuffer(NIOHeader.java:82)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:139)
at oracle.net.ns.NIOPacket.readFromSocketChannel(NIOPacket.java:101)
at oracle.net.ns.NIONSDataChannel.readDataFromSocketChannel(NIONSDataChannel.java:80)
at oracle.jdbc.driver.T4CMAREngineNIO.prepareForReading(T4CMAREngineNIO.java:98)
at oracle.jdbc.driver.T4CMAREngineNIO.unmarshalUB1(T4CMAREngineNIO.java:534)
at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:485)
at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:252)
at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:612)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:223)
at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:56)
at oracle.jdbc.driver.T4CCallableStatement.executeForRows(T4CCallableStatement.java:907)
... 29 common frames omitted
[pool-10-thread-1] INFO com.zaxxer.hikari.pool.ProxyLeakTask - Previously reported leaked connection oracle.jdbc.driver.T4CConnection@5ae99303 on thread pool-10-thread-1 was returned to the pool (unleaked)
[HikariPool-2 connection closer] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-2 - Closing connection oracle.jdbc.driver.T4CConnection@5ae99303: (connection is broken)
also its not active anymore (the active one is the second leaked one T4CConnection@7b911a8f on thread pool-6-thread-1 which somehow does not reset)
And right after this happens, our JAVA app starts to throw a lot of (even like 10 per second, sometimes less) below errors and becomes very slow
[pool-10-thread-1] WARN o.h.e.jdbc.spi.SqlExceptionHelper - SQL Error: 0, SQLState: null
[pool-10-thread-1] ERROR o.h.e.jdbc.spi.SqlExceptionHelper - Connection is closed
[pool-10-thread-1] ERROR l.p.p.s.i.DatabaseSequenceIdPool - error executing work
org.hibernate.exception.GenericJDBCException: error executing work
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:311)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2325)
at org.hibernate.internal.SessionImpl.doWork(SessionImpl.java:2310)
at jdk.internal.reflect.GeneratedMethodAccessor128.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.hibernate.context.internal.ThreadLocalSessionContext$TransactionProtectionWrapper.invoke(ThreadLocalSessionContext.java:350)
at com.sun.proxy.$Proxy326.doWork(Unknown Source)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.reserveIdBatch(DatabaseSequenceIdPoolDataUtil.java:29)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.reserveIdsInDatabase(DatabaseSequenceIdPool.java:106)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool.access$300(DatabaseSequenceIdPool.java:24)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPool$ReserveIdsTask.run(DatabaseSequenceIdPool.java:125)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.sql.SQLException: Connection is closed
at com.zaxxer.hikari.pool.ProxyConnection$ClosedConnection.lambda$getClosedConnection$0(ProxyConnection.java:489)
at com.sun.proxy.$Proxy307.prepareCall(Unknown Source)
at com.zaxxer.hikari.pool.ProxyConnection.prepareCall(ProxyConnection.java:290)
at com.zaxxer.hikari.pool.HikariProxyConnection.prepareCall(HikariProxyConnection.java)
at com.custom.pins.server.idgeneration.DatabaseSequenceIdPoolDataUtil.lambda$reserveIdBatch$0(DatabaseSequenceIdPoolDataUtil.java:31)
at org.hibernate.jdbc.WorkExecutor.executeWork(WorkExecutor.java:37)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2306)
at org.hibernate.internal.SessionImpl$1.accept(SessionImpl.java:2303)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.coordinateWork(JdbcCoordinatorImpl.java:306)
... 16 common frames omitted
so it seems like although T4CConnection@5ae99303 is marked as broken, its never evicted and the app still tries to use it.
Can this be related to HikariCP? Looks like we are using quite outdated version of Hikari and hibernate, but somehow those versions are baked in some custom com.custom package and not sure how we can "overwrite" those versions. We are using gradle for building the WAR file.
Oh and also just to add - there are no such issues when running without kubernetes. Maybe this is related to network/firewalls but somehow I think Hikari should work around those broken connections in case they happen.
I would really appreciate the help or even some pointers where/what to look for. Thanks
Upgraded to 5.0.1 and found the same issue (or looking the same, right now digging into it).
