redisson
redisson copied to clipboard
redisson
Microsoft Azure and Redisson: Unable to init enough connections amount
Context
We are in the process of migrating to Microsoft Azure and are planning to make use of the Azure Redis Cache. Because of this, I am in the process of investigating Redisson and have been for the most part impressed by the performance and feature set.
However, I have been running into this intermittent issue for a few days now. Every now and again, when the application starts up, Redisson will throw an error, telling me that it's only able to initialise 31 of 32 connections. My experiments with Lettuce didn't seem to run into anything similar to this.
Anyone have any thoughts or suggestions on things to look into to resolve this? Is there something particular about Azure Redis Cache that might need to be configured? Does Redisson need all 32 connections or would reducing the required number potentially resolve this?
Expected behavior
Successfully connects to Redis every time.
Actual behavior
Infrequently throws the following error on application start-up:
<21-Jun-2019 09:14:07,916 o'clock BST> <Error> <HTTP> <BEA-101216> <Servlet: "InitSWCTest" failed to preload on startup in Web application: "test/app".
java.lang.ExceptionInInitializerError
at com.demo.test.ejb.permissions.engine.cache.PermissionCacheManager.createManager(PermissionCacheManager.java:27)
at com.demo.test.ejb.permissions.engine.cache.PermissionCacheManager.<clinit>(PermissionCacheManager.java:9)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at com.demo.test.common.ClassUtil.initialise(ClassUtil.java:30)
Truncated. see log file for complete stacktrace
Caused By: org.redisson.client.RedisConnectionException: Unable to init enough connections amount! Only 31 from 32 were initialized. Redis server: test.redis.cache.windows.net/XX.XXX.XX.XXX:YYYY
at org.redisson.connection.pool.ConnectionPool$1.lambda$run$0(ConnectionPool.java:160)
at org.redisson.misc.RedissonPromise.lambda$onComplete$0(RedissonPromise.java:183)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:502)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:476)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:415)
Truncated. see log file for complete stacktrace
Caused By: io.netty.channel.ConnectTimeoutException: connection timed out: test.redis.cache.windows.net/XX.XXX.XX.XXX:YYYY
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
Redis version
3.2.7
Redisson version
5.10.7
Redisson configuration
Config config = new Config();
config.useSingleServer()
.setAddress(redisConnectionString)
.setPassword(redisPassword);
RedissonClient redissonClient = Redisson.create(config);
Hi,
connections or would reducing the required number potentially resolve this?
You need to set connectionMinimumIdleSize and connectionPoolSize to 30 if you experience connections limitation by Azure Redis Cache.
Hi mrniko, thanks for the quick reply.
That makes sense, though then my next question is, what if it then at a later date it starts only managing to initialise 29? Is there something that can be done to make it more robust?
From what I understand, Azure Redis Cache should allow thousands of connections, so I'm not sure why we'd be experiencing limitations on that end.
it then at a later date it starts only managing to initialise 29
It shouldn't. If Azure doesn't have any limitation then try to increase connection timeout. Though, even 10 connections should be enough.
We had the same issue then we read that the Azure Redis group suggests to using only Jedis or Lettuce. See: https://gist.github.com/warrenzhu25/1beb02a09b6afd41dff2c27c53918ce7 I'm not sure why they suggest to use only those two clients and not redisson (we liked the features reddison has) but we went with their suggestion...
@mrniko since you're here: Do you know why the Azure Redis group doesn't include Redisson on their suggested clients? Have you been in contact with them?.
@AlbozDroid
Do you know why the Azure Redis group doesn't include Redisson on their suggested clients?
I don't know either. I can say only that Redisson is fully compatible with Azure Redis Cache and Redisson customers use it.
Have you been in contact with them?
Not yet. I'll try to contact him. By the way even Microsoft uses it. Their team left a minor issue on github a while ago. This issue was fixed.
Not yet. I'll try to contact him. By the way even Microsoft uses it. Their team left a minor issue on github a while ago.
If I was you, I'll try to contact their Redis Engineer (the one I linked above) to have a clarification. Perhaps they noticed an issue with it? In that case I'd expect them to flag the issue here or send a pull request for a fix. The fact that they totaly ignore Redisson (the most famous Java client) without giving any reason for it is weird to me. I'd just be curious to know their reasons. Anyway, off topic.
@mrniko Still having issues with this. I set the ConnectionMinimumIdleSize and ConnectionPoolSize to 24 and increased the ConnectTimeout to 15000ms.
There also doesn't seem to be a particularly heavy load on our azure redis instance. Maximum server load about 2%~, cpu usage 2%~, memory usage 1%~.
Any other ideas we could try?
Config
Config config = new Config();
config.useSingleServer()
.setConnectionMinimumIdleSize(24)
.setConnectionPoolSize(24)
.setConnectTimeout(15000)
.setAddress(redisConnectionString)
.setPassword(redisPassword);
Stack Trace
ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 67) MSC000001: Failed to start service jboss.deployment.unit."test-scheduler.war".undertow-deployment: org.jboss.msc.service.StartException in service jboss.deployment.unit."test-scheduler.war".undertow-deployment: java.lang.ExceptionInInitializerError
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1378)
at java.lang.Thread.run(Thread.java:748)
at org.jboss.threads.JBossThread.run(JBossThread.java:485)
Caused by: java.lang.ExceptionInInitializerError
at com.demo.test.scheduler.app.SchedulerListener.contextInitialized(SchedulerListener.java:29)
at io.undertow.servlet.core.ApplicationListeners.contextInitialized(ApplicationListeners.java:187)
at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:216)
at io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:185)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1502)
at io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:250)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:96)
at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:78)
... 8 more
Caused by: org.redisson.client.RedisConnectionException: Unable to init enough connections amount! Only 23 from 24 were initialized. Redis server: test.redis.cache.windows.net/XX.XXX.XX.XXX:YYYY
at org.redisson.connection.pool.ConnectionPool$1.lambda$run$0(ConnectionPool.java:160)
at org.redisson.misc.RedissonPromise.lambda$onComplete$0(RedissonPromise.java:183)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:502)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:476)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:415)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:540)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:533)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:114)
at org.redisson.misc.RedissonPromise.tryFailure(RedissonPromise.java:96)
at org.redisson.connection.pool.ConnectionPool.promiseFailure(ConnectionPool.java:330)
at org.redisson.connection.pool.ConnectionPool.lambda$createConnection$1(ConnectionPool.java:296)
at org.redisson.misc.RedissonPromise.lambda$onComplete$0(RedissonPromise.java:183)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:502)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:495)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:474)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:415)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:540)
at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:533)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:114)
at org.redisson.misc.RedissonPromise.tryFailure(RedissonPromise.java:96)
at org.redisson.client.RedisClient$2$2.run(RedisClient.java:245)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:405)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: io.netty.channel.ConnectTimeoutException: connection timed out: test.redis.cache.windows.net/XX.XXX.XX.XXX:YYYY
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe$1.run(AbstractNioChannel.java:267)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
... 7 more
I could replicate the above(using 32/24 as pool size) with the Azure Cache for Redis (Standard tier) after retrying a few times...
What is weird is that when it failes it always fails by one. We had the same issue...
The Redis server of course doesn't know about the connectionPoolSize so this feels like a client side issue.
Are you connecting to Azure Cache from Azure network on external?
We are currently connecting externally.
I did a minor improvement redisson-3.11.1-SNAPSHOT.jar.zip
Could anyone give this version a try?
Thanks for that, I'll take a look and let you know if we have any luck
We are currently connecting externally.
Same as me. You can also try to connect to the server provided my configs.
Thanks for that, I'll take a look and let you know if we have any luck
I'm looking forward to your feedback
Had the same issue again. Even after reducing my connection size to 10. It's pretty infrequent so normally takes me several app server restarts before it occurs.
Config config = new Config();
config.useSingleServer()
.setConnectionMinimumIdleSize(10)
.setConnectionPoolSize(10)
.setConnectTimeout(15000)
.setAddress(redisConnectionString)
.setPassword(redisPassword);
I'll give some more context on my use case in case that helps:
The core of my application is in an .ear file deployed on one web application server. This initialises a RedissonClient, with the above config, on startup which is used in to retrieve user permission details from redis. It also publishes messages to certain topics and subscribes to another.
I also have a .war in a different web app server which initialises a RedissonClient on startup and subscribes to the topics the .ear publishes to.
When the app server shuts down, I call removeAllListeners() on the topic(s) and shutdown() on its RedissonClient.
Both the ear and war have thrown the error.
I'm getting this error from Azure Redis database if more than 10 Redisson instances are created:
org.redisson.client.RedisException: ERR max number of clients reached
Yes, Azure Cache has connections limitation in around 230 connections. As for AWS Elasticache it doesn't have such limitation. I recommend you to reduce connections amount per Redisson instance and/or share one Redisson instance among different wars through JNDI registry. More info about JNDI is here https://github.com/redisson/redisson/tree/master/redisson-tomcat#shared-redisson-instance
@AlbozDroid So this is not a bug of Redisson.
Yes, Azure Cache has connections limitation in around 230 connections.
Hmm, our subscription should be offering us 1000 connections, and the maximum we've hit is 77.
I reached 240 connections. How can I adjust it?
Details on pricing and connection limits etc. here: https://azure.microsoft.com/en-gb/pricing/details/cache/
for basic subscription I have 256 - 20000. But I can't reach more than 256.
Basic is divided into several different subscriptions (C0-C6). The cheapest option gives 256 connections, the most expensive gives 20,000.
What does your Connections metric graph show?
It shows that we briefly peaked at 77 connected clients a few hours ago, then shortly afterwards 44. Besides those, we've been between 3 and 9 for the past 24 hours
My graph shows that I reached the top - 256. I use c0 instance, the cheapest one. In trial period
@mrniko
Just to give an update on this issue. We ended up creating a new azure redis cache instance and deleting our old one and we've not managed to replicate the issue since, despite hammering it repeatedly. The new instance is in a different region, so it could have been a network issue or something
Ran into this issue with Azure Redis Cache.
Factory method 'redisson' threw exception; nested exception is org.redisson.client.RedisConnectionException: Unable to init enough connections amount! Only 5 of 24 were initialized. Redis server: xxx.redis.cache.windows.net/x.x.x.x:y
I'm not sure why this issue was closed, because we never had this issue with Lettuce client for several months, but see this within weeks of using Redisson. This is clearly a client-side problem and needs to be looked into further. @mrniko Do you want to reopen it or shall I create a new ticket?
@mrniko Thanks for reopening the ticket. I want to add some details that might help.
In my case, there are four types of clients that connect to Redis using Redisson.
- Long-running: This is a web app that once started, will keep running for days and weeks. There are more than one instances of this app.
- QA tests: Use a single client for the duration of the tests.
- Load tests: Use a single client for the duration of the tests.
- CLI: One-off request made by a human.
Clearly, these clients don't need the same number of connections for the same period of time. One option is for the user to configure the connection pool accordingly, but firstly, Redisson docs offer no guidance as to what the optimal parameters might be, and secondly, it is more code that, for some reason, clients like Lettuce don't require the user the write.
@asarkar You're forgetting those running in serverless environments. We're running into this issue in our production AWS Lambda environment.
