StackExchange
How to get the excat reason behind redis timeout
Could I get some guidance on how to identify the actual root cause behind the timeout ? Is there any configuration that can be enabled to print the actual error message instead of the generic timeout exception??
StackExchange.Redis.RedisTimeoutException: Timeout performing SET (5000ms), next: SET {hangfire}:locks:schedulepoller, inst: 0, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 385, serverEndpoint: xxxxxxx, mc: 1/1/0, mgr: 10 of 10 available, clientName: XXX(SE.Redis-v2.6.48.48654), IOCP: (Busy=0,Free=1000,Min=2,Max=1000), WORKER: (Busy=45,Free=32722,Min=2,Max=32767), POOL: (Threads=45,QueuedItems=34,CompletedItems=406), v: 2.6.48.48654 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
You can check SLOWLOG on the server, but given the 45 thread on the exception there I'd guess you're having contention issues. How big is the processor you're running on? And can you look at a memory dump/thread analysis when this happens?
- The Redis cluster is running on AWS and comprises of 3 nodes. Enabled SLOWLOG , but it's not providing any additional information.
- Are you asking about the processor size of the client application or the size of redis nodes ?
- Do you think increasing the Minimum thread pool count to 200 will help ?
- Working on collecting a memory dump as suggested.
- Are you asking about the processor size of the client application or the size of redis nodes ?
Client side here!
- Do you think increasing the Minimum thread pool count to 200 will help ?
It might, depends how quickly you're growing. If the CPU is actually pegged it won't help - if you're out of threads because of sync blocks, it can.