Timeout awaiting response
Hey team!
Now and then we encounter "Timeout awaiting response" We're hosting our env on Azure and using Azure Cache for Redis Tier: Standard 1 GB
Here is the error message: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 5884ms elapsed, timeout is 5000ms), command=GET, next: GET #, inst: 0, qu: 0, qs: 0, aw: False, bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, last-in: 210, cur-in: 0, sync-ops: 0, async-ops: 18238, serverEndpoint: #, conn-sec: 22769.29, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, clientName: client(SE.Redis-v2.7.20.59961), IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=1,Free=32766,Min=1,Max=32767), POOL: (Threads=2,QueuedItems=0,CompletedItems=460840,Timers=10), v: 2.7.20.59961 (Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)
The issue occurred at 23:49:36.5374749
Server load:
CPU:
Cache hit:
Cache read:
Total keys:
Would it be at all possible to lend a helping hand? We would highly appreciate your help. Thanks in advance!
This looks like the client is actually awaiting data. The first question that comes to mind is: how large are these values that the GET is fetching?
Hey @NickCraver I'm really grateful for your reply!
There are basically 2 scenarios when this issue is reproduced:
- GET for a health check (GET HEALTHCHECK).
- GET to fetch some stashed data.
In the first case, we're expecting just a response (nil) even though there is nothing stored by this key(Key: HEALTHCHECK). But still, we get a timeout from time to time.
SLOWLOG GET showed "GET HealthChech":
Regarding real data, the size of the value was about 4.5 KB, a simple array of strings.
It doesn't, however, occur all the time. It may happen a couple of times through 2 weeks period.
SLOWLOG GET showed:
@NickCraver any ideas regarding the topic, root cause and solution?