Timeout exception on Azure App Service

[gramotei]

We suddenly started experiencing timeouts in production.

Stack:

• Azure App Service - P2v3

• Azure Redis Cache - P1 (tried with Redis on VM, same result)

• .NET (asp.net) - 6.0.28

• Microsoft.Extensions.Caching.StackExchangeRedis - 6.0.28

• CPU load like - 2% without spikes

• Network like - 30/40 connections/sec without spikes also

• There are no slow queries logged

Explored all options, read many articles.

• Upgrading from 2.2.4 to 2.8.0 made situation better but problem is not gone :(
• Tried increasing timeout from 5sec to 15sec didn't help

Timeout awaiting response (outbound=0KiB, inbound=0KiB, 15249ms elapsed, timeout is 15000ms), 
command=HMGET, next: HMGET PROD:Tenant:XXXXXXXXX, 
inst: 0, qu: 0, qs: 0, aw: False, 
bw: SpinningDown, rs: ReadAsync, ws: Idle, in: 0, last-in: 0, cur-in: 0, sync-ops: 42, async-ops: 117823, 
serverEndpoint: xxxxxxx.redis.cache.windows.net:6380, 
conn-sec: 2107.98, aoc: 0, mc: 1/1/0, mgr: 10 of 10 available, 
clientName: xxxxxxxxxx_c3f6697074(SE.Redis-v2.8.0.27420), 
IOCP: (Busy=0,Free=1000,Min=4,Max=1000), 
WORKER: (Busy=1,Free=32766,Min=4,Max=32767), 
POOL: (Threads=8,QueuedItems=0,CompletedItems=2910368,Timers=27), 
v: 2.8.0.27420 
(Please take a look at this article for some common client-side issues that can cause timeouts: https://stackexchange.github.io/StackExchange.Redis/Timeouts)

It would be great to have at least direction how to troubleshoot because now obvious reason for that