StackExchange
Unexpected response when container is near memory limit
We're seeing the same issue posted here: https://github.com/StackExchange/StackExchange.Redis/issues/2577
StackExchange.Redis.RedisConnectionException: Unexpected response to HMGET: Integer: 1
Triggered when calling HashGetAsync and when the app is running OOM. App is running on k8s in a container with a mem limit.
StackExchange.Redis version: 2.7.17 on .NET 8, Asp.NET Core application.
Difficult to reproduce, it seems to happen when the application is close to the container mem limit. We see this happen alongside System.OutOfMemoryExceptions.
Additionally, (and much scarier) behavior is we're also occasionally getting the wrong value on GETs in this state. E.g. We check the cache for key:123, but the data we get back is what we expect for key:234. We're not sure if it's the GET returning the wrong data or if the initial SET was setting the wrong data.
Hmm, that is weird. It sounds like a protocol desync has occurred. This is very bad, and I plan to overhaul the IO core soon (prototype is active, etc). However, you could enable "high integrity mode" (see the configuration page, https://stackexchange.github.io/StackExchange.Redis/Configuration.html) - this will both confirm whether desync is the issue, and protect you from incorrect data fetched and all the consequences thereof. If it does report a desync, please let us know!
Note that high integrity mode has a slight performance impact, since it adds an additional (tiny) command per operation; in almost all cases, this is effectively invisible and is not a concern.
Can you also please elaborate on which type of container we're on here? We're assuming Linux from k8s, but which specific container base image? I don't think it'll be relevant but with networking bits, we'd like to know as much about the environment as possible.
Yep Linux/containerd. Base image is the standard Ubuntu distro from mcr.microsoft.com/dotnet/aspnet:8.0