partial sync buffers should be bound by memory usage and not element size

[kostasrim]

Based on a recent discussion #5431 (which is not yet confirmed why we see the 350mb) brought an interesting point:

We limit the how many elements the buffers can contain and NOT how much memory they consume.

I think this is not the best approach and the actual limit should be the memory used instead of how many elements the buffers can contain (which should be bounded as well -- just the main consideration should be the actual memory used)

[kostasrim]

@adiholden we should discuss

[applike-ss]

What's the status here? I guess it's not fixed in 1.32.0 yet?

[kostasrim]

Hi @applike-ss , it's not yet fixed due to other priorities atm

[applike-ss]

Thanks for the update. 👍

[applike-ss]

Is there an update for this? We are checking this upon each time we are updating our dragonflies, due to the need to set --shard_repl_backlog_len=1 to ensure no crashes are happening

[romange]

@applike-ss we won't be able to address this issue in the near term. As a workaround i suggest increasing the memory reservation of your dragonfly processes. Alternatively, you can try and fix it yourself and send a PR for review.

[applike-ss]

Any time frame after which i should bug you again? Just to not start annoying you with every release? I'll not be able to fix this in code or by memory increase on our end.