confluentinc
Consumer pause/Resume funtionality leasks memory
It looks like consumer pause/resume functionality leaks memory, at least in situation when consumed partitions has all the time lot of data available. I have tested this with only consuming data from kafka. Without Pause/Resume, With adaptive Pause/Resume and Aggressive Pause/Resume. Pod memory utilization shows clearly that memory consumption is not healthy. From consumer statistics I can't find any reason for memory issue.
In attachment pictures from memory usages, and producer configuration parameters
FYI I was not able to replicate this with the ruby bindings running for several minutes. With 10 000 pause/resume cycles the memory increase was close to 0.
In our system, during the test application received 120k reports/second and producer produced little more to keep back pressure. In aggressive case had hundred or thousand pause/resumes on second. That is not normal case it was only to test and get enough cycles.
Hi, here is still one picture that shows relation of pause calls and pod memory usage pause count graph has wrong y axis label it is not Bytes it is just count...