fastcache icon indicating copy to clipboard operation
fastcache copied to clipboard
verified publisher icon VictoriaMetrics

Implement API for getting memory allocation statistics

Open rjl493456442 opened this issue 2 years ago • 5 comments

This PR implements a API for retrieving offheap memory allocation statistics.

Fastcache allocates the offheap memory "manually" in order to mitigate the overhead of golang garbage collection. However it also means this part of memory won't be tracked by golang standard metrics.

In order to accurately know how much memory is used by our application, we need this additional statistics to sum up the standard memory usage.

rjl493456442 avatar Aug 08 '23 06:08 rjl493456442

@valyala please take a look?

rjl493456442 avatar Aug 08 '23 06:08 rjl493456442

For the record, our need for the metrics arose from a missed Reset operation which resulted in an eventual OOM. None of our monitoring systems detected the issue ahead of time since the memory was not reported by Go. Would be nice to be able to track it at least explicitly to have an early warning for similar issues in the future.

karalabe avatar Aug 08 '23 07:08 karalabe

I'm curious how you got an OOM. Did you set maxBytes higher than the available memory?

erikdubbelboer avatar Aug 08 '23 08:08 erikdubbelboer

No, we had ephemeral cache instances and a refactor lost the Reset when the outer temporary object got cleaned up.

karalabe avatar Aug 08 '23 08:08 karalabe

@valyala any chance to take a look?

rjl493456442 avatar Aug 21 '23 01:08 rjl493456442