buffering metrics on a disk

[azhiltsov]

When relay can't send a metrics out it keeps them in a memory till certain limit. It will be useful to store them on a disk when the memory buffer full. This will give more flexibility for storage reboot/maintenance etc.

[grobian]

Sounds like this would be a bit more thorough rewrite to become memory bound (or a percentage) and try pushing out blocks to disk when exceeding the threshold. It's sort of a 2-stage rocket, where one overflows into the other, which may be too slow to handle, so drop metrics, like the current 1-stage rocket does.

[azhiltsov]

The disk write will be linear, so even spinning disk should handle it fine.

[xneo64]

@grobian we also encounter this issue frequently. During a cluster member downtime, the relay can't contain all of the pending metrics in memory (even after tuning) - there is simply too much to hold.

Do you know of any approach we may use to counteract this? Could we have the "file" cluster act as a backup of sorts?

[grobian]

carbon-c-relay is more of a router than a cache, a failover target could maybe do what you need.

[tantra35]

@xneo64 you may use our version of grafsy https://github.com/tantra35/grafsy for this purposes