avalanche
avalanche copied to clipboard
exit gracefully if not enough memory
When I try to generate 100k metrics on a 1G VM, avalanche hangs and dies. Also the log messages are delayed by ~1 min, i.e. don't really match the printed timestamp.
./cmd --metric-count=100000 --label-count=10 --series-count=10 --value-interval=10 --series-interval=36000 --metric-interval=36000 --port=9001
Serving ur metrics at localhost:9001/metrics
2021-09-30 21:35:44.801681073 +0000 UTC m=+30.791942268: refreshing metric values
2021-09-30 21:35:54.83536445 +0000 UTC m=+40.825625685: refreshing metric values
Killed
How would you suggest we exit gracefully? There is no data / state that is lost if the process is OOM killed.
There are two decision points I can think of:
- exit right away if the expected memory is more than available (excluding swap)
- exit if actual refresh rate exceeds
value-interval