avalanche icon indicating copy to clipboard operation
avalanche copied to clipboard

exit gracefully if not enough memory

Open sed-i opened this issue 4 years ago • 2 comments

When I try to generate 100k metrics on a 1G VM, avalanche hangs and dies. Also the log messages are delayed by ~1 min, i.e. don't really match the printed timestamp.

./cmd --metric-count=100000 --label-count=10 --series-count=10 --value-interval=10 --series-interval=36000 --metric-interval=36000 --port=9001
Serving ur metrics at localhost:9001/metrics
2021-09-30 21:35:44.801681073 +0000 UTC m=+30.791942268: refreshing metric values
2021-09-30 21:35:54.83536445 +0000 UTC m=+40.825625685: refreshing metric values
Killed

sed-i avatar Sep 30 '21 21:09 sed-i

How would you suggest we exit gracefully? There is no data / state that is lost if the process is OOM killed.

bill3tt avatar Oct 07 '21 17:10 bill3tt

There are two decision points I can think of:

  1. exit right away if the expected memory is more than available (excluding swap)
  2. exit if actual refresh rate exceeds value-interval

sed-i avatar Oct 07 '21 17:10 sed-i