docs icon indicating copy to clipboard operation
docs copied to clipboard

Published 20 hours ago • verified publisher icon cloudposse

[FIRING:2] API server high latency

Open osterman opened this issue 7 years ago • 1 comments

what

  • Document remediation procedures

osterman avatar Sep 28 '18 21:09 osterman

The last time this happened, it was due to PrometheusOperator having insufficient memory. We were turned on to this fact as a result of another alert firing that the pod for prometheus was frequently restarting. After closer inspection, we learned that the pod was OOM. Thus, it was restarting and slamming the API with requests causing high latency. After adjusting the memory limits for prometheus operator, it stopped crashing and the API latency errors went away.

osterman avatar Sep 28 '18 21:09 osterman