keda icon indicating copy to clipboard operation
keda copied to clipboard

KEDA frequent reconciling causing ApiServer LIST verb Saturation

Open djsly opened this issue 1 year ago • 3 comments

Report

Keda operator goes into a "restart" loop causing a constant reconciliation loop. Due to the non cache LIST operation the apiserver was not able to answer LIST request for other services as well.

Expected Behavior

Keda Operator should use cached List operation and it should also not get into reconciliation loop

Actual Behavior

Graph showing the apiserver saturation time

apiserver-list-saturation

Steps to Reproduce the Problem

  1. Hard to tell, it started after Keda Operator lost it's leader election due to a slow ApiServer response : https://github.com/operator-framework/operator-sdk/issues/5186

Logs from KEDA operator

keda-pods-apiserver-list-saturation.csv

KEDA Version

2.6.1

Kubernetes Version

1.23

Platform

Microsoft Azure

Scaler Details

Prometheus

Anything else?

Azure Support: 2209160030000259 for the impact and severity of such a bug

djsly avatar Sep 19 '22 13:09 djsly

Hi, this is probably a duplicate of https://github.com/kedacore/keda/issues/2914

This issue has been solved most likely. By chance, could you please update KEDA version to the latest one?

zroubalik avatar Sep 19 '22 14:09 zroubalik

we certainly can, what do you mean "by chance" ?

djsly avatar Sep 19 '22 15:09 djsly

we certainly can, what do you mean "by chance" ?

the original sentece was missing a . - I hope that it makes sense now ;)

zroubalik avatar Sep 19 '22 18:09 zroubalik

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Nov 18 '22 18:11 stale[bot]

This issue has been automatically closed due to inactivity.

stale[bot] avatar Nov 25 '22 22:11 stale[bot]