tortoise Tortoise scales unreasonably when application misbehaves

https://mercari.slack.com/archives/C6HC4JBKM/p1717478614923889

When pods are stuck in crashbackloopoff, tortoise recommendation algorithm does not consider this and causes recommendations to scale HPA max replicas to a unreasonable amount

Jun 05 '24 05:06 randytqwjp

causes recommendations to scale HPA max replicas to a unreasonable amount

Does it mean the tortoise lowered the target utilization of HPA too much and consequently HPA increased the replica number?

Jun 11 '24 07:06 sanposhiho

there was a bug in application logic and caused some pods to be stuck in crashbackloop while remaining pods utilization went up. Tortoise then increased maxreplica for this service but since new pods also get stuck in crashbackloop, tortoise kept increasing maxreplica

Aug 23 '24 05:08 randytqwjp

My suggestion is that we can improve Tortoise to check all Pods' status, and then if the ratio of such crashed Pods is higher than the criterion (50% etc), stop changing the max replica (or maybe stop changing any parameters until the situation is stable).

Aug 23 '24 09:08 sanposhiho