HTTPScaledObject restarts existing pods

[ngoctin02806]

Report

I am using Keda http add-on to scale pods based on pending request. After installing it and setup forward request to keda-add-ons-http-interceptor-proxy service, I have simulated 502 requests to test scale pod automatically. But, number of pending request is equal or greater than spec.targetPendingRequests: 5, I see that HTTPScaledObject has restarted my old pods and then continuing to scale more pods to spec.replicas.max: 7.

I do not know this is bug or feature of http-add-on. Please helping me fix this ?

Expected Behavior

I expect http-add-on still keeps old pods and scales new pods basing on specification defined

Actual Behavior

My old pods is restarted by http-add-on when pending request is equal or greater spec.targetPendingRequests: 5

Steps to Reproduce the Problem

1. To simulate 502 requests
2. To watch pods by command watch kubectl -n <namesapce> get pod
3. When pending request is equal or greater than spec.targetPendingRequests: 5, I see that HTTPScaledObject restarted my old pods

Logs from KEDA HTTP operator

2024-06-07T02:58:44Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "e96a4b20-9d5d-41cd-8d03-1ff1e861a10d", "resource": "apps/v1.Deployment", "name": "my-scaler"}
2024-06-07T02:58:49Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "df6be875-be0c-4e6f-97ac-47c636af38b8"}
2024-06-07T02:58:49Z	ERROR	external_push_scaler	error running internalRun	{"type": "ScaledObject", "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "error": "rpc error: code = Canceled desc = context canceled"}
github.com/kedacore/keda/v2/pkg/scalers.(*externalPushScaler).Run.func1
	/workspace/pkg/scalers/external_scaler.go:249
github.com/kedacore/keda/v2/pkg/scalers.(*externalPushScaler).Run
	/workspace/pkg/scalers/external_scaler.go:268
2024-06-07T02:58:49Z	INFO	Successfully restored scaleTarget's replica count back to the original	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "df6be875-be0c-4e6f-97ac-47c636af38b8", "replicaCount": 3}
2024-06-07T02:58:49Z	INFO	Successfully finalized ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "df6be875-be0c-4e6f-97ac-47c636af38b8"}
2024-06-07T02:59:10Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "b464998d-9b43-455b-87a9-6be238604d7b"}
2024-06-07T02:59:10Z	INFO	Adding Finalizer for the ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "b464998d-9b43-455b-87a9-6be238604d7b"}
2024-06-07T02:59:10Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "b464998d-9b43-455b-87a9-6be238604d7b", "resource": "apps/v1.Deployment", "name": "my-scaler"}
2024-06-07T02:59:10Z	INFO	Creating a new HPA	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "b464998d-9b43-455b-87a9-6be238604d7b", "HPA.Namespace": "my-scaler", "HPA.Name": "keda-hpa-my-scaler-keda-autoscaler"}
2024-06-07T02:59:10Z	INFO	Initializing Scaling logic according to ScaledObject Specification	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "b464998d-9b43-455b-87a9-6be238604d7b"}
2024-06-07T02:59:10Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "0e430f33-840c-42fa-9df3-df7825f9d8e2"}
2024-06-07T02:59:10Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "0e430f33-840c-42fa-9df3-df7825f9d8e2", "resource": "apps/v1.Deployment", "name": "my-scaler"}
2024-06-07T02:59:40Z	INFO	Reconciling ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "7c077642-456b-4f24-8ab2-1cfb3c785d17"}
2024-06-07T02:59:40Z	INFO	Detected resource targeted for scaling	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"my-scaler-keda-autoscaler","namespace":"my-scaler"}, "namespace": "my-scaler", "name": "my-scaler-keda-autoscaler", "reconcileID": "7c077642-456b-4f24-8ab2-1cfb3c785d17", "resource": "apps/v1.Deployment", "name": "my-scaler"}

HTTP Add-on Version

0.7.0

Kubernetes Version

1.29

Platform

Other

Anything else?

apiVersion: http.keda.sh/v1alpha1
kind: HTTPScaledObject
metadata:
  name: my-scaler-name-keda-autoscaler
  namespace: my_namespace
spec:
  hosts:
  - sub.domain.com
  pathPrefixes:
  - /api/v1
  targetPendingRequests: 5
  scaleTargetRef:
    name: my-deployment-name
    kind: Deployment
    apiVersion: apps/v1
    service: my-service-svc
    port: 8000
  replicas:
    min: 3
    max: 7