serving Activator LB doesn't work well

What version of Knative?

0.9.x 0.10.x 0.11.x Output of git describe --dirty

0.25.2

Expected Behavior

Settings:

2 activator
- no istio sidecar injection
5 replica server
- CC: 1
- TBC : -1
- no istio sidecar injection

With CC: 1 and no istio sidecar injection, I understood that firstAvailableLBPolicy is used.

I expected that all 5 pod get requests evenly through Activator.

Actual Behavior

Four pod got requests evenly, but one pod didn't even get a single request at all. ( I sent around 20000 requests)

Steps to Reproduce the Problem

I made a simple FastAPI webserver with an API that waits for 0.1 seconds.

Sent requests using hey in local.

hey -t 0 -n 10000 -c 100 ${URL}

Mar 15 '22 08:03 hanwgyu

Similar issue : https://github.com/knative/serving/issues/12593

Mar 22 '22 06:03 hanwgyu

if the requests are done quickly enough, the same pod can handle them, then the activator might not need to send the any requests to the last pod

May 03 '22 19:05 nader-ziada

@nader-ziada

if the requests are done quickly enough, the same pod can handle them, then the activator might not need to send the any requests to the last pod

Even if all pods were busy, the remaining requests were not sent to some pods.

In the above test, there are 5 pods, and each pod processes 10 rps. Total traffic that can be handled is 50 rps. On client side, I sent 100 rps using hey.

Jun 03 '22 08:06 hanwgyu

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.