Some pods with dapr actors deployed on K8S seem idle

[kaluznyt]

Hi,

I'm seeing a issue I'm not able to understand anyhow, I have a deployment of .NET application utilising Dapr Actors component. We scaled it up recently from 25 pods to around 40 since we saw some performance issues (like cpu throttling etc, OOM).

However, after scale-up some Pods (~ around 5) seems completely idle, like not picking any work. Looking at the logs (attached, from sidecar) I'm not able to see any errors whatsoever, other Pods are running the actors normally, do you have any idea ?

I was wondering that there might be some scaling limits on the dapr itself (perhaps some default values or something), but it doesn't seem to be the case as far as I know.

This is how the CPU/MEM utilisation looks like on k8s. Those with low utilisation are idle, rest are fine (all basically are the same in terms of deployment, image etc)

This is the tail log from the daprd, from those pods

App also can call the actor without a problem, but it doesn't execute any actors that are created.

[philliphoff]

@kaluznyt This is probably a good question for the Dapr runtime repo and/or the Dapr Discord actors channel.