piraeus-operator
piraeus-operator copied to clipboard
piraeusdatastore
Work Queue Latency increased 10 times
When deploying Piraeus Operator Work Queue Latency increased 10 times
Deleting Custom Resources and the operator itself does not solve the problem. I noticed that after deleting the operator, CRDs remain in the cluster. internal.linstor.linbit.com, only after removing them Work Queue Latency returns to normal times. In the graph you can see the moment of installation and deletion of data internal.linstor.linbit.com CRDs.
At the time of operator installation, the cluster is not empty with 130 different CRDs already deployed.
How to reproduce
Deploy Piraeus Operator configure LinstorCluster Custom Resource observe Work Queue Latency increased. Removing Linstore CRDs (group internal.linstor.linbit.com) normalizes response times.
Environment
- OS: Talos v1.8.3
- Kubernetes version: v1.30.7
- Piraeus Operator: v2.7.1
I guess the apiserver is not optmized for many CRDs? I wonder what it is actually doing that takes so long...
I guess the apiserver is not optmized for many CRDs? I wonder what it is actually doing that takes so long...
20% more CRDs slow down service in order of magnitude. So I think there is something 'special' in these CRDs.
20% more CRDs slow down service in order of magnitude
Does this happen with all CRDs? In any case, this sounds like a Kubernetes issue to me.
No, only when LINSTOR's CRDs are installed Installing other CRDs, even more than LINSTOR's, has no negative impact
It seems that is not a Piraeus issue but Linstore, see https://github.com/LINBIT/linstor-server/issues/428