operator icon indicating copy to clipboard operation
operator copied to clipboard

VMAgent doesn't work with VPA

Open sdomme opened this issue 2 years ago • 9 comments

Hello, we are trying to setup a VPA on top of a VMAgent. Unfortunately we get the following error from the VPA

E0718 13:54:51.559558 1 cluster_feeder.go:570] Cannot get target selector from VPA's targetRef. Reason: Unhandled targetRef operator.victoriametrics.com/v1beta1 / VMAgent / application, last error Resource victoria/application has an empty selector for scale sub-resource

Here the manifest of the VPA:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: vmagent-application-vpa
  namespace: victoria
spec:
  targetRef:
    apiVersion: operator.victoriametrics.com/v1beta1
    kind: VMAgent
    name: application
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
      - containerName: vmagent
        maxAllowed:
          cpu: 1

Looking at the CRD of the VMAgent I would expect it to work since

 subresources:
      scale:
        labelSelectorPath: .status.selector
        specReplicasPath: .spec.shardCount
        statusReplicasPath: .status.shards
      status: {}

is configured.

Adding another piece to the CRD

- additionalPrinterColumns:
    - description: current selector
      jsonPath: .status.selector
      name: Selector
      type: string

shows an empty string.

k get vmagent application -n victoria
NAME          SHARDS COUNT   REPLICA COUNT   SELECTOR
application   0              1

Any help is appreciated. Maybe we just do something wrong?

sdomme avatar Jul 18 '23 14:07 sdomme

Hello! Looks like vpa has some conflict with operator on scale subresource, so it's not working now. What's your case to scale vmagent? Maybe horizontal scaling can help.

Haleygo avatar Jul 19 '23 15:07 Haleygo

The issue at vpa is from us as well. We can't pinpoint the root cause of this issue yet. We have a fleet of EKS Clusters with the same configuration for metrics. Each cluster has different workloads so the VMAgent has different resource requirements. The compute resources are scaled very dynamically in dependency of the consumption. Therefore the VPA.

sdomme avatar Jul 20 '23 07:07 sdomme

Oh, looks like operator needs to change something here to fit vpa then.

Haleygo avatar Jul 20 '23 15:07 Haleygo

Hello! Are there any updates on this? seems to effect us

ajaykumarmandapati avatar Oct 10 '23 09:10 ajaykumarmandapati

We also have an issue with this as we have policies in place which checks for existence of a VPA but we cant define a working one.

monotek avatar Feb 12 '24 16:02 monotek

Sorry for delay. I think, we should prioritize it.

f41gh7 avatar Apr 17 '24 00:04 f41gh7

Fix was added with linked PR. It'll be a part of the next release.

Also, I think, that we should add direct support for VPA and HPA resources as .spec field for:

  • vmalert - VPA
  • vmagent - VPA+HPA
  • vmcluster - VPA+HPA for select and insert components
  • vmauth - HPA + VPA
  • vmalertmanager - VPA.

Operator is responsible for setting targetRef field with corresponding value for reference (deployment/statefulset).

f41gh7 avatar Apr 26 '24 13:04 f41gh7