autoscaler VPA not producing reasonable recommendations on memory for Cronjobs

Which component are you using?: VPA Recommender What version of the component are you using?: 0.8.1 What k8s version are you using (kubectl version)?:

➜ vpa kubectl version Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T18:43:01Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

What environment is this in?: EKS What did you expect to happen?: Recommendations that make sense What happened instead?: Recommendations dont match recorded metrics from prometheus

We are noticing after configuring VPA recommender for 100+ cronjobs the results we are getting dont seem to match the memory used by the pods. Many of our cronjobs are very short lived and ive found a few notes around that say it doesnt work well for jobs that run under a minute. I think this explains the majority of our cases but im still left with multiple examples of jobs that run for multiple minutes but their recommendations are much higher or lower than the expected values.

One other thing I noticed is that multiple cronjobs all seem to get the exact same target memory value. Should this be possible? These are all different cronjobs that run different workloads. Not sure how they could end up with all the same target values. Is the issue here the cronjobs only run for a few minutes a day and there isnt enough data to work with?

628694953 job1
628694953 job2
628694953 job3
628694953 job4
628694953 job5
628694953 job6
628694953 job7
628694953 job8
628694953 job8
628694953 job9
628694953 job10
628694953 job11
628694953 job12

  name: job1
  namespace: default
  resourceVersion: "48748421"
  selfLink: /apis/autoscaling.k8s.io/v1/namespaces/default/verticalpodautoscalers/analytics-extracts-perform
  uid: 9e13d83f-87af-47c8-9e86-df99b0144818
spec:
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      controlledResources:
      - memory
      maxAllowed:
        memory: 2500Mi
      minAllowed:
        memory: 250Mi
  targetRef:
    apiVersion: batch/v1beta1
    kind: CronJob
    name: job1
  updatePolicy:
    updateMode: "Off"
status:
  conditions:
  - lastTransitionTime: "2021-03-25T13:26:00Z"
    status: "True"
    type: RecommendationProvided
  recommendation:
    containerRecommendations:
    - containerName: rails
      lowerBound:
        memory: "293363252"
      target:
        memory: "628694953"
      uncappedTarget:
        memory: "628694953"
      upperBound:
        memory: 2500Mi
        
  name: job2
  namespace: default
  resourceVersion: "48748421"
  selfLink: /apis/autoscaling.k8s.io/v1/namespaces/default/verticalpodautoscalers/analytics-extracts-perform
  uid: 9e13d83f-87af-47c8-9e86-df99b0144818
spec:
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      controlledResources:
      - memory
      maxAllowed:
        memory: 2500Mi
      minAllowed:
        memory: 250Mi
  targetRef:
    apiVersion: batch/v1beta1
    kind: CronJob
    name: job2
  updatePolicy:
    updateMode: "Off"
status:
  conditions:
  - lastTransitionTime: "2021-03-25T13:26:00Z"
    status: "True"
    type: RecommendationProvided
  recommendation:
    containerRecommendations:
    - containerName: rails
      lowerBound:
        memory: "293363252"
      target:
        memory: "628694953"
      uncappedTarget:
        memory: "628694953"
      upperBound:
        memory: 2500Mi

Mar 25 '21 17:03 rdubya16

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

Jul 20 '21 17:07 fejta-bot

/remove-lifecycle stale

Jul 20 '21 17:07 rdubya16

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

Dec 14 '21 15:12 k8s-triage-robot

/remove-lifecycle stale

Dec 14 '21 15:12 mike-stewart

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

Mar 14 '22 16:03 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

Apr 13 '22 16:04 k8s-triage-robot

/remove-lifecycle rotten

Apr 13 '22 18:04 mike-stewart

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

Jul 12 '22 18:07 k8s-triage-robot

/remove-lifecycle stale

Jul 13 '22 00:07 mike-stewart

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle stale
Mark this issue or PR as rotten with /lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

Oct 11 '22 00:10 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue or PR as fresh with /remove-lifecycle rotten
Close this issue or PR with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

Nov 10 '22 01:11 k8s-triage-robot

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Dec 10 '22 01:12 k8s-triage-robot

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied

After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied

After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen

Mark this issue as fresh with /remove-lifecycle rotten

Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Dec 10 '22 01:12 k8s-ci-robot

autoscaler autoscaler copied to clipboard

VPA not producing reasonable recommendations on memory for Cronjobs

autoscaler
autoscaler copied to clipboard