opentelemetry-operator icon indicating copy to clipboard operation
opentelemetry-operator copied to clipboard

Allow configuration of target CPU Utilization for HPA

Open moh-osman3 opened this issue 2 years ago • 4 comments

Currently the HPA code hard codes 90% as the target cpu utilization. I am currently running tests with HPA and it would be helpful to be able to change the target utilization to a custom value. Opening this issue to track my change to add this field to the collector config.

moh-osman3 avatar Aug 30 '22 09:08 moh-osman3

cc) @kevinearls

pavolloffay avatar Aug 30 '22 12:08 pavolloffay

Can I piggy back off this request to scale on memory as well? Otherwise I can create a new issue.

ajsaclayan avatar Sep 07 '22 18:09 ajsaclayan

@moh-osman3 could you please as well comment why do you need a custom value fo HPA? What is the use-case?

pavolloffay avatar Sep 20 '22 08:09 pavolloffay

@pavolloffay The purpose of adding TargetCPUUtilization to the CRD is because we wish to tune this value and provide a recommended setting to users. Previously with 90% we are experiencing a lot of flapping when load testing. If targets are uneven and one of the collectors has significantly fewer targets than other collectors, it becomes less likely that scale up will happen (average utilization will be too low, this can lead to throttling in collectors with too many targets which can cause scaling down). Giving users the control over this value might allow them to tune the HPA to their specific needs, if the default is unhelpful.

Previously talked with @jaronoff97 about adding the option to scale on memory for workloads with fluctuating memory needs. I can work on that in a followup PR in the next few weeks, if you don't mind creating an issue for this @ajsaclayan and specifying the use case?

moh-osman3 avatar Sep 20 '22 21:09 moh-osman3

Resolved by https://github.com/open-telemetry/opentelemetry-operator/pull/1066

moh-osman3 avatar Dec 05 '22 21:12 moh-osman3