helm: GOMAXPROCS automatic value set too high for low-utilisation clusters

[bruksnys]

Describe the bug

Default values for GOMAXPROCS cause cgroup CPU throttling.

To Reproduce

Steps to reproduce the behavior:

1. Deploy mimir-distributed helm chart using default values.
2. Default GOMAXPROCS environment variable value for mimir-distributor is automatically set to 8.
3. cgroups will start to throttle mimir-distributor almost immediately after deployment and we can observe spikes in p99 latency.

Manually setting GOMAXPROCS to 1 does not cause CPU throttling or latency spikes, but it does negatively impact the throughput (slightly).

Expected behavior

GOMAXPROCS default value for less than 1000milicore requests should be lower (1 or 2) or tuneable (since the tradeoff appears to be between throughput and latency).

Environment

• Infrastructure: Kubernetes (EKS).
• Deployment tool: helm (mimir-distributed)

Additional Context

Example with slightly increased requests set to 300m and limits set at 600m using default GOMAXPROCS (8)

Example latency spikes:

Let me know if there is anything I've missed or any further information is needed.

Thank you!

[pracucci]

Example latency spikes:

Are latency spikes caused by the CPU throttling? I struggle to correlate a 1m 99th latency spike with CPU throttling.

GOMAXPROCS default value for less than 1000milicore requests should be lower (1 or 2) or tuneable (since the tradeoff appears to be between throughput and latency).

I'm wondering if instead we should check whether a CPU limit is set and, if so, ensure the GOMAXPROCS is not set higher than 2x the limit.

[narqo]

How about we, by default, derived GOMAXPROCS from k8s CPU limits?

- name: GOMAXPROCS
  valueFrom:
    resourceFieldRef:
      resource: limits.cpu
      divisor: 1

In the snippet above, the value for GOMAXPROCS will be set to what Kubernetes sees as the limit for this deployment: either what operator set explicitly, or what is the maximum on the node.

Note on the "by default" part,

1. an operator should be able to overwrite that with a custom env passed from the values
2. for some modules (e.g. querier), our chart already tries to calculate an ideal GOMAXPROCS from default CPU requests.

Fallback'ing to the valueFrom.resourceFieldRef is only to help an operator to not "shoot themselves into the foot", by setting CPU limits, but forgetting to bound the Go runtime (e.g. in "ingester").

[tiithansen]

Just started to upgrade and noticed also that it sets GOMAXPROCS to 8. In our setup we have multiple distributors and they average ~0.5 CPU during normal operating. We push ~400k samples sec. ~11 mil active series. We have used approach GOMAXPROCS = requests.cpu and limits = requests.cpu * 1.25 where limits ceils at requests + 1.