documentation: quantities and metric relation, case sensitiveness, metric context/selection

[cforce]

quantities and metric relation

Taken from the docs.. "They are specified as Quantities, so you can use m | "" | k | M | G | T | P | E to easily represent the value you want to use."

The above sentence quotes that highWatermark and lowWatermark values can or must? be extended by quantity. I assume its dependent on the type of metric.

What type of quantity is "" ? What about percent values like i think it shall be able to use all kind of metrics which have a defined max and a min value like e.g kubernetes.memory.usage_pct or kubernetes.cpu.usage_pct. Do i need to set only the value 90 or 90% ?

If i don't add a quantity but just the number, will it take a default (which) or will it fail?

case sensitiveness

There are different metrics i can use . eg.

• https://docs.datadoghq.com/agent/kubernetes/data_collected/
• https://docs.datadoghq.com/agent/docker/?tab=standard

The docker Metrics e.g "Memory" have upper case. Is there generally any case sensitiveness ?

metric context/selection

Taking above two metric sources - system/docker and kubernetes - i could use similar metrics from both e.g system.mem.used vs kubernetes.memory.usage If i deploy a resource wpa

      metricSelector:
        matchLabels:
          app: {{ .Chart.Name }}
          release: {{ .Release.Name }}

will the both metric measurements be scoped to the container level per pod or do they have different scopes and which?

Which of these might be the better choice to control the scale

[CharlyF]

Hi there!

The above sentence quotes that highWatermark and lowWatermark values can or must? be extended by quantity.

They can. They will be correctly interpreted whether you specify a Quantity identifier (m, G, k ...) as can be seen here. So concretely using the following is equivalent:

  metrics:
  - external:
      highWatermark: "70"
      lowWatermark: "60"
      metricName: sinus
      metricSelector:
        matchLabels:
          foo: bar

to

  metrics:
  - external:
      highWatermark: "70000m"
      lowWatermark: "60"
      metricName: sinus
      metricSelector:
        matchLabels:
          foo: bar

What type of quantity is "" ?

If you don't specify anything the value will be just be an int64. The conversion in the code makes it so if you don't specify a quantity and the value will be converted to a quantity and be compared as a millivalue (so 70 becomes 70000m) and compared to the value returned by the metrics server.

For context, the smallest value that can be represented is 1 milli, anything below will be 0. The External Metrics server exposes a Quantity type, so if you query it directly you will see the value converted accordingly. Example:

➜  ~ kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/datadogmetric@datadog-agent:clustercheck-worker" | jq .
{
  "kind": "ExternalMetricValueList",
  "apiVersion": "external.metrics.k8s.io/v1beta1",
  "metadata": {},
  "items": [
    {
      "metricName": "datadogmetric@datadog-agent:clustercheck-worker",
      "metricLabels": null,
      "timestamp": "2021-06-17T15:26:30Z",
      "value": "39658165378n"
    }
  ]
}

Then the autoscaler client used in the Watermark Pod Autoscaler controller is from kubernetes upstream and converts the value into an int64 as a milli, per here. As you can see, all methods in the interface return an int64. So the above example will be converted to 39658165 and used in the WPA controller as a milli later in the code.

What about percent values like i think it shall be able to use all kind of metrics which have a defined max and a min value like e.g kubernetes.memory.usage_pct or kubernetes.cpu.usage_pct. Do i need to set only the value 90 or 90% ?

I think we retrieve a float (so 0.9) from the backend, which aligns with "Shown as fraction". The cluster agent will convert this to a Quantity (from first receiving/storing the float from the API and then converting it to a quantity), so if queried the cluster agent will return 900m. Which will then become 900 from the upstream podautoscaler client and be converted back to 900m in the WPA controller (as detailed above).

I would therefore recommend using 950m as high watermark to represent 95% for instance (or whatever your watermark might be).

The docker Metrics e.g "Memory" have upper case. Is there generally any case sensitiveness ?

I don't want to misinform you on this one, I don't see upper cases in the officially collected docker metrics, but I might be missing something. I would expect everything to be lowercased but could be wrong (if the above link does not clarify the metrics, I can circle back and confirm internally).

will the both metric measurements be scoped to the container level per pod or do they have different scopes and which?

If you don't scope over a specific container with the docker metric or a specific pod with the kubernetes one, you will end up with an average of what the metric represents. The best practice is to visualise the metric in the metrics explorer and see how the value changes depending on the scope so I make sure I set my watermarks correctly.

Which of these might be the better choice to control the scale

Hard to say without having the full context of your application and infrastructure, what I would say is that docker metrics are direct interfaces of the cgroups whereas the kubernetes ones are interfaced by the CRI which itself interfaces the cgroups.

I hope this clarifies your questions!

[CharlyF]

Hi there - Circling back on this issue. Did my response help?