semantic-conventions [k8s] Define semantic conventions for k8s resource quota metrics

Propose new conventions

Area(s)

area:k8s

Is your change request related to a problem? Please describe.

As part of the K8s SemConv stability work we need to define k8s resource quota related metrics that are already in use by the Opentelemetry Collector.

Related to https://github.com/open-telemetry/semantic-conventions/issues/1032

Describe the solution you'd like

The existing metrics in use that we need to define as semantic conventions are the following:

k8s.resource_quota.hard_limit k8s.resource_quota.used

Apr 09 '25 11:04 ChrsMark

A ResourceQuota object will come with a status like the following:

status:
  hard:
    limits.cpu: "2"
    limits.memory: 2Gi
    requests.cpu: "1"
    requests.memory: 1Gi
  used:
    limits.cpu: 800m
    limits.memory: 800Mi
    requests.cpu: 400m
    requests.memory: 600Mi

(ref)

At https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.123.0/receiver/k8sclusterreceiver/internal/resourcequota/resourcequotas.go#L15-L36 we retrieve all those but instead of emitting 8 different metrics we only emit 2 with 4 attributes each. Example:

Resource SchemaURL: https://opentelemetry.io/schemas/1.18.0
Resource attributes:
     -> k8s.resourcequota.uid: Str(31c6317a-d8d4-48ea-adb0-e35074818623)
     -> k8s.resourcequota.name: Str(mem-cpu-demo)
     -> k8s.namespace.name: Str(default)
ScopeMetrics #0
ScopeMetrics SchemaURL:
InstrumentationScope github.com/open-telemetry/opentelemetry-collector-contrib/receiver/k8sclusterreceiver 0.123.0-dev
Metric #0
Descriptor:
     -> Name: k8s.resource_quota.hard_limit
     -> Description: The upper limit for a particular resource in a specific namespace. Will only be sent if a quota is specified. CPU requests/limits will be sent as millicores
     -> Unit: {resource}
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> resource: Str(limits.cpu)
StartTimestamp: 2025-04-14 09:34:35.770466928 +0000 UTC
Timestamp: 2025-04-14 09:34:45.968422544 +0000 UTC
Value: 2000
NumberDataPoints #1
Data point attributes:
     -> resource: Str(limits.memory)
StartTimestamp: 2025-04-14 09:34:35.770466928 +0000 UTC
Timestamp: 2025-04-14 09:34:45.968422544 +0000 UTC
Value: 2147483648
NumberDataPoints #2
Data point attributes:
     -> resource: Str(requests.cpu)
StartTimestamp: 2025-04-14 09:34:35.770466928 +0000 UTC
Timestamp: 2025-04-14 09:34:45.968422544 +0000 UTC
Value: 1000
NumberDataPoints #3
Data point attributes:
     -> resource: Str(requests.memory)
StartTimestamp: 2025-04-14 09:34:35.770466928 +0000 UTC
Timestamp: 2025-04-14 09:34:45.968422544 +0000 UTC
Value: 1073741824
Metric #1
Descriptor:
     -> Name: k8s.resource_quota.used
     -> Description: The usage for a particular resource in a specific namespace. Will only be sent if a quota is specified. CPU requests/limits will be sent as millicores
     -> Unit: {resource}
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> resource: Str(limits.cpu)
StartTimestamp: 2025-04-14 09:34:35.770466928 +0000 UTC
Timestamp: 2025-04-14 09:34:45.968422544 +0000 UTC
Value: 1000
NumberDataPoints #1
Data point attributes:
     -> resource: Str(limits.memory)
StartTimestamp: 2025-04-14 09:34:35.770466928 +0000 UTC
Timestamp: 2025-04-14 09:34:45.968422544 +0000 UTC
Value: 1073741824
NumberDataPoints #2
Data point attributes:
     -> resource: Str(requests.cpu)
StartTimestamp: 2025-04-14 09:34:35.770466928 +0000 UTC
Timestamp: 2025-04-14 09:34:45.968422544 +0000 UTC
Value: 1000
NumberDataPoints #3
Data point attributes:
     -> resource: Str(requests.memory)
StartTimestamp: 2025-04-14 09:34:35.770466928 +0000 UTC
Timestamp: 2025-04-14 09:34:45.968422544 +0000 UTC
Value: 1073741824

This is quite weird and does not seem to follow the SemConv guidelines. Shouldn't we have 8 different metrics instead?

i.e.:

k8s.resource_quota.hard.cpu_limit
k8s.resource_quota.hard.cpu_request
k8s.resource_quota.hard.memory_limit
k8s.resource_quota.hard.memory_request

k8s.resource_quota.used.cpu_limit
k8s.resource_quota.used.cpu_request
k8s.resource_quota.used.memory_limit
k8s.resource_quota.used.memory_request

I'm not 100% sure if hard and used should be namespaces or not. @open-telemetry/semconv-k8s-approvers I'd love your feedback here.

Apr 14 '25 09:04 ChrsMark

Resource quotas aren't limited to cpu and memory. It can be set for storage, gpu, object counts (k8s native and custom resources), etc. Here's an example -

apiVersion: v1
kind: ResourceQuota
metadata:
  creationTimestamp: "2024-04-03T19:16:22Z"
  name: default-resource-quotas
  namespace: istio-ingress
  resourceVersion: "242026"
  uid: <uid>
spec:
  hard:
    count/gateways.networking.istio.io: "20"
    count/targetgroupbindings.elbv2.k8s.aws: "50"
    services.loadbalancers: "10"
    services.nodeports: "30"
status:
  hard:
    count/gateways.networking.istio.io: "20"
    count/targetgroupbindings.elbv2.k8s.aws: "50"
    services.loadbalancers: "10"
    services.nodeports: "30"
  used:
    count/gateways.networking.istio.io: "10"
    count/targetgroupbindings.elbv2.k8s.aws: "17"
    services.loadbalancers: "6"
    services.nodeports: "15"

Given the wide range of possible resource types, the existing pattern of using an attribute to indicate the resource type seems more scalable, and flexible to me.

Apr 16 '25 14:04 jinja2

Thank's @jinja2 for the note, I guess that might was the idea for it being implemented like this in the Collector.

However, I'm not sure this design can be covered in SemConv. Can the same metric be of different type/unit based on its attributes? For instance, memory quotas will be measured in Bytes while cpu ones in cores, right?

I wonder if scoping each resource under its namespace makes more sense here, potentially starting from the most common ones?

Apr 17 '25 13:04 ChrsMark

However, I'm not sure this design can be covered in SemConv. Can the same metric be of different type/unit based on its attributes? For instance, memory quotas will be measured in Bytes while cpu ones in cores, right?

Good point — in that case, the resource should probably be part of the metric name (or namespace) and define the unit accordingly.

I'm not 100% sure if hard and used should be namespaces or not.

Differently than the resource, the resource quota hard or used status do not influence the metric type/unit. Making them part of an attribute will reduce by half the amount of metrics, which feels like a cleaner abstraction (similar to how system.io.direction is used).

k8s.resource_quota.cpu_limit  (k8s.resource_quota.status = {used/hard})
k8s.resource_quota.cpu_request (k8s.resource_quota.status = {used/hard})
k8s.resource_quota.memory_limit (k8s.resource_quota.status = {used/hard})
k8s.resource_quota.memory_request (k8s.resource_quota.status = {used/hard})

Apr 22 '25 10:04 rogercoll

I couldn't help noticing k8s.resource_quota.* spelled out with an _ throughout this issue. Any chance you all changed your mind about naming these 😉? or is it just due to the implementation not catching up with semconv?

Apr 22 '25 19:04 trask

I couldn't help noticing k8s.resource_quota.* spelled out with an _ throughout this issue. Any chance you all changed your mind about naming these 😉? or is it just due to the implementation not catching up with semconv?

I think k8s.resource_quota.* is just coming from the Collector implementation. I guess based on what was already decided the correct one would be resourcequota :)

Apr 24 '25 13:04 ChrsMark

We had a chat with @rogercoll about this during the K8s SemConv SIG meeting last week. It seems that the existing implementation is indeed convenient since it can cover well the wide range of possible resources which can be used as ResourceQuota targets. The main idea came up was to find a way to group ResourceQuota metrics per type.

I also looked into how the area can be better defined and scoped from K8s side based on https://kubernetes.io/docs/concepts/policy/resource-quotas/ and https://github.com/kubernetes/kubernetes/tree/v1.33.0/pkg/quota/v1/evaluator/core.

A possible solution here that would by-pass the type and unit issue described at https://github.com/open-telemetry/semantic-conventions/issues/2076#issuecomment-2812936919 is to create different metrics for cpu, memory, storage etc resources that are gauges and have their own type and one updowncounter that would cover the resource quota targets that are of type count/ or are counters by definition like the services.loadbalancers. This would also be feasible to be implemented in the Collector since we should check for specific resources or the count/* pattern and decide accordingly which metric to emit.

The metrics can roughly by listed as below:

# TYPE gauge, UNIT "cores"
k8s.resourcequota.cpu.limit  (k8s.resourcequota.status = {used/hard})
k8s.resourcequota.cpu.request (k8s.resourcequota.status = {used/hard})

# TYPE gauge, UNIT "By"
k8s.resourcequota.memory.limit (k8s.resourcequota.status = {used/hard})
k8s.resourcequota.memory.request (k8s.resourcequota.status = {used/hard})

# TYPE gauge, UNIT "By"
k8s.resourcequota.hugepages(k8s.resourcequota.status = {used/hard}, k8s.resourcequota.hugepages.size = {<size>})

# TYPE gauge, UNIT "By"
k8s.resourcequota.storage.request (k8s.resourcequota.status = {used/hard}, k8s.resourcequota.storage.class_name = {<storage-class-name>})

# TYPE updowncounter, UNIT "1"
k8s.resourcequota.resource_count(k8s.resourcequota.status = {used/hard}, k8s.resourcequota.resource_type= {<resource_type>})
k8s.resourcequota.resource_count(k8s.resourcequota.status = {used/hard}, k8s.resourcequota.resource_type= {<resource_type>})

@jinja2 would that make sense?

May 05 '25 11:05 ChrsMark

@ChrsMark The new proposed metrics seem to cover most types of quotas. However, I have concerns about the user experience with this change from using an attribute to indicate the resource to naming separate metrics for each. With the existing style of resourcequota metric, a user can make generic queries like sum(k8s.resourcequota.used{resource=~".+"}) by (namespace, resource). Imo, this makes it easier to display all resource quotas with a single chart/query, as users don't need to have prior knowledge or look up which resources have quotas.

I'm not opposed to the new proposal and understand that it aligns more closely with existing conventions. I just want to highlight that this may add a bit more complexity for users when creating visualizations.

I have a similar dilemma in this issue. HPA metrics mentioned in the issue could be about cpu, memory or other random resource (or just any random metric tracked by hpa). I anticipate encountering more such use cases in k8s.

May 14 '25 05:05 jinja2

Had a discussion about this in the semantic conventions meeting and we agreed separate metrics won't take away much from the query-side experience. The proposal to namespace metrics by resource type sounds good!

May 14 '25 15:05 jinja2