prometheus.exporter.cadvisor not adding container labels to metrics

[roshini-cp]

Hello,

I am experiencing an issue where container labels are not being added to the metrics from prometheus.exporter.cadvisor.

Configuration:

prometheus.exporter.cadvisor "cadvisor" {
    storage_duration = "5m"
    store_container_labels = false
    allowlisted_container_labels = ["io.kubernetes.container.name"]
}

Issue:

Despite configuring allowlisted_container_labels, the container labels are not included in the exported metrics. I am unsure if this is due to a configuration mistake or if I am missing something in the setup.

Steps to Reproduce:

Set up prometheus.exporter.cadvisor with the above configuration in grafana-alloy.

Observe that the exported metrics do not include the io.kubernetes.container.name label. Expected Behavior:

The exported metrics should include the io.kubernetes.container.name label as specified in the allowlisted_container_labels.

Actual Behavior:

The container labels are not present in the exported metrics.

Environment:

Please let me know if there is a correction needed in the configuration or if additional steps are required to ensure that container labels are included in the metrics. Any help or guidance on this matter would be greatly appreciated.

Have tried the recommendations from here https://github.com/grafana/alloy/issues/830 it is not helping with my requirements

Thank you!

[github-actions[bot]]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

[jannst]

Hello, I also spend quite a time with this issue. In my case, I missed to add some folder mounts to my Alloy Docker container, that were needed by cAdvisor in order to get information about container metadata and labels.

These mounts are documented in the cAdvisor documentation, but a hint in the Alloy docs would surely be helpful. https://github.com/google/cadvisor/blob/master/docs/running.md

In the end, the following compose configuration for my Alloy container worked and labels were shown:

services:
  alloy:
    image: grafana/alloy:v1.5.1
    hostname: monitor-alloy
    restart: unless-stopped
    command: run /etc/alloy/config
    volumes:
      - "/:/rootfs:ro"
      - "/var/run:/var/run:ro"
      - "/sys:/sys:ro"
      - "/var/lib/docker/:/var/lib/docker:ro"
      - /var/run/docker.sock:/var/run/docker.sock
      - ./files/config.alloy:/etc/alloy/config/config.alloy

Alloy config snippet:

prometheus.exporter.cadvisor "local_system" {
    store_container_labels = true
    docker_only = true
    enabled_metrics = ["cpu", "memory", "network"]
}

[Liquidmasl]

I still havent found out how to successfully whitelist container labels... have you been successful?

[mcapodici]

See also https://github.com/grafana/alloy/issues/990

[chris-barbour-as]

Is your kubernetes environment running on ContainerD by any chance? There is/was an issue where Kubernetes was only applying pod labels to the pause image.

https://github.com/google/cadvisor/issues/3652

[laserpedro]

Does anyone know how to do this via the official helm chart of Grafana alloy ?