Using Integrations with the Grafana Agent Operator helm chart

[duncanwraight]

Hi folks

We're using the Grafana Operator Helm chart and we've had great success exporting application metrics to Amazon Prometheus. We've managed to export kubelet, cAdvisor and various custom applications' metrics to the AWS-managed Prometheus workspaces.

The last piece of the puzzle is that we're trying to create an Integration for exporting metrics from our Kubernetes nodes. We have cunningly called this node-exporter-integration, and its manifest looks like this:

apiVersion: v1
items:
- apiVersion: monitoring.grafana.com/v1alpha1
  kind: Integration
  metadata:
    annotations:
      meta.helm.sh/release-name: grafana-agent
      meta.helm.sh/release-namespace: grafana-agent
    labels:
      agent: grafana-agent
      app.kubernetes.io/managed-by: Helm
      metricsSelectorServiceLabel: node-exporter
    name: node-exporter-integration
    namespace: grafana-agent
  spec:
    config:
      autoscrape:
        enable: true
        metrics_instance: grafana-agent/node-exporter-metrics
      procfs_path: host/proc
      rootfs_path: /host/root
      sysfs_path: /host/sys
    name: node-exporter
    type:
      allNodes: true
      unique: true
    volumeMounts:
    - mountPath: /host/root
      name: rootfs
    - mountPath: /host/sys
      name: sysfs
    - mountPath: /host/proc
      name: procfs
    volumes:
    - hostPath:
        path: /
      name: rootfs
    - hostPath:
        path: /sys
      name: sysfs
    - hostPath:
        path: /proc
      name: procfs
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

We're not seeing any node* metrics in our Prometheus workspace, and I'm curious as to how these integrations are supposed to work. The API schema for the config block appears to be missing from the Integration CRD.

Our other applications are using a GrafanaAgent, which selects a MetricsInstance resource, which then selects a ServiceMonitor and this looks for an actual Service with specified labels.

In each of these resources, the parent has a selector to find the child, e.g. GrafanaAgent has spec.metrics.instanceSelector and MetricsInstance has spec.serviceMonitorSelector.

Conversely, the MetricsInstance has no selector an Integration child, but the Integration resource has a spec.config.autoscrape.metrics_instance value which is apparently in format namespace/metrics-instance-name.

We think we've done this correctly, there is definitely a MetricsInstance resource with name node-exporter-metrics in the grafana-agent namespace:

❯ kubectl get -n grafana-agent MetricsInstance node-exporter-metrics
NAME                    AGE
node-exporter-metrics   42m

... but yet we still see no metrics.

Can anyone enlighten us on how to use these Integrations? Is it possible we're missing an actual Deployment and Service to go with this Integration?

Are they're even the right thing to use at all? All we want to do is pull out node metrics from EKS.

Thanks in advance

[vmax]

@duncanwraight have you tried configuring spec.integrations.selector in GrafanaAgent resource such that it matches the labels set on Integration resource? kubectl explain grafanaagent.spec.integrations contains more details on what arguments it expects