VMAgent global relabelling cannot access SD labels

[predakanga]

Usecase

I'm trying to use VMAgent's inlineRelabelConfig to add the node name to all metrics.

Problem

Because the relabelling is implemented using -remoteWrite.relabelConfig, it runs after all of the service discovery labels have been discarded, leaving the label empty.

Configuration

Kubernetes v1.23.3, self-hosted VM-Operator v0.28.5 VMAgent resource

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  name: demo
  namespace: monitoring
spec:
  serviceScrapeNamespaceSelector: {}
  podScrapeNamespaceSelector: {}
  podScrapeSelector: {}
  serviceScrapeSelector: {}
  nodeScrapeSelector: {}
  nodeScrapeNamespaceSelector: {}
  staticScrapeSelector: {}
  staticScrapeNamespaceSelector: {}
  replicaCount: 1
  remoteWrite:
    - url: "http://vmsingle-demo.monitoring.svc.cluster.local:8429/api/v1/write"
  resources:
    limits:
      memory: 2Gi
    requests:
      memory: 2Gi
  inlineRelabelConfig:
    - action: replace
      regex: "([^;]+);?.*"
      replacement: "$1"
      separator: ";"
      source_labels: [__meta_kubernetes_node_name, __meta_kubernetes_pod_node_name, __meta_kubernetes_endpoint_node_name]
      target_label: node_name
    - source_labels: [__meta_kubernetes_node_name, __meta_kubernetes_pod_node_name, __meta_kubernetes_endpoint_node_name]
      target_label: combined_node_names
      separator: ";"

Expected results

All metrics scraped by this agent should be written to VictoriaMetrics with the node_name label set to the node they were scraped from.

Actual results

Metrics are all missing the node_name label (because the regex doesn't match), and the combined_node_names label is set to ";;", demonstrating that the service discovery labels are not present.

Comments

I was confused by this behaviour because of the feature being called "global relabelling" in the documentation. I had to read through the operator source to figure out what was actually going on.

My expectation was that all of the scrape_configs would be modified, appending the extra relabel config. This would allow them to access the service discovery and any other internal labels.

[f41gh7]

Hello, relabeling is really complicated part of vmagent. inlineRelabeling is used as global relabeling for vmagent, it doesn't have access to SD metadata labels and only applied to ingested data via push protocols. Documentation must be improved for this case, thanks for pointing.

I think, nodeScrapeRelabelTemplate or serviceScrapeRelabelTemplate will work for your case. It will add defined relabeling rules to the each CRD object. You could define rules for one of options or for both depends on how metrics collection configured.

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  name: demo
  namespace: monitoring
spec:
  serviceScrapeNamespaceSelector: {}
  podScrapeNamespaceSelector: {}
  podScrapeSelector: {}
  serviceScrapeSelector: {}
  nodeScrapeSelector: {}
  nodeScrapeNamespaceSelector: {}
  staticScrapeSelector: {}
  staticScrapeNamespaceSelector: {}
  replicaCount: 1
  remoteWrite:
    - url: "http://vmsingle-demo.monitoring.svc.cluster.local:8429/api/v1/write"
  resources:
    limits:
      memory: 2Gi
    requests:
      memory: 2Gi
  nodeScrapeRelabelTemplate:
    - action: replace
      regex: "([^;]+);?.*"
      replacement: "$1"
      separator: ";"
      source_labels: [__meta_kubernetes_node_name, __meta_kubernetes_pod_node_name, __meta_kubernetes_endpoint_node_name]
      target_label: node_name
    - source_labels: [__meta_kubernetes_node_name, __meta_kubernetes_pod_node_name, __meta_kubernetes_endpoint_node_name]
      target_label: combined_node_names
      separator: ";"