`loki.source.kubernetes` instance label conflicts with self defined instance labels

[Skaronator]

What's wrong?

I'm using the loki.source.kubernetes component, and I have issues trying to replicate the original promtail helm-chart config.

The promtail config contains an "instance" label that is set when the pod has an "instance" label, as you can see here: https://github.com/grafana/helm-charts/blob/1338c741e3ba6fa2c7b17f20c4aced673fa74943/charts/promtail/values.yaml#L509-L514

Using the same config causes conflicts with Grafana Agent. The "instance" label doesn't always exist on every pod, and therefore this label is usually not set/defined.

The issue is that Grafana Agent always defines this when it’s missing, as you can see here: https://github.com/grafana/agent/blob/a7153e4bb940f4a1d65e01b173a24e236d15c043/internal/component/loki/source/kubernetes/kubetail/target.go#L222-L226

I'd expect to have this configurable. Allow user to disable this and allow user to change the name of that label.

Steps to reproduce

discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets

  // either use this rule to see my exact issue
  rule {
    source_labels = [
      "__meta_kubernetes_pod_label_app_kubernetes_io_instance",
      "__meta_kubernetes_pod_label_instance",
    ]
    regex         = "^;*([^;]+)(;.*)?$"
    target_label  = "instance"
  }

  // or run this rule and you'll see that "instance" label still exists
  rule {
    regex  = "instance"
    action = "labeldrop"
  }
}

loki.source.kubernetes "pods" {
  targets    = discovery.relabel.pods.output
  forward_to = [loki.write.default.receiver]
}

Running with this config, you'll see that you have instance labels that are not Pod Labels.

System information

No response

Software version

v0.43.3

Configuration

No response

Logs

No response

[rfratto]

Hi there :wave:

On April 9, 2024, Grafana Labs announced Grafana Alloy, the spirital successor to Grafana Agent and the final form of Grafana Agent flow mode. As a result, Grafana Agent has been deprecated and will only be receiving bug and security fixes until its end-of-life around November 1, 2025.

To make things easier for maintainers, we're in the process of migrating all issues tagged variant/flow to the Grafana Alloy repository to have a single home for tracking issues. This issue is likely something we'll want to address in both Grafana Alloy and Grafana Agent, so just because it's being moved doesn't mean we won't address the issue in Grafana Agent :)

[github-actions[bot]]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

[rfratto]

Hi, I'm a little confused about what the specific problem is here in terms of this being a bug. This is the behaviour I currently expect:

• If an instance label is not found on a target, one will be automatically provided.
• If an instance label is found on a target, it should take precedence over the automatic one.
• If an instance label is completely undesired, it can be dropped later on in the pipeline after log collection (i.e., via loki.relabel)

Is the issue you're describing in one of these three behaviours? Or are you suggesting that one of these behaviours should be changed?

(I'm really sorry about how long this has been sitting around without a response)

[Skaronator]

Hi, yeah, this issue can be closed.

Initially, my plan was to disable the agent instance label and apply it only if a Kubernetes pod has the instance label. I prefer not to mix agent instance values with Kubernetes label values.

I believe it's feasible to assign the Kubernetes instance label value to a __tmp_instance_label during the discovery stage. Then, using loki.relabel, we can drop the agent instance label and set the "instance" label from the __tmp_instance_label.

I did not try that. I'm now using "k8s_instance" for my kubernetes labels and instance is left default.

Is it beneficial to provide the "instance" label to loki if you're never going to use it anyway?