feat(docs): use job=cadvisor label consistently

[flokli]

This ensures cadvisor metrics have the job=cadvisor label set, which is used in many Grafana Kubernetes Dashboards, Alerts and Recording rules, and required to make them work.

Compatibility with Grafana Cloud's Kubernetes integration is dropped, which was already behind a big warning sign.

In some cases, the labels were also set using metricRelabelings, which only affected metrics themselves, and not the up metric. Consistently use relabelings everywhere.

Fixes #1606.

PR Description

Which issue(s) this PR fixes

Fixes #1606.

Notes to the Reviewer

production/tanka/grafana-agent/v2/internal/helpers/k8s.libsonnet might need an update too.

PR Checklist

• [x] CHANGELOG updated
• [x] Documentation added
• [ ] Tests updated

[hjet]

hey there @flokli thanks so much for contributing this!

all the changes lgtm, but for custom-resource-quickstart.md i'd like to preserve compatibility with the cloud k8s integration (so keep the labels as integrations/kubernetes/*) as we point our users and engineers there at the moment. this will soon change (in the next month or so), at which point we can make the job labels totally compatible with the OSS mixin. i would maybe leave the defaults as integrations/kubernetes/* with a note to omit /integrations/kubernetes/ if needed for OSS use. see below suggestions!

wdyt?

[flokli]

all the changes lgtm, but for custom-resource-quickstart.md i'd like to preserve compatibility with the cloud k8s integration (so keep the labels as integrations/kubernetes/*) as we point our users and engineers there at the moment.

The main idea for this PR was to fix compatibility with the dashboards and alerts out there. Reading the docs, I assumed the Grafana Cloud Kubernetes integration was somewhat expected to break eventually, until it aligns with the OSS label names.

The suggested changes don't make things less confusing. If you want to avoid having a time the quickstart here doesn't work with the Grafana Cloud Integration, I'd rather prefer to keep this open and merge it the same time the Grafana Cloud Integration switched to the new labels.

[flokli]

@hjet

[…] this will soon change (in the next month or so), at which point we can make the job labels totally compatible with the OSS mixin […]

Is there an update on that? Can we merge this in soonish?

[hjet]

@flokli these docs were all hand-written and will soon be updated with K8s yaml generated from jsonnet (PR here)

the job labels are top level parameters so we will set them to the OSS defaults in the OSS docs

[flokli]

@hjet with https://github.com/grafana/agent/pull/1921 merged, what's the state of this?

I saw that PR didn't touch the markdown files - has there been a followup PR? Should this be rebased?

[hjet]

@hjet with #1921 merged, what's the state of this?

I saw that PR didn't touch the markdown files - has there been a followup PR? Should this be rebased?

hey there @flokli,

• https://github.com/grafana/agent/pull/2070 has been merged which fixes the original issue (https://github.com/grafana/agent/issues/1606) - the issue being broken/duplicate up metrics
• at this time, we would like to keep integrations/kubernetes/ in the default job labels in this custom resource quickstart guide, until we deliver these operator manifests to users directly in grafana cloud (which will happen very soon)
• there is a more recent set of example manifests that can be found here, which includes kube-state-metrics, the k8s events integration, and some other updates
  • we need to fix the same relabelings/metricsrelabelings issue in this set of example manifests
  • we can set the default job labels to cadvisor and kubelet here and remove integrations/kubernetes
  • ideally this should be split into multiple manifests, one for each component, and then the custom resource quickstart can reference these manifests directly, instead of having them hard coded in the doc. the custom resource quickstart also is in desperate need of an update...
  • PRs welcome for the above 3 bullets, as i likely won't have time to tackle this in the immediate future

[hjet]

update: i set the job labels to the OSS defaults in the new sample manifests, and fixed the relabel configs (see https://github.com/grafana/agent/pull/2073)

[github-actions[bot]]

This PR has been automatically marked as stale because it has not had any activity in the past 30 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed in 7 days if there is no new activity. Thank you for your contributions!

[rfratto]

IIUC, this is superseded by #2073, closing. If I'm wrong feel free to reopen :)