Automatically discover and scrape Prometheus targets in Kubernetes

[valyala]

Is your feature request related to a problem? Please describe.

Both vmagent and a single-node VictoriaMetrics support Kubernetes service discovery (aka kubernetes_sd_config). Unfortunately, this type of service discovery isn't trivial to configure. See, for example, a typical configuration for Kubernetes service discovery, which contains hundreds of lines of non-trivial yaml config. There are multiple commonly used Prometheus configs for Kubernetes service discovery and scraping exist over the internet. Unfortunately, they aren't compatible with each other, e.g. they may produce different set of targets, metric names, labels and/or label values. This complicates creating a generic Grafana dashboard for Kubernetes and generic alerting rules for Prometheus.

Describe the solution you'd like

It would be great if vmagent and a single-node VictoriaMetrics could automatically discover scrape targets in Kuberntes and scrape them in a generic way without the need to write and maintain complex yaml configs and relabeling rules. This will make possible providing generic Grafana dashboards for the collected metrics and generic alerting rules for vmalert.

This can be done by adding a -promscrape.scrapeKubernetes command-line flag to vmagent and a single-node VictoriaMetrics, which could accept a list of target types for the scrape. Possible target types are:

nodes
pods
endpoints
services
ingress
all

There should be a documentation for metrics and labels produced by each target type, so users could rely on them when building custom dashboards with alerting and recording rules.

For example, -promscrape.scrapeKubernetes=nodes would scrape metrics from Kubernetes nodes and apply the default relabeling rules to them, while -promscrape.scrapeKubernetes=pods,endpoints would scrape metrics from all the pods and endpoints. A special value = -promscrape.scrapeKubernetes=all would enable scraping for all the available target types in Kubernetes cluster in a generic way.

Describe alternatives you've considered

An alternative is to copy-n-paste hundred-lines of unmaintainable yaml configs found on the internet.