k0s Prometheus is enabled using annotations by default on most components

Before creating an issue, make sure you've checked the following:

[X] You are running the latest released version of k0s
[X] Make sure you've searched for existing issues, both open and closed
[X] Make sure you've searched for PRs too, a fix might've been merged already
[X] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Not important (rh8)

Version

1.2.26 - but master has it too

Sysinfo

`k0s sysinfo`

➡️ Please replace this text with the output of `k0s sysinfo`. ⬅️

What happened?

prometheus annotations enable scraping for kube-system components (https://github.com/k0sproject/k0s/blob/73a8a11d56c8e0c7535f45c22f9739440c12635e/pkg/component/controller/kubeproxy.go#L312), would be great to make it togglable and deactivable for instances where prometheus is not intended for core components, in particular when it does not work out of the box cause the node ip is not accessible to prometheus for ex.

Steps to reproduce

Start k0s (in single node mode for ex) .

Expected behavior

Check annotations of coredns, kube-proxy and kube-router, should get scrape=false for prometheus default setup

Actual behavior

scrape=true

Screenshots and logs

Additional context

May 24 '23 20:05 rmannibucau

@rmannibucau Just to make sure I got it right, you'd want to have some config toggle to enable/disable k0s setting up prometheus.io/scrape: annotation value?

May 25 '23 10:05 jnummelin

@jnummelin yes, ideally in k0s.yaml or alike.

May 25 '23 12:05 rmannibucau

I've found prometheus.io/scrape: true annotation in these components:

konnectivity
coredns
kubeproxy
kuberouter
pushgateway
calico-node

I propose to add a boolean configuration under each component called prometheusScrapeAnnotation which by default will be true (for backwards compatibility).

Example k0s.yaml:

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  ...
  coredns:
    prometheusScrapeAnnotation: true
  konnectivity:
    ...
    prometheusScrapeAnnotation: true
  network:
    ...
    kubeProxy:
      ...
      prometheusScrapeAnnotation: true
    kuberouter:
      ...
      prometheusScrapeAnnotation: true

WDYT?

Jun 04 '23 23:06 alessiodionisi

Looks perfect to me! Thank you very much.

Jun 05 '23 04:06 rmannibucau

I'm working on it.

Jun 06 '23 09:06 alessiodionisi