metrics: move prometheusrules under metricServer

[erikkn]

Hi, folks!

I don't mean to be pedantic, but the keda-operator-metrics-apiserver app is exposing these (https://keda.sh/docs/2.8/operate/prometheus/) metrics. At the moment we create Prometheus rules under the prometheus.operator directive which is subsequently used in keda/templates/15-keda-prometheusrules.yaml. The name of the Prometheus rule is also {{ .Values.operator.name }}.

This all is a bit confusing to me and frankly, doesn't make sense, since it belongs to the metricServer section.

Would like to her you guys opinion on this :)

Checklist

• [ x] Commits are signed with Developer Certificate of Origin (DCO - learn more)
• [ ] A PR is opened to update KEDA core (repo) (if applicable, ie. when deployment manifests are modified)
• [ x] README is updated with new configuration values (if applicable)

Fixes #

[erikkn]

@tomkerkhove decided to just tag you on this since you also reviewed my last PR. I would love to hear your opinion on this; thanks!

[erikkn]

I do recognize that this is a breaking change, you want to support both for now?

[tomkerkhove]

I think it's best to do, yes

[tomkerkhove]

We actually already have metrics on the operator so we should make sure both are still accessible in a backwards compatible way - https://github.com/kedacore/keda/issues/3586

So instead of removing the operator pieces, I'd just keep them in addition to your new additions.

@JorTurFer as FYI

[v-shenoy]

I think having metrics on both pods is the way, as it makes sense to record certain metrics on particular pods. So, adding configuration for metric server while keeping the operator would be great.

[tomkerkhove]

What is the status on this PR? This might be a bit outdated due to our recent Prometheus changes