Prometheus ServiceMonitors only work with MCAC

[burmanm]

What happened? While trying the feature, setting

telemetry:
  prometheus:
    enabled: true

when using Cassandra 5.0.x, the generated ServiceMonitors are incorrect. They refer to certain labels that are not found on the target pods and the Prometheus rewrites it generates are wrong (and only apply to the MCAC).

Also, the targetPort is wrong.

Did you expect to see something different? The correct approach would look something like:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/component: telemetry
    app.kubernetes.io/managed-by: k8ssandra-operator
    app.kubernetes.io/part-of: k8ssandra
    k8ssandra.io/cluster-name: test
    k8ssandra.io/cluster-namespace: k8ssandra-operator
    k8ssandra.io/datacenter: dc1
  name: test-dc1-cass
  namespace: k8ssandra-operator
spec:
  endpoints:
  - bearerTokenSecret:
      key: ""
      name: ""
    interval: 15s
    path: /metrics
    targetPort: 9000
    scheme: http
    scrapeTimeout: 15s
  namespaceSelector:
    matchNames:
    - k8ssandra-operator
  selector:
    matchExpressions:
    - key: cassandra.datastax.com/cluster
      operator: In
      values:
      - test
    matchLabels:
      app.kubernetes.io/managed-by: cass-operator
  targetLabels:
  - cassandra.datastax.com/cluster
  - cassandra.datastax.com/datacenter

There's no need for additional rewrite rules and the port must be set to 9000. Also, remove additional label filters that do not apply to our pods.

How to reproduce it (as minimally and precisely as possible):

Environment

• K8ssandra Operator version:

  Insert image tag or Git SHA here

• Kubernetes version information:

  kubectl version

• Kubernetes cluster kind:

  insert how you created your cluster: kops, bootkube, etc.

• Manifests:

insert manifests relevant to the issue

• K8ssandra Operator Logs:

insert K8ssandra Operator logs relevant to the issue here

Anything else we need to know?: