Service orchestrator does not set Prometheus metrics port annotation.

[jrouly]

https://github.com/SeldonIO/seldon-core/blob/1748a22f36d34843e2a3c40b14e45580c7f841fe/operator/controllers/seldondeployment_engine.go#L405

The Seldon service orchestrator pod is not created with the Prometheus metrics port annotation (prometheus.io/port) which leads to the Prometheus pod scraper trying to scrape the gRPC port and failing with noisy warnings.

2023-09-18T16:39:03.293Z    warn    internal/transaction.go:125    Failed to scrape Prometheus endpoint    {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_timestamp": 1695055143288, "target_labels": "{__name__=\"up\", app=\"jrouly-jrouly-svc-orch\", instance=\"10.41.122.234:5001\", job=\"kubernetes-pods\", kubernetes_namespace=\"seldon\", kubernetes_pod_name=\"jrouly-jrouly-svc-orch-77c74bb667-92h6b\", pod_template_hash=\"77c74bb667\", seldon_app=\"jrouly-jrouly\", seldon_deployment_id=\"jrouly-jrouly\", version=\"jrouly\"}"}
2023-09-18T16:39:18.293Z    debug    scrape/scrape.go:1353    Scrape failed    {"kind": "receiver", "name": "prometheus", "data_type": "metrics", "scrape_pool": "kubernetes-pods", "target": "http://10.41.122.234:5001/prometheus", "error": "Get \"http://10.41.122.234:5001/prometheus\": net/http: HTTP/1.x transport connection broken: malformed HTTP response \"\\x00\\x00\\x06\\x04\\x00\\x00\\x00\\x00\\x00\\x00\\x05\\x00\\x00@\\x00\""}

Here is the header of the service orchestrator pod.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    prometheus.io/path: /prometheus
    prometheus.io/scrape: "true"

I can't find any way to override this behavior in the Seldon controller.

I fully suspect (and have yet to confirm) that metrics are being collected on the correct port, but these warnings are definitely occurring on the wrong (gRPC) port.