Move Prometheus scrape config to services

[adriangonz]

When leveraging Prometheus' pod annotations to configure the inference graph's scraping, you are not able to specify multiple ports to scrape. This poses some issues when considering multi-node inference graphs, where each node runs on a separate container, and thus we need to scrape multiple metrics endpoints on a single pod (i.e. one per container).

At the moment, we work around this by tweaking Prom's config and naming the multiple ports to scrape in the pod as metrics. However, this requires adding that extra piece of config, and may break when other tools make assumptions about what needs scraping in a pod (e.g. Istio sidecars metrics merging feature).

A potential solution to the above would be to expose the metrics ports as separate Service resources, and configure the metrics scraping on these services through the "standard" Prometheus annotations. This solution was suggested by @pierluigilenoci on this other thread which is worth checking for further background on this issue.