Self managed: Push external metrics to prometheus

[robinAwallace]

Is your feature request related to a problem? Please describe.

For developers that have external services that want to push metrics to Prometheus. Can not at the moment do this. As we do not expose Prometheus for external access.

Describe the solution you'd like

There are multiple solutions that needs to be investigated.

1. Open up Prometheus/Thanos in the service cluster. For this we need to add basic auth. Also if we give access to thanos, is it possible for the developer to delete metrics? This should not be possible.
2. Open up Prometheus in the workload cluster. For this we need to add basic auth.
3. Add a new Prometheus in the workload cluster. This to separate internal and external metrics. To ensure that external metrics can not tank Prometheus for internal metrics. Which could be en issue for alerting.

For all of these options add the needed information in a secret that the developer can access.

Describe alternatives you've considered

Not allowing external metrics pushed into the environment.

Additional context

Definition of done:

It is possible for developers to send metrics from external services

[OlleLarsson]

If they for whatever reason (networking or similar) can't configure scraping of external endpoints via ServiceMonitors and instead want to push metrics to prometheus, shouldn't they simply be able to run a prometheus pushgateway in the cluster which they then can configure prometheus to scrape?

[robinAwallace]

That is true, as long as they can start up a pushgateway. But I do not see that there should be a issue. This can maybe be closed then :+1: