Filip Petkovski

Ah, looking at my comment, I haven't linked the actual example: https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/prometheus-to-sd/kubernetes/prometheus-to-sd-kube-state-metrics.yaml This leads me to believe that either it is somehow possible to combine the two metric endpoints, or...

I see, I believe they use to process_start_time_seconds to detect restarts for counters (aka cumulative metrics), something that is already built into Prometheus. IMO adding this metric makes sense, but...

> 1. It seems possible to specify multiple endpoints in the configuration of prometheus-to-sd (https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/prometheus-to-sd/kubernetes/prometheus-to-sd-kube-state-metrics.yaml#L10-L12) I think this is just the container port for the KSM pod. The source seems...

Are you sure that the price is per MB scraped instead of MB stored? Prometheus metrics use far less resources on disk (~1.6B per sample) compared to the text exposition...

What if we add a flag to export the telemetry metrics on the same port, but also allow opting out of those metrics using the `--metrics-denylist` flag? That should satisfy...

One way to do it is to decorate the gatherer and filter out the metric families it returns. Here's a POC which can be used as a starting point: https://github.com/kubernetes/kube-state-metrics/pull/1666/files

@WesCossick if would you be interested in contributing this feature? You can use that PR as an example of how telemetry metrics can be filtered with existing flags.

Hey :wave: thanks for reporting this. Do you know for which ingress this happens? Do you have any ingresses without a backend or a service backend? Looking at the code,...

Thanks for checking that. I agree that this is still a bug and we should keep the issue open until it is resolved :+1:

I am traveling currently, should be able to look at the PR next week.