David Ashpole

/remove-sig instrumentation

Are you able to tell which collector is scraping which node at any point in time (e.g. by adding a constant label from each collector)? It would be helpful to...

If the up metric is missing during the window then that almost certainly indicates that the problem is happening after the Prometheus receiver. It emits up=0 when a scrape fails,...

Have you set up the collector self-observability metrics (see https://opentelemetry.io/docs/collector/internal-telemetry/)? That should have metrics from all of the components that can tell you if any of them are dropping points.

Yeah, the prometheus receiver doesn't use the common scraper library. It uses the Promethues server's scraping logic

looks like this needs some go mod updates. The diff currently downgrades otel dependences

``` --- a/receiver/prometheusremotewritereceiver/go.mod +++ b/receiver/prometheusremotewritereceiver/go.mod @@ -77,7 +77,7 @@ require ( go.opentelemetry.io/collector/pdata/pprofile v0.112.0 // indirect go.opentelemetry.io/collector/pipeline v0.112.0 // indirect go.opentelemetry.io/collector/receiver/receiverprofiles v0.112.0 // indirect - go.opentelemetry.io/collector/semconv v0.105.0 // indirect + go.opentelemetry.io/collector/semconv...

From https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/1209-metrics-stability/kubernetes-control-plane-metrics-stability.md#stability-classes: > From an ingestion point of view, it is backwards-compatible to add or remove possible values for labels which already do exist (but not labels themselves). Therefore, adding...

second commit looks good. I'll approve once it is rebased on the http server PR.

I think this is ready to be merged