Wiard van Rij

Honestly, your questions are deeper than I'm working with our stack. So I haven't formed an entire opinion on it, other than that I pretty much agree with everything you...

Do you mean this? https://github.com/prometheus/demo-site I'm curious how the Prometheus folks experience these demo sites (Like does it get used a lot for example?)

@tend2infinity will be working on this via LFX! 🥳

> There's an opportunity to collaborate with a workstream launched out of https://github.com/cncf/tag-observability ([WG: observe-k8s](https://github.com/cncf/tag-observability/projects/2)) > > https://github.com/observe-k8s/hub/blob/main/charter.md#vision > > How/where can I learn more about this effort? > >...

> Are the infra and tooling decided yet? There are mentions of Ansible, but IIRC observe-k8s is aiming to use tools that fit k8s use cases better. Yes and no....

Short term we should try to check if receiver response is correct. OutOfBounds is unrecoverable. Do we also tell this back to Prometheus? That is at least to resolve the...

> > Hit this issue quite a lot recently. > > We used to hit it too, but at least since switching to the receive router & ingestor mode we...

What would help me, is a very easy to reproduce example. I understand this is perhaps hard to do but I've already spend some time on this, not even triggering...

Do you monitor prometheus with https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/grafana-dashboardDefinitions.yaml - see prometheus-remote-write.json ? If so, can you show me how it looks like? I will try to reproduce this again tomorrow or so.

I could reproduce this by fiddling around with my local time, causing; ``` level=debug name=receive ts=2022-02-05T22:00:12.323668Z caller=writer.go:94 component=receive component=receive-writer msg="Out of bounds metric" lset="{__name__=\"prometheus_tsdb_head_gc_duration_seconds_sum\", instance=\"localhost:9090\", job=\"prometheus\", prometheus=\"prom0\", replica=\"1\"}" sample="unsupported value...