Stan Kwong

> Do they expose timestamps on metrics? @roidelapluie Nope

hey @roidelapluie just wanted to check in and see if there's any other debugging information we could provide you with! :)

1. Here is what the **scrape durations** (seconds) look like during this timeframe (note these timestamps (UTC) look different than the original (PST)). It seems to average around 1s across...

Hope this is okay -- 

Ah my mistake - here you go. This shows `up{job=~"statsd_.*|prometheus.*"}[10m]` at 16:05 (UTC), or 9:05 (PST) https://gist.github.com/jpdstan/97a5b9098c4ddd06440cbff5b819badc

@JensErat thanks for noticing! I wonder if that issue applies to us - our Prometheus stack doesn't do any federation. I'd also suspect larger gaps that parallel the 15 minute...

@roidelapluie sorry I just got a chance to get to this! these are goroutine dumps from this morning when we saw another gap from around **11:00 to 11:04**. **11:01 dump**:...

The gaps still appear to be happening right on the dot every 2 hours at every odd hour PST (1:00, 3:00, etc). Though today we also noticed a smaller gap...

Also something we just noted is that these gaps correlate with drops in `prometheus_remote_storage_samples_in_total` as well (this shows `rate(prometheus_remote_storage_samples_in_total{host=~''}[5m])`: 

Robinhood is using Vector in many ways! - EC2 application logs -> kafka (replaced filebeat) - Kubernetes pods logs -> kafka (replaced fluentd) - Kafka -> Loki We've had a...