avalanche icon indicating copy to clipboard operation
avalanche copied to clipboard

Avalanche doesn't send remote-write requests when --metric-interval alone is set

Open harry671003 opened this issue 1 year ago • 3 comments

Issue

  • Thanos uses avalanche in e2e tests. About two weeks ago, one of the e2e tests which uses avalanche started failing.
  • The test was configuring --metric-interval=30s and --remote-write-interval=30s and expecting requests to be made every 30s from avalanche.
  • However, in write.go#L143 no timeseries is returned and avalanche doesn't make write requests.

Avalanche configuration

Avalanche configuration that wasn't working

--metric-count=10 \
--series-count=1 \
--metric-interval=30 \
--series-interval=3600 \
--value-interval=3600 \
--remote-write-interval=30s \
--remote-batch-size=10 \
--remote-requests-count=5

Avalanche configuration that worked

--metric-count=10 \
--series-count=1 \
--metric-interval=3600 \
--series-interval=30 \
--value-interval=30 \
--remote-write-interval=30s \
--remote-batch-size=10 \
--remote-requests-count=5

Related

https://github.com/thanos-io/thanos/pull/7620

harry671003 avatar Aug 12 '24 17:08 harry671003

hey @harry671003, can you share more details about what you're seeing?

I see samples being sent correctly when testing locally with both sets of CLI arguments you've provided.

cstyan avatar Aug 23 '24 18:08 cstyan

We changed a bit of code since you last check. Do you mind double checking if this is still the case?

bwplotka avatar Jan 15 '25 07:01 bwplotka

It still seems to be a problem in 0.7.0.

harry671003 avatar Jan 15 '25 23:01 harry671003