thanos icon indicating copy to clipboard operation
thanos copied to clipboard

sidecar: Add `/api/v1/flush` endpoint

Open nashluffy opened this issue 1 year ago • 1 comments

  • [x] I added CHANGELOG entry for this change.
  • [ ] Change is not relevant to the end user.

Changes

Adds a sidecar API with one endpoint: /api/v1/flush which calls the TSDB snapshot endpoint on the prometheus instance, then uploads all not-already-present blocks in the snapshot to object store.

There are a few issues that explain the motivation:

  • https://github.com/thanos-io/thanos/issues/7295
  • https://github.com/prometheus-operator/prometheus-operator/issues/6540

Essentially if this is the last time sidecar will be running (ie. cluster is being deleted, shard being removed, etc...) then without some flushing mechanism you will permanently lose up to 2 hours of data.

Verification

Beside the unit tests, running prometheus locally and calling the endpoint works as expected.

image

nashluffy avatar May 14 '24 16:05 nashluffy

If we want a flush api we probably should add it to shipper so that ruler and receiver also get it, right?

MichaHoffmann avatar May 19 '24 08:05 MichaHoffmann