grafana
Docs: cluster migration guide
Hi, we want to migrate our infrastructure to a new k8s cluster and bucket, but we can't find a correct way to do it. The main issues are:
- how can we flush the old ingester
- how can we copy the data from the old bucket to the new without compromising it
We tried to do it by simply switching the remote writes to the new cluster and copying the old data from the origin bucket to the new but we get the following error in grafana:
Validation failed the consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed). The non-queried blocks are: 01GNNG744NV8CA8JH7EJWYT65T 01GNTM3R1S1TZKSBFXZ60YATJ6 01GNDQNFYAMJPVR8P8DKW3YTHE 01GNGA2MVS384Y5XYSB3SD3Y30
We tried to do it by simply switching the remote writes to the new cluster and copying the old data from the origin bucket to the new but we get the following error in grafana:
In case of a migration done as you described, this error should resolve after a while, once compactor updated the bucket index and store-gateway synched all blocks.
how can we flush the old ingester
See: https://grafana.com/docs/mimir/v2.7.x/references/http-api/#flush-chunks--blocks
[Edit by @osg-grafana to include resolving link.]
Ok, thanks for the reply but there is a guide on how to correctly migrate from an instance to another?
Ok, thanks for the reply but there is a guide on how to correctly migrate from an instance to another?
We currently don't have a guide on how to migrate a Mimir cluster between two different Kubernetes clusters.