Improve documentation for importing thanos blocks to Mimir

[pstibrany]

Improve documentation for uploading Thanos blocks to Mimir. Steps shared by Taylor Mutch in the Slack thread are:

I use GCS but it's applicable to any block storage Copy the bucket to an intermediate bucket

gsutil -m cp -r "gs://{Thanos bucket}/*" "gs://{intermediate bucket}/{tenant}/"

Inspect the bucket to ensure it's valid from Thanos perspective:

thanos tools bucket inspect --objstore.config-file bucket.yaml

Remove the downsampled, duplicated blocks First we have to mark the downsampled blocks to be deleted

thanos tools bucket retention --objstore.config-file bucket.yaml --retention.resolution-1h=2h --retention.resolution-5m=2h --retention.resolution-raw=0d

Cleanup the blocks marked for deletion

thanos tools bucket cleanup --objstore.config-file bucket.yaml --delete-delay=0

Ignore the blocks from other replicas, only keep replica 0. Mark replicas 1 and 2 as deletable, and then clean them up

thanos tools bucket inspect --objstore.config-file bucket.yaml --output=tsv | grep prometheus_replica=prometheus-kronus-prometheus-1 | awk '{print $1}' | xargs -n 1 -P 1 thanos tools bucket mark --marker="deletion-mark.json" --objstore.config-file bucket.yaml --details="Removed as duplicate" --id
thanos tools bucket inspect --objstore.config-file bucket.yaml --output=tsv | grep prometheus_replica=prometheus-kronus-prometheus-2 | awk '{print $1}' | xargs -n 1 -P 1 thanos tools bucket mark --marker="deletion-mark.json" --objstore.config-file bucket.yaml --details="Removed as duplicate" --id
thanos tools bucket inspect --objstore.config-file bucket.yaml --output=tsv | grep __replica__=prometheus-kronus-prometheus-1 | awk '{print $1}' | xargs -n 1 -P 1 thanos tools bucket mark --marker="deletion-mark.json" --objstore.config-file bucket.yaml --details="Removed as duplicate" --id
thanos tools bucket inspect --objstore.config-file bucket.yaml --output=tsv | grep __replica__=prometheus-kronus-prometheus-2 | awk '{print $1}' | xargs -n 1 -P 1 thanos tools bucket mark --marker="deletion-mark.json" --objstore.config-file bucket.yaml --details="Removed as duplicate" --id

Cleanup the blocks marked for deletion

thanos tools bucket cleanup --objstore.config-file bucket.yaml --delete-delay=0

Inspect to make sure we only have 1 replica's worth of data

thanos tools bucket inspect --objstore.config-file bucket.yaml

See https://grafana.com/docs/mimir/latest/migration-guide/migrating-from-thanos-or-prometheus/ for details on why we must do it this way Convert blocks to Cortex using thanosconvert (replace with this mimirtool equivalent if there is one)

docker run -e GOOGLE_APPLICATION_CREDENTIALS=/var/secret/gcp.json quay.io/cortexproject/thanosconvert:master-42c68cd -config /conf/bucket.yaml

Inspect the blocks again, they should now have __org_id__={tenant} as the only label associated with the blocks

thanos tools bucket inspect --objstore.config-file thanos-mp-suite-qos-dev.yaml

Relabel the blocks to include cluster, gcp_project external labels

Now we need to rewrite the data so that each series includes the cluster and gcp_project labels on the metrics. Create a rewrite config like the following:

# relabel-config-{tenant}.yaml
action: "replace"
  target_label: "cluster"
  replacement: "{cluster_name}" # cluster_name label from the Thanos labels
action: "replace"
  target_label: "gcp_project"
  replacement: "{gcp_project}"  # gcp_project label from the Thanos labels

Perform the rewrite dry run to confirm all works (note the --dry-run flag)

thanos tools bucket inspect --objstore.config-file bucket.yaml --output=tsv | grep prometheus | awk '{print $1}' | xargs -n 1 -P 1
thanos tools bucket rewrite  --objstore.config-file bucket.yaml --rewrite.to-relabel-config-file relabel-config.yaml --dry-run --id 

Scale down Cortex compactor (Mimir compactor) Cortex compactor should not be running when we are making out-of-band changes to the storage system

Copy blocks to Cortex bucket (Mimir bucket)

Scale up compactor

[TaylorMutch]

Looks right to me! Worth noting I used Thanos 0.28.0 in the thanos bucket tools ... commands

[bboreham]

What is left to do here? Looks like the info was added.

[pstibrany]

Agree, we can close this.