mimirtool backfill - compactor: can't check block state/block validation failed

[litetex]

trafficstars

Describe the bug

When running mimirtool backfill I get caller=block_upload.go:207 level=warn component=compactor user=tdr feature="block upload" block=01J4WNXPS4GSR8AMK9A4732704 operation="start block upload" request_id=17ec82ff283a34ff msg="can't check block state" response="block validation failed" status=400

Setting the log level to debug doesn't help at all.

It would be nice to know WHY the validation failed so that I can diagnose and address the problem...

To Reproduce

Sorry but I can't paste my complete setup here.

Basically:

1. Start mimir
2. Run backfill for prometheus: mimirtool backfill --address=http://mimir-backend:9009 --id=<redacted> /data/prometheus/01J4WNXPS4GSR8AMK9A4732704 ...

Expected behavior

See an error message that says what caused the block validation error so that I can fix it.

Environment

irrelevant Mimir version: 2.13.0

Additional Context

Same problem was already encountered in https://github.com/grafana/mimir/issues/8164 but no efforts were made to improve the situation :/

Likely these pieces of code should rethrow the error and not just "swallow" it: https://github.com/grafana/mimir/blob/3872ccbbb4b2d54f2bf535659db37af5e5f8eac9/pkg/compactor/block_upload.go#L766-L768 https://github.com/grafana/mimir/blob/3872ccbbb4b2d54f2bf535659db37af5e5f8eac9/pkg/compactor/block_upload.go#L733-L735