Mimir returns HTTP status 422 in cases where 5xx makes more sense

[jkirkwood]

We often receive a 422 HTTP response from Mimir with the following error message:

{
  "status": "error",
  "errorType": "execution",
  "error": "attempted to read series at index 0 from store-gateway chunks stream, but the stream has failed: rpc error: code = Internal desc = loading chunks: block 01HXN90CNDFJGRBZWY5HK5FMV8: get range reader: fetching range [27712000, 27728000]: Get \"<bucket_url>/<tenant_id>%2F01HXN90CNDFJGRBZWY5HK5FMV8%2Fchunks%2F000011\": http2: client connection force closed via ClientConn.Close"
}

In this case it seems like a 500-level error is more appropriate given this appears to be a server error, and retrying the same query usually works. Many HTTP clients have the ability to retry 5xx-level error codes automatically, so returning the proper status code prevents additional workarounds.

[56quarters]

I'm able to reproduce this locally. It appears to be caused by our error translation code not covering the entire querier hierarchy (queryable -> querier -> series set -> series -> iterator).