getsentry
Missing samples due to 400 in metrics explorer
For many queries, the samples endpoint returns a 400 with a "Query timeout" message - this should not happen with built-in functionality on span metrics (since we should always have spans)
The capture_exception call shows that we hit the Snuba rate limit with our queries (SENTRY-3DCH) because Snuba tries to allocate 80GB for the single query.
Reached out to @Zylphrex to check whether we can/should modify the table to allow for different queries with a sample_rate - waiting to hear back if this is something we can solve through a different query.
This error is not due to snuba allocating memory for the query. But more specifically, the issue is that we're scanning 80GB for the query which is hitting the limits imposed by snuba and resulting in this rate limit exceeded error.
The precise reason why this is happening is due to the query being used which is trying to read the description column (one of the largest columns in the spans table) over 14 days resulting in clickhouse having to scan this much data.
There's not much we can do about this product side but if this is a requirement, we should have this conversation with the SnS team and see what can be done. Feel free to loop me in on this conversation as I have seen similar issues in other places.
