sentry icon indicating copy to clipboard operation
sentry copied to clipboard

Search for Session Replays are rate limited in the UI

Open masinette opened this issue 1 year ago • 5 comments

Environment

SaaS (https://sentry.io/)

Steps to Reproduce

  1. Search for session replays (with many results)

specific example linked in shadow ticket

Expected Result

Search loads a list

Actual Result

Image

Image

Product Area

Replays

Link

No response

DSN

No response

Version

No response

┆Issue is synchronized with this Jira Improvement by Unito

masinette avatar Aug 15 '24 20:08 masinette

Auto-routing to @getsentry/product-owners-replays for triage ⏲️

getsantry[bot] avatar Aug 15 '24 20:08 getsantry[bot]

We seem to have a quite short rate limit going here. When I tried to repro, I couldn't for a few tries. Data loaded properly, I believe, 3x full page refresh. Then I got rate limited.

Thanks for raising, we'll dig into what's going on.

bruno-garcia avatar Aug 16 '24 16:08 bruno-garcia

Hi Bruno, do you have any updates? This is still affecting our whole engineering and support teams, as we're rarely able to search for session replays.

I'm happy to record a loom video reproducing the issue, if that helps. We're keen to have this issue resolved, as it heavily affects our productivity. Thanks.

rafalzv7labs avatar Aug 26 '24 08:08 rafalzv7labs

Thanks for writing in @rafalzv7labs. This is a p0 for us and we'll be looking into to it asap.

bruno-garcia avatar Aug 26 '24 15:08 bruno-garcia

We identified the rate limit is coming from Snuba: https://github.com/getsentry/sentry/blob/97a906cd8b69a52dd2c56928438e897e033a55ad/src/sentry/utils/snuba.py#L338

Still looking at values and other conditions

bruno-garcia avatar Aug 26 '24 18:08 bruno-garcia

We increased limits on our end and I couldn't repro being rate limited after a few refreshes. Could you verify things work as you expected?

If not, please give us more information about the usage patterns. Is there a script calling into our APIs on your end? Does this happen every N hours? Some times of the day? etc

bruno-garcia avatar Aug 27 '24 18:08 bruno-garcia

For the tag querying issue we need to apply an index to the tags field and update our query optimizer to use the index when possible. The way we'll index the field requires a new materialized column and will take 90 days to back fill. Though we may use it opportunistically before then by introspecting the time range of the query.

cmanallen avatar Sep 16 '24 14:09 cmanallen

I haven't seen this issue come up again. Let me know if customers are still seeing errors. The tags index is still in the process of rolling out but it should complete in the next ~45 days. The tags index should be working for anyone with a stats-period of 30 days or lower.

cmanallen avatar Oct 17 '24 15:10 cmanallen