Investigate Rate Limiting Queries

[joe-elliott]

Is your feature request related to a problem? Please describe. Tempo queries can be costly and we should investigate rate limiting our customers. The cortex libraries we depend on already have rate limiting based on queue size:

https://github.com/cortexproject/cortex/blob/master/pkg/frontend/v1/frontend.go#L334

However, some tempo queries (such as recent trace search) will only create a single queue item even though they can be very impactful to ingesters.

Describe the solution you'd like Investigate ways to rate limit queries that reflect their cost in Tempo.