TraceQL: Sorting traces when searching from Grafana

[actionbronson]

Is your feature request related to a problem? Please describe. When searching tempo from Grafana, I'd like to be able to sort the returned traces by duration

Describe the solution you'd like A proposed sorting mechanism, e.g. duration or count.

Describe alternatives you've considered None

Additional context I'm unsure if this is a Grafana problem at the presentation layer or a Tempo problem.

[joe-elliott]

Howdy, thanks for the request.

I'm unsure if this is a Grafana problem at the presentation layer or a Tempo problem

I suppose it depends on what the real need is here. Currently Tempo's search results are non deterministic. If you choose a 24 hour window it will return the first N results that happen to match the query.

If you want to be able to explicitly retrieve the first N results in the specified time range in time order then we would need to make changes to deterministically search the Tempo backend.

If you want to just have a sortable table in Grafana with the results you are getting back then this would be a frontend/Grafana feature request.

WDYT?

[actionbronson]

Hey Joe,

Thanks for the reply.  Ideally it would be to be able to have Tempo sort and return the results with an order-by field, e.g. an intrinsic field such as duration.  An example use-case is to query traces based on a few fields and return the Top 10 worst performers.  Right now we can do something like {.field = “ABC” && duration > 10m} which typically returns our worst-cases but this is based on a few heuristics.

It’d be awesome to be able to {.field=“ABC” && duration > 10m} | sort(duration).

On Jul 17, 2023, at 16:05, Joe Elliott @.***> wrote:

Howdy, thanks for the request.

I'm unsure if this is a Grafana problem at the presentation layer or a Tempo problem

I suppose it depends on what the real need is here. Currently Tempo's search results are non deterministic. If you choose a 24 hour window it will return the first N results that happen to match the query.

If you want to be able to explicitly retrieve the first N results in the specified time range in time order then we would need to make changes to deterministically search the Tempo backend.

If you want to just have a sortable table in Grafana with the results you are getting back then this would be a frontend/Grafana feature request.

WDYT?

— Reply to this email directly, view it on GitHub https://github.com/grafana/tempo/issues/2659#issuecomment-1638794014, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAY3QPH5ODMIOOGMEVCEUEDXQWLKLANCNFSM6AAAAAA2KT6Z5A. You are receiving this because you authored the thread.

[joe-elliott]

So this would be difficult to do generically, but not impossible. For a given time range we would always have to do a completely exhaustive search and then pass all results up to the query frontend to do the sorting and return the requested number of traces.

Specifically start time would be a deterministic version of the search that always returned the N most recent traces which we have considered in the past. This would be a bit easier b/c we could use the block time ranges to not perform a completely exhaustive search.

I'm not opposed to this feature, and will tag it so it stays on our radar, but I do think this is a back burner item.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

[swar8080]

+1 for the ability to sort by duration

We have a dashboard to explore latency issues for a specific span, so it'd be nice to display the longest duration traces in the dashboard's time range

[ADavis11]

We have a dashboard to explore latency issues for a specific span, so it'd be nice to display the longest duration traces in the dashboard's time range

We have the same thing in our env and would love an option to get the longest durations.

[peterdk]

Same issue here, I notice that my overview of the last hour Tempo traces in Grafana doesn't display latest results, but apparently random results from the selected timeframe (1 hour). I would prefer a order by time desc or similar solution in the TraceQL query.