Grafana Cloud Tempo: Default Start Time view for traces in Tempo does not order by most recent

[liamoddell]

Describe the bug Using the Tempo data sources view within Explore on Grafana Cloud, the Start Time of shown traces does not sort as most recent when changing timeframe. For example, if I choose the last 6 hours worth of data in Tempo and the current time is 12:47pm, the first traces we see would be from 6am or 3am that same day and not from the X number of shown traces up to the current time period.

To Reproduce Steps to reproduce the behavior:

1. Access Tempo via Grafana Cloud
2. Have constant trace data over the last 6 hours
3. Change the timerange to Last 1/2/6 hours.
4. See that the Traces are not the most recent

Expected behavior If I sort by the last 1/2/6/12 hours, the first traces that show should be the most recent without having to manually sort by StartTime.

Environment:

• Grafana Cloud version: v10.3.0-62838

Additional Context

[mapno]

Hi @liamoddell. Tempo search is non-deterministic, which means that Tempo will return the first results that it finds, without a preference to more recent traces.

Related issue: https://github.com/grafana/tempo/issues/2476

[joe-elliott]

I'd like to move towards deterministic search. We did not at first because search was slow. Parquet search is much faster now. That plus the streaming endpoint makes me think we can go to deterministic.

Also, for future questions I'd consider using support. This is kind of a generic question so it's easy to answer, but if your question was specific to your instance it would be easier to answer through that channel.

[gouthamve]

I think this is not very intuitive behavior and borders on being a bug.

With logs, you see the most recent logs first and the expectation is that its the same with Tempo. But with the current behaviour makes it seem like the trace ingestion stopped because we are seeing quite old traces. This is even worse with explore view because it opens with a default 6hr view. So you could see traces that are 6hrs old and none of the recent traces.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

[mghildiy]

I can give it a try.

[joe-elliott]

@mghildiy I am currently in the middle of a frontend rewrite/refactor. Now is not a great time to make changes there.

One of the goals of the refactor is to make adding functionality such as this easier.

[rolfik]

Was this ever resolved? We still have problems viewing recent/current traces in Grafana loaded from Tempo. We see older traces instead of newer ones which is odd.

[joe-elliott]

It has not been, but I intend to tackle it as part of the refactor I'm working on now:

https://github.com/grafana/tempo/pull/3400

[elburnetto-intapp]

Hey @joe-elliott, is there any further updates on this issue at all?

[joe-elliott]

Unfortunately, I was unable to address this while working on the refactor. Perhaps as TraceQL metrics wind down I will find time to revisit.

[BradErz]

We are considering moving over to tempo from jaeger and we ran into this unexpected behaviour as well.

Like others have mentioned it makes it quite confusing, is it something planned to be done in the upcoming days, weeks, quaters?

[joe-elliott]

I am working on this now. I have a refactor PR up that is step 1:

https://github.com/grafana/tempo/pull/3888