Bug: 2.8 Query hint with with (most_recent=true) doesn't always give latest

[tiffanyfay]

Describe the bug with (most_recent=true) is not deterministic as expected, and it will jump between the latest time and a time prior that is within about 10 minutes

To Reproduce

1. Run docker compose up on https://github.com/grafana/tempo/tree/main/example/docker-compose/local
2. Go to 1 hour or 3 hours (I haven't tried other timeframes) and just continually run the query: {} with(most_recent=true)

Expected behavior The results should be deterministic and give the latest traces every time.

Environment:

• Infrastructure: laptop, mac apple silicon
• Deployment tool: docker-compose

Additional Context Tested with both versions below

grafana/tempo                              latest                                                                        c17dd8863ab7   6 days ago      148MB
grafana/tempo                              latest                                                                        666d5d36857a   4 hours ago     152MB

https://github.com/user-attachments/assets/721a69b6-177f-4d9c-9428-087f3350e8cf

Will add more info as I dig into it further.