Search service names and tags from persistent storage

[pavolloffay]

Is your feature request related to a problem? Please describe.

Service and tag names disappear from the drop-down after 15 minutes if they are not reported.

This is configured by complete_block_timeout which defaults to 15 minutes see https://grafana.com/docs/tempo/latest/configuration/#ingester.

Describe the solution you'd like

The service name and tags in the drop-down should be listed even if the service recently didn't report any data. The service and tags should be listed if there are present for a chosen time range.

Describe alternatives you've considered

Additional context The issue was document in https://github.com/grafana/tempo/pull/1651/

[joe-elliott]

Agree this is the goal. Without additional data structures, however, this is simply not possible with the v2 format. I'm thinking 2.1 would be a good target for this functionality.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

[deathsaber]

Hi @joe-elliott , just wondering if this enhancement is being released with tempo v2.2.0 as I was looking in the release notes for 2.2.0 but could not find a mention of this issue.

[joe-elliott]

This is not included in 2.2. The code to search the backend is easy in a sense, but it will not perform at scale. We are currently investigating options.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

[sam-mcbr]

We encountered this issue recently while trying to migrate from Jaeger backend to Tempo. It's definitely a big downside of Tempo right now for us. Just curious if there is a rough ETA for this or options are still being investigated. Thanks so much, we're really excited for all Tempo has to offer!

[joe-elliott]

So @mapno is currently focused on making attribute search filter by other conditions. This will make the following query correctly filter span.http.path values by resource.service.name="foo". After this we were going to start looking at pulling tag values from the backend.

{ resource.service.name = "foo" && span.http.path =

There was a community attempt to pull from the backend but it's been abandoned.

[sam-mcbr]

Got it, thanks for the update!

[bentqc]

I would also like to record the need for listing all service names regardless where traces are stored. We have applications that can potentially have hundreds of service names and once the trace leave the ingesters, the service names disappears from the list making searches more complicated.

We are also trying to move from Jaeger to Tempo so that we can normalize the environment with other products such as Loki and improve efficiencies. So this functionality will be needed in order to migrate.

[serrrios]

+1

I'd also like to point out that it would be nice if after selecting a resource name, the spans only related to that resource name, rather than all the ones in the system.