TraceQL autocomplete

[mapno]

Context

As we develop TraceQL and its main integration with Grafana, it is important for the language that we make it easy to use and provide a gentle learning curve. A big part of that experience is making writing queries easy for the users.

Many well-known query languages, such as PromQL and LogQL, provide the user with suggestions of possible conditions that will result in a successful query. We aim to add this concept to TraceQL as well.

As it is right now, suggestions are very limited by the API design and implementation. Grafana queries Tempo for all tags and their values, and Tempo will return everything that it has, without any filtering based on the current query. That makes it possible for Grafana to suggest tags and values that produce no matches in Tempo.

Objectives

• Design and implement an API that generates suggestions filtered by a query that's being typed

Tasks

Phase 1

• [ ] Implement search of attributes in parquet blocks

Phase 2

• [ ] Implement search of attributes in ingester search: recent traces and WAL
• [ ] Extend the TraceQL engine to parse incomplete queries and extract conditions to filter down by

Phase 3

• [ ] Design and API that returns possible attributes and labels filtered down by a set of already defined attributes
• [ ] Integrate this API with Grafana

[joe-elliott]

I believe phase 1 and the first checkbox of phase 2 is complete. I'd also not worry about the in memory traces and only search the WAL for attributes since that is what search is doing anyway.

[97amarnathk]

@joe-elliott @mapno I find this interesting and would like to work on this if this issue still open.

[mapno]

Hi @97amarnathk. Thanks for the interest, unfortunately, I'm already working on this issue. Apologies for not noting it here, my bad. Although I'm sure there'll be plenty of tasks that can be done around this project, it's still on its early stages. I will ping you in those.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

[09jvilla]

(Stretch) Support more than one spanset selector {} per query

@mapno -- This means that with what you intend to deliver in the next 3 months, I'll be able to see the filtered tags if I've already specified cluster=foo, but if I have as a filter cluster=foo && environment=dev, you can't show me a filtered view that incorporates both of those?

Oh actually reading this more closely, what you're saying you won't support out of the gate is filtering for a query like

{cluster=foo} && {environment=dev}

You would support {cluster=foo && environment=dev}

Am I right?

[09jvilla]

only values that can produce a successful query are returned.

To be clear, when you say "successful" what you mean is "only values that can produce a query that returns a non-empty results set" right?

[mapno]

Oh actually reading this more closely, what you're saying you won't support out of the gate is filtering for a query like {cluster=foo} && {environment=dev}. You would support {cluster=foo && environment=dev} Am I right?

That's exactly it

To be clear, when you say "successful" what you mean is "only values that can produce a query that returns a non-empty results set" right?

Yes. That's a better way of saying it.

[09jvilla]

Awesome - thanks for clarifying.

you won't support out of the gate is filtering for a query like {cluster=foo} && {environment=dev}. You would support {cluster=foo && environment=dev}

This seems like the right way to phase out the work. @adrapereira 's initial implementation of the visual query builder for traceQL only let's you create queries with 1 spanset, so what you're doing with objective 1 will cover what the frontend needs for their phase 1 =)

[09jvilla]

Autocomplete bug found in the Grafana frontend: https://github.com/grafana/tempo/issues/2200

Think this would be nice if we could add this fix to Objective 1 since status is a pretty common field people want to filter on.

[09jvilla]

@mapno does objective 1 also include any changes to GET /api/search/tags (in other words, the endpoint that lists the tag names themselves)?

As I understand it, what you have in objective 1 will handle filtering down to valid values (i.e. values that will produce a non-zero result).

So if all my traces with tag app=app1 also have the tag dc with values dc1 and dc2 but not dc3 (which exists on traces with tag app=app2), then I would expect autocomplete to just return dc1 and dc2 (not dc3) when I pass the app=app1 filter).

However, imagine I have some spans with app=app3. None of those spans have the dc attribute.

Will I be able to use autocomplete to tell me that? I think the answer is yes, but not via the search tags endpoint, but probably through search tag values? I'd have to hit search tag values with app=app3 for the values of dc, see that there were no valid values for dc, and then conclude that dc isn't an attribute attached to spans with app=app3?

I'm asking because I'm trying to imagine what this will look like in the Grafana frontend. I'm curious if we'll be able to filter down the list of tag names we let the user select (not just tag values) given their choices of Service Name and Span Name (or any other already applied key<>value filter).

In the screenshot below, I'd rather just not have been shown .available_dbs as an option of a tag name to filter by if there were no valid values for it.

If this is an "objective 2" thing that's totally fine - just want to be sure I understand scope in terms of what use cases we'll be able to handle with each objective.

[mapno]

Will I be able to use autocomplete to tell me that? I think the answer is yes, but not via the search tags endpoint, but probably through search tag values? I'd have to hit search tag values with app=app3 for the values of dc, see that there were no valid values for dc, and then conclude that dc isn't an attribute attached to spans with app=app3?

Exactly that. I wasn't planning on extending /search/tags, although I suppose that once implemented for tag values, it wouldn't be that difficult. The main drawback is that it would require a change on how Grafana interacts with the API—currently queries /search/tags once at the beginning. I'm not sure it's worth the effort.

[09jvilla]

The main drawback is that it would require a change on how Grafana interacts with the API—currently queries /search/tags once at the beginning. I'm not sure it's worth the effort.

Let's cross that bridge when we get there (specifically "let's cross that bridge when objective 1 is complete"). Perhaps not worth the effort and/or the performance would be crappy but if the alternative is waiting on objective 2 which is a multi-month effort, we may want to consider doing this as a stopgap if this is low effort.

WDYT?

[09jvilla]

I've created this bug issue so we don't lose this. However, as I mention above, I think fixing this bug is lower priority than getting to objective 1. https://github.com/grafana/tempo/issues/2202

[mapno]

Superseded by https://github.com/grafana/tempo/issues/2957