Allow searching by event attribute in TraceQL

[CinchBlue]

Is your feature request related to a problem? Please describe. With TraceQL, I currently cannot find a way to filter for traces by event attributes (it seems I can only filter by span or resource attributes).

Describe the solution you'd like Within TraceQL Concepts, scoped attribute fields are present. It may be useful to also add another "scope" such as event so that we can extend TraceQL to also filter on event attribute fields. This would allow queries like this:

{
  .service.name="foo-service" && 
  .deployment.environment="production"
}
&& {
  span.query=~".*specific query type.*"
}
&& {
  event.message=~".*cache miss.*"
}

Describe alternatives you've considered Could possibly derive the mappings/groupings from logs + regex + manual aggregation, but that sort of defeats the purpose of having a trace search/aggregation language built into Tempo already.

Additional context This is relevant to the case where you want to filter traces by specific events, such as a cache miss message or log:

[joe-elliott]

Agree this would be a nice feature. Note that we currently encode the value using proto and store it here:

https://github.com/grafana/tempo/blob/main/tempodb/encoding/vparquet2/schema.go#L98

To be able to query this efficiently we would need to encode Event attributes the same way we encode span attributes which may have an adverse effect on file size.

[midN]

👍 On that, this will be useful on very many occasions.

For example, we've enabled Mimir traces and were interested in finding particular queries executed to see how they were executed, through which shards requests went through and how many queries the search was split into. Finding a specific query executed was only doable with http.url parameter, which has encoded query values. At the same time Events for Mimir spans had a full query listed in it's original format. Being able to search by Events in TraceQL would have allowed to find these queries in much more granular way

[Bruno-DaSilva]

Another use case: we use opentelemetry autoinstrumentation in our apps and events are what contain stacktraces - we'd like to be able to search those if possible.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.

[nrakltx]

Is there any workaround for @Bruno-DaSilva's use case? I want to be able to see stacktraces from my Python application in Grafana (connected to a Tempo datasource), and the Python SDK records exceptions and adds them as events, but there is no way of seeing those events on Tempo. The only option I can think of is to copy all the span events and make them a list of attributes, but this is quite ad-hoc and probably not best practice.

[joe-elliott]

Currently there is no way to query events, but this is being worked on. We are defining it in the language and one of the goals of vParquet4 will be to encode events directly in the Parquet for increased efficiency.

Expect to see a PR in the coming weeks that defines the TraceQL extensions similar to our original core concepts doc. We've already started conversations on two extensions to the language we have defined:

https://github.com/grafana/tempo/issues/2810 https://github.com/grafana/tempo/issues/2805

[JorTurFer]

Is there any update here? We'd love to have this feature 😄

[joe-elliott]

A wip PR is up now: https://github.com/grafana/tempo/pull/3748

there are some technical hurdles to overcome, but it is in development.

[knylander-grafana]

Span events are available in Tempo 2.6. Also @ie-pham did an excellent blog post that explains how to query events with TraceQL: https://grafana.com/blog/2024/08/15/all-about-span-events-what-they-are-and-how-to-query-them/

[Bruno-DaSilva]

Yesssss!!!!!!! Let's go tempo team @joe-elliott @ie-pham @knylander-grafana 🎉 🚢 !

[joe-elliott]

closing this issue. thx @ie-pham for knocking this one out :)