tempo
tempo copied to clipboard
Allow querying traces by span instrumentation "Library Name"
Is your feature request related to a problem? Please describe. I'm trying to find traces with spans created by specific library name.
Describe the solution you'd like TraceQL query should support finding traces that have spans with specific library name. For example, smth like following pseudocode:
{ resource.service.name="skeleton-nestjs" && instrumentationLibrarySpans.instrumentationLibrary.name="@opentelemetry/instrumentation-http"}
to match traces similar to the example screenshot from Grafana:
When exporting the example trace to JSON (from Grafana), then it looks like following:
{
"batches": [
{
"resource": {
"attributes": [
...
],
"droppedAttributesCount": 0
},
"instrumentationLibrarySpans": [
{
"spans": [
{
"traceId": "1245cf2e87b6be30ed3700f1c126b557",
"spanId": "0882ee038dfdb585",
"parentSpanId": "0000000000000000",
"traceState": "",
"name": "GET /ping",
"kind": "SPAN_KIND_SERVER",
"startTimeUnixNano": 1710846323303000000,
"endTimeUnixNano": 1710846323303740000,
"attributes": [
...
],
"droppedAttributesCount": 0,
"droppedEventsCount": 0,
"droppedLinksCount": 0,
"status": {
"code": 0,
"message": ""
}
}
],
"instrumentationLibrary": {
"name": "@opentelemetry/instrumentation-http", // this is the attribute to match using TraceQL
"version": "0.49.1"
}
}
]
},
]
}
Additional context If it would be possible to find traces based on instrumentation library name, i could figure out the traces that are used for specific service (without looking at its code) to understand what instrumentation libraries could be disabled (to speed up application startup time that is super-slow when using auto-instrumentation for nodejs applications)
We have defined this in the language:
https://github.com/grafana/tempo/blob/main/docs/design-proposals/2023-11%20TraceQL%20Extensions.md#instrumentation-scope
but have yet to implement it. Currently all of our internal efforts are focused on delivering adhoc metrics from traces, but we do intend to add this eventually.
We would also accept an external contribution for this, but it might be difficult. We have had external TraceQL additions before but they require quite a deep dive.
Here's an example: https://github.com/grafana/tempo/pull/2848/files
This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.
This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.
Based on my reading of https://github.com/grafana/tempo/blob/7961fb746daeea8f274db3e8ebbcac78b74f05b8/docs/design-proposals/2023-11%20TraceQL%20Extensions.md#instrumentation-scope, https://github.com/grafana/tempo/pull/3967 implements this.
the PR #3967 that implements this feature is now merged :+1: (although the task was closed as stale before it got merged)