Do not add service.name tag values to search data

[tarvip]

Is your feature request related to a problem? Please describe.

I am trying to replace our current Jaeger and Elasticsearch setup with Grafana Tempo. So far it has been quite promising.

However, there is a small annoyance with search functionality. We have tracing enabled in Traefik and it sets service.name tag, see here

This service.name tag value is also picked up by Tempo for search data and used in Service Name drop-down in UI see attached pictures.

On the other hand Jaeger has no such behavior.

Describe the solution you'd like

Would like to avoid having service.name tag values in Service Name drop-down.

Describe alternatives you've considered At the moment I am renaming service.name tag in opentelemetry-collector, but it feels a bit wrong.

Additional context

There is similar case with DataDog

[kvrhdn]

Hi, glad you're trying out Tempo!

This is an unfortunate conflict between conventions from OpenTelemetry and OpenTracing and also a consequence of how Tempo search is implemented...

1. Why we use service.name:

Traces ingested by Tempo are converted into the OpenTelemetry format (even if you are ingesting Jaeger, Zipkin, etc.), internally we use the same receivers as the OpenTelemetry Collector which works with the OpenTelemetry format.

In OpenTelemetry service.name is a special tag: by convention it should contain the name of the service this span is coming from. See the OpenTelemetry spec: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/semantic_conventions/README.md#service So this is why we use service.name in the search dropdown.

2. Why search shows service.name: ping@internal:

On your spans you have 2 service.name tags: you have them in the tags and in the process attributes. It's possible to distinguish between the actual service name and the tag that's accidentally also called service.name... but when building the search index Tempo squashes all tags and attributes together. That is, we have a single map with all tags and process attributes together. So if we search for the values of service.name all values will show up, including those in the tags.

Fixing this will be tricky as the search API is not able to differentiate between tags and process attribtues. We could only index service.name tags if they exist in the process attributes but this would make regular service.name tags unsearchable.

[annanay25]

@kvrhdn gave a nice explanation of how tag extraction and search works!

I'd like to just complement that with another setting we have, a deny list of tags that will not be extracted from trace data:

    # List of tags that will **not** be extracted from trace data for search lookups
    # This is a global config that will apply to all tenants
    [search_tags_deny_list: <list of string> | default = ]

under the distributor config: https://grafana.com/docs/tempo/latest/configuration/#distributor

However, the caveat is that the denylist applies to process tags as well, so even service.name=traefik will not show up in your tag dropdown.

In case you are interested, the logic for that is here (look at the uses of the extractTag function): https://github.com/grafana/tempo/blob/ba93a400a4968c303cdbff9f5ccf88f6194489ca/modules/distributor/search_data.go#L25-L88

[tarvip]

@kvrhdn thank you for explanation. I was hoping that it is enough if we could index only service.name from process attributes and not from tags attributes, but I understand that in this case it is not correct for other stuff.

@annanay25 I already tried search_tags_deny_list, but in this case dropdown was empty as expected as service.name from process atrributes was discarded as well. I also looked into the code and found that at the moment there is no choice.

Anyway, I guess I can continue using Attributes Processor in opentelemetry-collector to rename the service.name in tags attributes, seems to work fine.

Feel free to close this issue.

[mdisibio]

The upcoming TraceQL will be able to distinguish between resource- and span-level attributes with the same name: https://github.com/grafana/tempo/blob/main/docs/design-proposals/2022-04%20TraceQL%20Concepts.md#scoped-attribute-fields

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. The next time this stale check runs, the stale label will be removed if there is new activity. The issue will be closed after 15 days if there is no new activity. Please apply keepalive label to exempt this Issue.