open-telemetry
Clarify cardinality requirements of a span name
Came up in: https://github.com/open-telemetry/semantic-conventions/pull/35#issuecomment-1574020499
In #3470 we realized that http.request.method can have high cardinality when custom HTTP methods are used.
While it clearly represents a problem for metrics, it's not clear what the consequences are for the span name.
The spec does not clearly require span name to have low cardinality, instead, it talks about names being general, identifying a class of operations rather than individual ones:
https://github.com/open-telemetry/opentelemetry-specification/blob/6941cada4adc5cca06abd6f5d41ac71e6b8b7477/specification/trace/api.md?plain=1#L303-L311
Based on this and the history of the changes (#557, #270), the idea is to identify a class of operations and work as a grouping key. In this sense, the cardinality restrictions are soft and the cardinality term doesn't quite apply.
In the context of http method, grouping operations should work well even when invalid methods are recorded in the span name - there just will be a long tail of rare custom methods.
Question: Do we use/envision using the span name as a dimension on metrics?
- if yes: We should clearly state that span names SHOULD have low cardinality
- if no: Let's keep recommending instrumentations to use the name that identifies a class of operations, but clarify that telemetry pipeline/consumers of telemetry should not assume span names to have low cardinality.
Generally, I feel it is not good for a span name to have high cardinality.
The question is if we want to live with the "defect" we currently have in the HTTP conventions, or find another solution like normalization (nontrivial) or using a rather unhelpful span name (like "HTTP client" instead of "HTTP <method>")
Generally, I feel it is not good for a span name to have high cardinality.
I agree. And yes, the questions are:
- is high cardinality in edge cases a problem?
- is there any benefit in normalization?
The question is if we want to live with the "defect" we currently have in the HTTP conventions, or find another solution like normalization (nontrivial)
isn't this defect solved if we go with Limit HTTP request method cardinality: original_method (option A)?
this is the one reason I'm currently leaning towards "Option A"
cc @reyang @jsuereth @arminru
I think normally one would not use high-cardinality attributes as input for the span name, instead of normalizing. But that would leave nothing at all for (client-side) HTTP except for "HTTP client"... (Server-side, one could consider reverting #2998, or clarifying that it only applies to accepted requests)
What's exactly the scope of this issue? Just span names recommended by semantic conventions, or all span names in general?
If it's the latter, I'd be reluctant to add this as a SHOULD into the API specification, as there is no way to enforce this (as it's given by the end user of the API).
IMHO the wording should/could address the consumers here: SHOULD they be able to handle high cardinality in a reasonable manner or MAY they assume span names to be low-cardinality (and exhibit bad behavior like dropping spans entirely in an unsystematic way, or creating UIs pages that are so large that they freeze the browser on load, etc.)
A (at least as strict) recommendation for the semantic conventions can be made additionally.
I would rather be strict about assumptions and say "Consumers SHOULD NOT assume span names to have low cardinality".
Even when every instrumentation does its best and creates names from low-cardinality attributes, in a big system number of distinct operations can reach hundreds (thousands?). We don't define what constitutes low or high cardinality (#2996), but it feels like at least middle-ish.
What's exactly the scope of this issue? Just span names recommended by semantic conventions, or all span names in general?
great question. Agree we can't control span names in general. The question is how much effort we need to do for semconv and otel-authored instrumentations.
Discussed by GC - We concur that clarifying language to consumers of OTel Spans should indicate that span names MAY be high cardinality, but that the interpretation of 'statistically interesting' in the linked spec document should be clarified to read as 'low cardinality'.
I'm curious what the current thinking is on this question in the original post:
Do we use/envision using the span name as a dimension on metrics?
Setting aside the problem of high cardinality span names, it is pretty powerful to be able to break down, for example, a service's duration metrics based on span name (essentially a transaction in many vendor products).
hi @cheempz! we have had some recent discussions about this in semantic conventions, and the direction we're going (you can see this in http and database semantic conventions already) is that we are defining span name in terms of low-cardinality attributes (like http.route and db.query.summary), which we are also putting on to metrics
we considered going further than this and introducing a specific attribute to represent span name on metrics (see https://github.com/open-telemetry/semantic-conventions/issues/1567), but backed off from that proposal for now at least