Normalise `ai` related span data.
The Python SDK sends AI related span data as measurements. Because those are deprecated, relay should normalise this data into span.data (and in the future span.attributes. There are also some attribute names in span.data data that are deprecated in Otel, those should also be mapped to the new name. (This will bread the AI cost calulation in OpenTelemetry based SDKs)
This is how the mapping should be:
-
span.data["ai.prompt_tokens.used"]->span.data["gen_ai.usage.input_tokens"] -
span.measurements["ai_prompt_tokens_used"]->span.data["gen_ai.usage.input_tokens"] -
span.data["ai.completion_tokens.used"]->span.data["gen_ai.usage.output_tokens"] -
span.measurements["ai_completion_tokens_used"]->span.data["gen_ai.usage.output_tokens"] -
span.data["ai.total_tokens.used"]->span.data["gen_ai.usage.total_tokens"] -
span.measurements["ai_total_tokens_used"]->span.data["gen_ai.usage.total_tokens"]
If those total_tokens properties are there, then use them. If not, they should be calculated as span.data["gen_ai.usage.input_tokens"] + span.data["gen_ai.usage.output_tokens"].
Do not break the existing product the measurements need to stay where the are right now. This bill be handled in the issue Update metrics extraction to new span data names.