relay icon indicating copy to clipboard operation
relay copied to clipboard

Update metrics extraction to new span data names

Open antonpirker opened this issue 8 months ago • 0 comments

This issue must be handled after the issue Normalise ai related span data.

We need to update the metrics extraction for the AI insights and the calculation of the total cost of requests to LLMs (which involves calculating the total tokens).

What needs to be done:

  • Use span.data["gen_ai.usage.input_tokens"] for metrics extraction (instead of span.measurements["ai_prompt_tokens_used"])
  • Use span.data["gen_ai.usage.output_tokens"] for metrics extraction (instead of span.measurements["ai_completion_tokens_used"])
  • Save the total token amount in span.data["gen_ai.usage.total_tokens"] (instead of span.measurements["ai_total_tokens_used"])
  • Use the new span.data["gen_ai.usage.total_tokens"] for calculating the total cost of the AI request.

After this changes have been done, we can remove the existing code that normalized the AI related data into measurements.

A good starting point to find the places in relay to change is to grep for ai_total_tokens_used (the old measurement used for cost calculation), or ai.total_tokens.used for the old normalisation code (I guess, not an relay expert).

There is this simple test project that makes a request to the OpenAI API that can be used for testing this. (This needs a OPENAI_API_KEY env var. Ping @antonpirker to get an account that works.)

antonpirker avatar May 13 '25 07:05 antonpirker