Simpler events for recording (local) GenAI function tool calling as generalized function tracing events

[michaelsafyan]

Area(s)

area:gen-ai

What's missing?

Although there are existing events for tool calling (gen_ai.assistant.message, gen_ai.tool.message), these are more geared towards the Gen AI model's view of what tools should be invoked/called (rather than from the standpoint of instrumenting the tool that ultimately got invoked). Additionally, these are somewhat complicated to generate and may not always be available (if the tool calling is deeply nested inside the logic of the client).

Where the model invokes local tools, it would be useful to be able to implement recording the input/output of those tool calls by intercepting the tool registration and having a somewhat simpler interface for tracing those calls. I also think that some of this modelling could be more general than just Gen AI.

I propose that we have a more generalized representation of function tracing to events that could be used.

Describe the solution you'd like

I'd like for there to be a more general set of events associated with function tracing:

Function Start

Represents a function being invoked and provides a way to record the parameters:

• Attributes:
  • event.name: "function_start"
  • code.function.name: the name of the function that was called
  • positional_argument_count: the count of positional args
  • keyword_argument_count: the count of keyword or optional arguments
• Body:
  • positional_arguments: an array of the positional arguments
  • keyword_arguments: a dictionary of the keyword arguments

Function End

Represents the result of a function being invoked.

• Attributes:
  • event.name: "function_end"
  • code.function.name: the name of the function that was called
  • error.type: if there was an error
• Body:
  • result: JSON representation of the (succesful) result
  • error: JSON representation of the (error) result

[michaelsafyan]

There is a draft PR related to Google Gen AI SDK instrumentation that illustrates how this might be used:

• https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3446/

[lmolkova]

@michaelsafyan agree it would be useful to define general-purpose function call semantic conventions. Any reason to define events though? We can capture params and result as arguments on the function call span.

Some relevant data points:

1. OTel Java supports something like this

   @WithSpan
   public void myMethod(@SpanAttribute("parameter1") String parameter1,
       @SpanAttribute("parameter2") long parameter2) {
       <...>
   }
   

   https://opentelemetry.io/docs/zero-code/java/agent/annotations/#adding-attributes-to-the-span-with-spanattribute

2. We should follow mapping from arbitrary object to OTel AnyValue for params and result

3. We added something similar in one of the Azure AI SDKs as an experiment https://github.com/Azure/azure-sdk-for-python/blob/ebf29c4252308b77788b65ccf69482e9eeef925b/sdk/ai/azure-ai-projects/azure/ai/projects/telemetry/_trace_function.py. It records code.function.parameter.{key} and code.function.return.value attributes on the function call span. Once/if complex attributes on spans are supported, these attributes could become complex and we could do JSON in the meantime.

   Both of those attribute should be opt-in and we'll need to figure out a friendly way to let users opt-in.

   E.g. in python, I'd assume we'd define a new decorator that either allows you to opt-in into additional attributes or it's obvious from its name, like trace_with_parameters. It'd also be nice to have a decorator that can work with global tracer instead of @tracer.start_as...

/cc @trask - I recall you mentioned that something similar came up in Java, wonder if you have any thoughts on generic conventions/instrumentation for capturing function params and return value.

[lmolkova]

also cc @open-telemetry/semconv-code-attribute-approvers if they have any thoughts

[trask]

/cc @trask - I recall you mentioned that something similar came up in Java, wonder if you have any thoughts on generic conventions/instrumentation for capturing function params and return value.

Java instrumentation doesn't have a generic naming conventions for function params / return value.

I was probably thinking of https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/13590, but that also requires user to provide explicit attribute names for things params / return value that they want to capture.

[michaelsafyan]

A span is fine, but it would be useful if there was some standard way to do it.

I do think it would be useful, though, to know how the parameters relate to the signature ...

I'm not sure how you would use parameter name in the case of "varargs" or other things like that. I think it would be better to use the official signature mechanism as the primary naming mechanism with the associated name that it appears with as perhaps additional metadata if the name is not part of the signature.

[SylvainJuge]

Regarding the ability to capture through generic code.* attributes a function/method call arguments, is that we should enable to capture at least two things:

• the arguments value: the use-case this issue is about
• the arguments type: there isn't (yet) a way to capture those in semconv but would effectively participate into method/function "identity".

For example, if we prefix those with code.function.parameter.{key}, where {key} is the argument name, I think it would make sense to capture the argument value with code.function.parameter.{key}.value, as it leaves the possibility to later define code.function.parameter.{key}.type.

It would also be consistent with the code.function.return.value that has been mentioned here, with that we could later add code.function.return.type as well.

On the other hand, maybe the function parameter types, function "signature" that are part of its identity would be better captured as a single string attribute, for example code.function.signature, in that case we could directly use code.function.parameter.{key} to provide directly the argument value and use code.function.return for the return value. Doing so could also mean the return value type would have to be included in the code.function.signature.

There are also a few other aspects that need to be addressed when capturing arguments:

• what value of {key} to use when argument name is not available ?
• how to deal with arguments that can't be directly converted to strings ?

[michaelsafyan]

I really like the proposal to have:

• code.function.parameter.<key>.type
• code.function.parameter.<key>.value
• code.function.return.type
• code.function.return.value

This is what I ended up adopting in https://github.com/open-telemetry/opentelemetry-python-contrib/pull/3446 .