[feature request] CrewAI - add instrumentation for AgentAction to capture tool usage as separate attribute

[annguyenarize]

Tool usage is formatted within LLM span output as Action: {tool name} and requires additional parsing in order to evaluate. Adding instrumentation for AgentAction and capturing tool usage as separate attributes would make it easier to trace agent decisions.

[caroger]

hi @annguyenarize, thanks for filing the ticket.

Do you mind adding the current(existing) the desired version of the attributes in json format in addition to the screenshot? That'd be very helpful.

Thanks!

[annguyenarize]

@caroger sure, see below. The agent_tools attribute on agent attributes was manually added to track which tools the agent called. Open to other formatting ideas - I just needed a way to grab the tools called without parsing the outputs in the LLM span attributes.

llm_attributes.json

agent_attributes.json

[ialisaleh]

Hi @annguyenarize - I have enabled the tracing support for CrewAI AgentAction to capture tool usage values from _handle_agent_action method in the CrewAgentExecutor class having the AgentAction (represents an action to be taken by an agent) & AgentFinish (represents the final answer from an agent), which are being stored as JSON in the INPUT & OUTPUT blocks in a new span (with separate attributes) for evaluations. Please test the updates when you have a chance. Let me know if you have any feedback on this. Thanks!

PR: https://github.com/Arize-ai/openinference/pull/2330

Branch: https://github.com/ialisaleh/openinference/tree/alisaleh/ticket-2037

[caroger]

parse json string in openai instrumentor instead of in CrewAI instrumentor

[ialisaleh]

Hi Roger - I am keeping this ticket in my backlog for visibility. I will incorporate Nate’s feedback once provided. Thanks!