[bug] Beeai - LLM spans are not properly capturing tools in llm attribute

[annguyenarize]

Describe the bug When using BeeAI autoinstrumentor, LLM spans are not capturing tools / functions available according to semantic conventions, so they are not rendered in Arize. The number of available functions is sometimes inconsistent e.g. shows 3 functions available but there are only 2

To Reproduce Run the getting started example

Expected behavior The LLM spans should list available tools and which one was called

Screenshots

**Additional context** Add any other context about the problem here.

[dosubot[bot]]

Hi @annguyenarize! I'm Dosu and I’m helping the openinference team.

This is a known gap in the BeeAI autoinstrumentor: it does not consistently extract and log available tools/functions in LLM spans according to OpenInference semantic conventions, which is why tools/functions are missing or inconsistent in the trace and not rendered in Arize. The OpenAI instrumentation has more robust logic for capturing and recording available tools and which were called, following conventions like llm.tools and openinference.available_functions—BeeAI currently lacks this level of detail and consistency, leading to the issues you described. The root cause is incomplete or non-robust extraction and logging of tools/functions at the right span lifecycle points, and not fully adhering to the conventions for tool/function attributes. There are related issues and discussions about this in the OpenInference repo, and the OpenAI instrumentor can serve as a reference for how BeeAI should handle this for correct and consistent reporting in traces and UIs like Arize (source, source, source).

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[Tomas2D]

I can't see the Functions tab in my instance. Is there a way to make it like yours?

I use docker run -p 6006:6006 -i -t arizephoenix/phoenix:latest.

[nate-mar]

@annguyenarize Is your screenshot from the latest of Phoenix or AX?

[annguyenarize]

@nate-mar this was on latest of AX. Looks like tools are inside the input attribute

I ran a quickstart script:

import os
from dotenv import load_dotenv

load_dotenv()

arize_space_id = os.getenv("ARIZE_SPACE_ID")
arize_api_key = os.getenv("ARIZE_API_KEY")
openai_api_key = os.getenv("OPENAI_API_KEY")

from arize.otel import register
from openinference.instrumentation.beeai import BeeAIInstrumentor
import asyncio

from beeai_framework.agents.experimental import RequirementAgent
from beeai_framework.agents.experimental.requirements.conditional import ConditionalRequirement
from beeai_framework.backend import ChatModel
from beeai_framework.adapters.openai import OpenAIChatModel
from beeai_framework.errors import FrameworkError
from beeai_framework.middleware.trajectory import GlobalTrajectoryMiddleware
from beeai_framework.tools import Tool
from beeai_framework.tools.handoff import HandoffTool
from beeai_framework.tools.search.wikipedia import WikipediaTool
from beeai_framework.tools.think import ThinkTool
from beeai_framework.tools.weather import OpenMeteoTool


tracer_provider = register(
    space_id=arize_space_id,
    api_key=arize_api_key,
    project_name="beeai-cookbook",
)

BeeAIInstrumentor().instrument(tracer_provider=tracer_provider)


async def main() -> None:
    knowledge_agent = RequirementAgent(
        llm=OpenAIChatModel(model="gpt-4o-mini"),
        # llm=ChatModel.from_name("ollama:granite3.3:8b"),
        tools=[ThinkTool(), WikipediaTool()],
        requirements=[ConditionalRequirement(ThinkTool, force_at_step=1)],
        role="Knowledge Specialist",
        instructions="Provide answers to general questions about the world.",
    )

    weather_agent = RequirementAgent(
        llm=OpenAIChatModel(model="gpt-4o-mini"),
        # llm=ChatModel.from_name("ollama:granite3.3:8b"),
        tools=[OpenMeteoTool()],
        role="Weather Specialist",
        instructions="Provide weather forecast for a given destination.",
    )

    main_agent = RequirementAgent(
        name="MainAgent",
        llm=OpenAIChatModel(model="gpt-4o-mini"),
        # llm=ChatModel.from_name("ollama:granite3.3:8b"),
        tools=[
            ThinkTool(),
            HandoffTool(
                knowledge_agent,
                name="KnowledgeLookup",
                description="Consult the Knowledge Agent for general questions.",
            ),
            HandoffTool(
                weather_agent,
                name="WeatherLookup",
                description="Consult the Weather Agent for forecasts.",
            ),
        ],
        requirements=[ConditionalRequirement(ThinkTool, force_at_step=1)],
        # Log all tool calls to the console for easier debugging
        middlewares=[GlobalTrajectoryMiddleware(included=[Tool])],
    )

    question = "If I travel to Rome next weekend, what should I expect in terms of weather, and also tell me one famous historical landmark there?"
    print(f"User: {question}")

    try:
        response = await main_agent.run(question, expected_output="Helpful and clear response.")
        print("Agent:", response.answer.text)
    except FrameworkError as err:
        print("Error:", err.explain())


if __name__ == "__main__":
    asyncio.run(main())

[satyadevai]

@nate-mar I've investigated the tool tracking issue in the BeeAI instrumentor and identified the root cause. The instrumentor was not properly capturing tools in LLM spans according to the OpenInference semantic conventions. After implementing a fix, tools are now visible in Phoenix UI as expected, which confirms the instrumentation is working correctly.

However, full tools are still not appearing in the AX UI despite the fix. Sometimes it's showing only 1 tool instead of all tools, and subsequent LLM calls are not showing tools at all. The same instrumentor is working as expected in Phoenix, which suggests there may be differences in how AX processes or renders tool attributes. I will continue furthur investigation.

Phoenix UI After Fix:

AX UI:

[satyadevai]

@caroger I am attaching logs file which contains all the intractions with LLM in both AX & Phoenix.

BeeaiInstrumentorLogs.json

[satyadevai]

@caroger

I updated the attributes to the proper tool format. After these changes, the first LLM call in AX showed the tools, but subsequent LLM calls did not display the tools even though they were present. However, in Phoenix, the actual tool calls were displayed correctly.

I am attaching screenshots, attributes for your reference.

AX 1st LLM Call:

AX 2nd LLM Call:

Phoenix 1st LLM Call:

Phoenix 2nd LLM Call:

Attribues: Ax Attributes: ax-llm-call-attributes.json

Phoenix Attributes: phoeniix-llm-call-attributes.json