openinference
openinference copied to clipboard
Published
3 months ago
•
Arize-ai
Arize-ai
[feature request] Trace internal tool usage for LangChain/OpenAI instrumentors
When using the responses API, internal tools are not currently being traced
Example:
from langchain.agents import create_agent
from langchain_openai import ChatOpenAI
model = ChatOpenAI(model_name="gpt-4.1", temperature=0.7, use_responses_api=True)
web_search_tool = {"type": "web_search_preview"}
agent = create_agent(model=model, tools=[web_search_tool])
message = {
"role": "user",
"content": [
{"type": "text", "text": "could you check upcoming events in Madrid?"}
],
}
response = agent.invoke({"messages": message})
related https://github.com/Arize-ai/openinference/issues/2284
These are non-message objects, and the OpenInference semantic convention currently can accommodate only message objects.
fyi. openai response payload
{
"id": "resp_004a7bb90d2c503b0069161e5eeee0819d9b30518b082b0d91",
"created_at": 1763057246,
"error": null,
"incomplete_details": null,
"instructions": null,
"metadata": {},
"model": "gpt-4.1-2025-04-14",
"object": "response",
"output": [
{
"id": "ws_004a7bb90d2c503b0069161e5fbd10819d88910842a2e84a50",
"action": {
"query": "upcoming events in Madrid",
"type": "search"
},
"status": "completed",
"type": "web_search_call"
},
{
"id": "msg_004a7bb90d2c503b0069161e60fef4819d988da28d18a528cc",
"content": [
{
"annotations": [
{
"end_index": 455,
"start_index": 353,
"title": "What's On In Madrid - November 2025 - Tio tours madrid",
"type": "url_citation",
"url": "https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai"
},
{
"end_index": 821,
"start_index": 661,
"title": "What's on in Madrid in November 2025? — idealista/news",
"type": "url_citation",
"url": "https://www.idealista.com/en/news/lifestyle-in-spain/2025/10/31/869291-what-to-do-in-madrid-in-november-2025-the-best-plans?utm_source=openai"
},
{
"end_index": 1156,
"start_index": 1054,
"title": "What's On In Madrid - November 2025 - Tio tours madrid",
"type": "url_citation",
"url": "https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai"
},
{
"end_index": 1530,
"start_index": 1428,
"title": "What's On In Madrid - November 2025 - Tio tours madrid",
"type": "url_citation",
"url": "https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai"
},
{
"end_index": 1817,
"start_index": 1715,
"title": "What's On In Madrid - November 2025 - Tio tours madrid",
"type": "url_citation",
"url": "https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai"
}
],
"text": "Madrid offers a vibrant array of events throughout November 2025, catering to diverse interests. Here are some highlights:\n\n\n\n**JazzMadrid International Festival** \nRunning throughout November, this festival features a mix of Spanish and international jazz acts, performances, and talks at venues like Fernán Gómez Theater and the Biblioteca Nacional. ([tiotoursmadrid.com](https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai))\n\n\n\n\n**Festival de Otoño (Autumn Festival)** \nFrom November 6 to 30, this cultural festival showcases theater, dance, and performances by both Spanish and international artists across various city venues. ([idealista.com](https://www.idealista.com/en/news/lifestyle-in-spain/2025/10/31/869291-what-to-do-in-madrid-in-november-2025-the-best-plans?utm_source=openai))\n\n\n\n\n**La Almudena Celebrations** \nHonoring Madrid's patron saint, festivities from November 8 to 10 include masses at Almudena Cathedral, processions through the historic center, and the traditional 'Corona de la Almudena' pastry. ([tiotoursmadrid.com](https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai))\n\n\n\n\n**Christmas Lights Switch-On & Plaza Mayor Market** \nOn November 22, over 13 million LED bulbs illuminate the city, marking the start of the Christmas season. The Plaza Mayor Christmas market opens on November 27, offering handcrafted ornaments and seasonal treats. ([tiotoursmadrid.com](https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai))\n\n\n\n\n**Madrid Craft Week** \nFrom November 20 to 30, over 250 local shops, hotels, and galleries host artisan showcases, workshops, and design pop-ups, celebrating local craftsmanship. ([tiotoursmadrid.com](https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai))\n\n\nThese events provide a rich cultural experience, blending music, art, tradition, and festive cheer throughout the month. ",
"type": "output_text",
"logprobs": []
}
],
"role": "assistant",
"status": "completed",
"type": "message"
}
],
"parallel_tool_calls": true,
"temperature": 0.7,
"tool_choice": "auto",
"tools": [
{
"type": "web_search_preview",
"search_context_size": "medium",
"user_location": {
"type": "approximate",
"city": null,
"country": "US",
"region": null,
"timezone": null
}
}
],
"top_p": 1,
"background": false,
"max_output_tokens": null,
"max_tool_calls": null,
"previous_response_id": null,
"prompt_cache_key": null,
"reasoning": {
"effort": null,
"summary": null
},
"safety_identifier": null,
"service_tier": "default",
"status": "completed",
"text": {
"format": {
"type": "text"
},
"verbosity": "medium"
},
"top_logprobs": 0,
"truncation": "disabled",
"usage": {
"input_tokens": 309,
"input_tokens_details": {
"cached_tokens": 0
},
"output_tokens": 483,
"output_tokens_details": {
"reasoning_tokens": 0
},
"total_tokens": 792
},
"user": null,
"billing": {
"payer": "developer"
},
"prompt_cache_retention": null,
"store": true
}
fyi. corresponding langchain output
{
"generations": [
[
{
"text": "Madrid offers a vibrant array of events throughout November 2025, catering to diverse interests. Here are some highlights:\n\n\n\n**JazzMadrid International Festival** \nRunning throughout November, this festival features a mix of Spanish and international jazz acts, performances, and talks at venues like Fernán Gómez Theater and the Biblioteca Nacional. ([tiotoursmadrid.com](https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai))\n\n\n\n\n**Festival de Otoño (Autumn Festival)** \nFrom November 6 to 30, this cultural festival showcases theater, dance, and performances by both Spanish and international artists across various city venues. ([idealista.com](https://www.idealista.com/en/news/lifestyle-in-spain/2025/10/31/869291-what-to-do-in-madrid-in-november-2025-the-best-plans?utm_source=openai))\n\n\n\n\n**La Almudena Celebrations** \nHonoring Madrid's patron saint, festivities from November 8 to 10 include masses at Almudena Cathedral, processions through the historic center, and the traditional 'Corona de la Almudena' pastry. ([tiotoursmadrid.com](https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai))\n\n\n\n\n**Christmas Lights Switch-On & Plaza Mayor Market** \nOn November 22, over 13 million LED bulbs illuminate the city, marking the start of the Christmas season. The Plaza Mayor Christmas market opens on November 27, offering handcrafted ornaments and seasonal treats. ([tiotoursmadrid.com](https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai))\n\n\n\n\n**Madrid Craft Week** \nFrom November 20 to 30, over 250 local shops, hotels, and galleries host artisan showcases, workshops, and design pop-ups, celebrating local craftsmanship. ([tiotoursmadrid.com](https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai))\n\n\nThese events provide a rich cultural experience, blending music, art, tradition, and festive cheer throughout the month. ",
"generation_info": null,
"type": "ChatGeneration",
"message": {
"lc": 1,
"type": "constructor",
"id": [
"langchain",
"schema",
"messages",
"AIMessage"
],
"kwargs": {
"content": [
{
"type": "text",
"text": "Madrid offers a vibrant array of events throughout November 2025, catering to diverse interests. Here are some highlights:\n\n\n\n**JazzMadrid International Festival** \nRunning throughout November, this festival features a mix of Spanish and international jazz acts, performances, and talks at venues like Fernán Gómez Theater and the Biblioteca Nacional. ([tiotoursmadrid.com](https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai))\n\n\n\n\n**Festival de Otoño (Autumn Festival)** \nFrom November 6 to 30, this cultural festival showcases theater, dance, and performances by both Spanish and international artists across various city venues. ([idealista.com](https://www.idealista.com/en/news/lifestyle-in-spain/2025/10/31/869291-what-to-do-in-madrid-in-november-2025-the-best-plans?utm_source=openai))\n\n\n\n\n**La Almudena Celebrations** \nHonoring Madrid's patron saint, festivities from November 8 to 10 include masses at Almudena Cathedral, processions through the historic center, and the traditional 'Corona de la Almudena' pastry. ([tiotoursmadrid.com](https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai))\n\n\n\n\n**Christmas Lights Switch-On & Plaza Mayor Market** \nOn November 22, over 13 million LED bulbs illuminate the city, marking the start of the Christmas season. The Plaza Mayor Christmas market opens on November 27, offering handcrafted ornaments and seasonal treats. ([tiotoursmadrid.com](https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai))\n\n\n\n\n**Madrid Craft Week** \nFrom November 20 to 30, over 250 local shops, hotels, and galleries host artisan showcases, workshops, and design pop-ups, celebrating local craftsmanship. ([tiotoursmadrid.com](https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai))\n\n\nThese events provide a rich cultural experience, blending music, art, tradition, and festive cheer throughout the month. ",
"annotations": [
{
"end_index": 455,
"start_index": 353,
"title": "What's On In Madrid - November 2025 - Tio tours madrid",
"type": "url_citation",
"url": "https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai"
},
{
"end_index": 821,
"start_index": 661,
"title": "What's on in Madrid in November 2025? — idealista/news",
"type": "url_citation",
"url": "https://www.idealista.com/en/news/lifestyle-in-spain/2025/10/31/869291-what-to-do-in-madrid-in-november-2025-the-best-plans?utm_source=openai"
},
{
"end_index": 1156,
"start_index": 1054,
"title": "What's On In Madrid - November 2025 - Tio tours madrid",
"type": "url_citation",
"url": "https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai"
},
{
"end_index": 1530,
"start_index": 1428,
"title": "What's On In Madrid - November 2025 - Tio tours madrid",
"type": "url_citation",
"url": "https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai"
},
{
"end_index": 1817,
"start_index": 1715,
"title": "What's On In Madrid - November 2025 - Tio tours madrid",
"type": "url_citation",
"url": "https://tiotoursmadrid.com/whats-on-in-madrid-november-2025/?utm_source=openai"
}
]
}
],
"additional_kwargs": {
"tool_outputs": [
{
"id": "ws_004a7bb90d2c503b0069161e5fbd10819d88910842a2e84a50",
"action": {
"query": "upcoming events in Madrid",
"type": "search"
},
"status": "completed",
"type": "web_search_call"
}
]
},
"response_metadata": {
"id": "resp_004a7bb90d2c503b0069161e5eeee0819d9b30518b082b0d91",
"created_at": 1763057246,
"metadata": {},
"model": "gpt-4.1-2025-04-14",
"object": "response",
"service_tier": "default",
"status": "completed",
"model_name": "gpt-4.1-2025-04-14"
},
"type": "ai",
"id": "msg_004a7bb90d2c503b0069161e60fef4819d988da28d18a528cc",
"usage_metadata": {
"input_tokens": 309,
"output_tokens": 483,
"total_tokens": 792,
"input_token_details": {
"cache_read": 0
},
"output_token_details": {
"reasoning": 0
}
},
"tool_calls": [],
"invalid_tool_calls": []
}
}
}
]
],
"llm_output": null,
"run": null,
"type": "LLMResult"
}