litellm
litellm copied to clipboard
Published
1 week ago
•
BerriAI
BerriAI
[Bug]: I/O operation on closed pipe while using litellm with mcp
What happened?
Here was the code I used
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm
import asyncio
import json
server_params = StdioServerParameters(
command="npx",
args=["-y", "[email protected]"],
env={"TAVILY_API_KEY": "<key>"}
)
async def main():
async with stdio_client(server_params) as (read, write):
async with ClientSession(read, write) as session:
# Initialize the connection
await session.initialize()
# Get tools
tools = await experimental_mcp_client.load_mcp_tools(session, 'openai')
print("MCP TOOLS: ", tools)
messages = [{"role": "user", "content": "What is the newest Salesforce model?"}]
llm_response = await litellm.acompletion(
model="ollama/hf.co/DevQuasar/Salesforce.Llama-xLAM-2-8b-fc-r-GGUF:Q8_0",
api_key='sk-1234',
messages=messages,
tools=tools,
)
print("LLM RESPONSE: ", json.dumps(llm_response, indent=4, default=str))
asyncio.run(main)
### Relevant log output
```shell
MCP TOOLS: [{'type': 'function', 'function': {'name': 'tavily-search', 'description': "A powerful web search tool that provides comprehensive, real-time results using Tavily's AI search engine. Returns relevant web content with customizable parameters for result count, content type, and domain filtering. Ideal for gathering current information, news, and detailed web content analysis.", 'parameters': {'type': 'object', 'properties': {'query': {'type': 'string', 'description': 'Search query'}, 'search_depth': {'type': 'string', 'enum': ['basic', 'advanced'], 'description': "The depth of the search. It can be 'basic' or 'advanced'", 'default': 'basic'}, 'topic': {'type': 'string', 'enum': ['general', 'news'], 'description': 'The category of the search. This will determine which of our agents will be used for the search', 'default': 'general'}, 'days': {'type': 'number', 'description': "The number of days back from the current date to include in the search results. This specifies the time frame of data to be retrieved. Please note that this feature is only available when using the 'news' search topic", 'default': 3}, 'time_range': {'type': 'string', 'description': "The time range back from the current date to include in the search results. This feature is available for both 'general' and 'news' search topics", 'enum': ['day', 'week', 'month', 'year', 'd', 'w', 'm', 'y']}, 'max_results': {'type': 'number', 'description': 'The maximum number of search results to return', 'default': 10, 'minimum': 5, 'maximum': 20}, 'include_images': {'type': 'boolean', 'description': 'Include a list of query-related images in the response', 'default': False}, 'include_image_descriptions': {'type': 'boolean', 'description': 'Include a list of query-related images and their descriptions in the response', 'default': False}, 'include_raw_content': {'type': 'boolean', 'description': 'Include the cleaned and parsed HTML content of each search result', 'default': False}, 'include_domains': {'type': 'array', 'items': {'type': 'string'}, 'description': 'A list of domains to specifically include in the search results, if the user asks to search on specific sites set this to the domain of the site', 'default': []}, 'exclude_domains': {'type': 'array', 'items': {'type': 'string'}, 'description': 'List of domains to specifically exclude, if the user asks to exclude a domain set this to the domain of the site', 'default': []}}, 'required': ['query']}, 'strict': False}}, {'type': 'function', 'function': {'name': 'tavily-extract', 'description': 'A powerful web content extraction tool that retrieves and processes raw content from specified URLs, ideal for data collection, content analysis, and research tasks.', 'parameters': {'type': 'object', 'properties': {'urls': {'type': 'array', 'items': {'type': 'string'}, 'description': 'List of URLs to extract content from'}, 'extract_depth': {'type': 'string', 'enum': ['basic', 'advanced'], 'description': "Depth of extraction - 'basic' or 'advanced', if usrls are linkedin use 'advanced' or if explicitly told to use advanced", 'default': 'basic'}, 'include_images': {'type': 'boolean', 'description': 'Include a list of images extracted from the urls in the response', 'default': False}}, 'required': ['urls']}, 'strict': False}}, {'type': 'function', 'function': {'name': 'tavily-crawl', 'description': 'A powerful web crawler that initiates a structured web crawl starting from a specified base URL. The crawler expands from that point like a tree, following internal links across pages. You can control how deep and wide it goes, and guide it to focus on specific sections of the site.', 'parameters': {'type': 'object', 'properties': {'url': {'type': 'string', 'description': 'The root URL to begin the crawl'}, 'max_depth': {'type': 'integer', 'description': 'Max depth of the crawl. Defines how far from the base URL the crawler can explore.', 'default': 1, 'minimum': 1}, 'max_breadth': {'type': 'integer', 'description': 'Max number of links to follow per level of the tree (i.e., per page)', 'default': 20, 'minimum': 1}, 'limit': {'type': 'integer', 'description': 'Total number of links the crawler will process before stopping', 'default': 50, 'minimum': 1}, 'query': {'type': 'string', 'description': 'Natural language instructions for the crawler'}, 'select_paths': {'type': 'array', 'items': {'type': 'string'}, 'description': 'Regex patterns to select only URLs with specific path patterns (e.g., /docs/.*, /api/v1.*)', 'default': []}, 'select_domains': {'type': 'array', 'items': {'type': 'string'}, 'description': 'Regex patterns to select crawling to specific domains or subdomains (e.g., ^docs\\.example\\.com$)', 'default': []}, 'allow_external': {'type': 'boolean', 'description': 'Whether to allow following links that go to external domains', 'default': False}, 'categories': {'type': 'array', 'items': {'type': 'string', 'enum': ['Careers', 'Blog', 'Documentation', 'About', 'Pricing', 'Community', 'Developers', 'Contact', 'Media']}, 'description': 'Filter URLs using predefined categories like documentation, blog, api, etc', 'default': []}, 'extract_depth': {'type': 'string', 'enum': ['basic', 'advanced'], 'description': 'Advanced extraction retrieves more data, including tables and embedded content, with higher success but may increase latency', 'default': 'basic'}}, 'required': ['url']}, 'strict': False}}, {'type': 'function', 'function': {'name': 'tavily-map', 'description': 'A powerful web mapping tool that creates a structured map of website URLs, allowing you to discover and analyze site structure, content organization, and navigation paths. Perfect for site audits, content discovery, and understanding website architecture.', 'parameters': {'type': 'object', 'properties': {'url': {'type': 'string', 'description': 'The root URL to begin the mapping'}, 'max_depth': {'type': 'integer', 'description': 'Max depth of the mapping. Defines how far from the base URL the crawler can explore', 'default': 1, 'minimum': 1}, 'max_breadth': {'type': 'integer', 'description': 'Max number of links to follow per level of the tree (i.e., per page)', 'default': 20, 'minimum': 1}, 'limit': {'type': 'integer', 'description': 'Total number of links the crawler will process before stopping', 'default': 50, 'minimum': 1}, 'query': {'type': 'string', 'description': 'Natural language instructions for the crawler'}, 'select_paths': {'type': 'array', 'items': {'type': 'string'}, 'description': 'Regex patterns to select only URLs with specific path patterns (e.g., /docs/.*, /api/v1.*)', 'default': []}, 'select_domains': {'type': 'array', 'items': {'type': 'string'}, 'description': 'Regex patterns to select crawling to specific domains or subdomains (e.g., ^docs\\.example\\.com$)', 'default': []}, 'allow_external': {'type': 'boolean', 'description': 'Whether to allow following links that go to external domains', 'default': False}, 'categories': {'type': 'array', 'items': {'type': 'string', 'enum': ['Careers', 'Blog', 'Documentation', 'About', 'Pricing', 'Community', 'Developers', 'Contact', 'Media']}, 'description': 'Filter URLs using predefined categories like documentation, blog, api, etc', 'default': []}}, 'required': ['url']}, 'strict': False}}]
LLM RESPONSE: "ModelResponse(id='chatcmpl-10c94bda-c04c-4447-9db7-5c7e45a2e413', created=1747182311, model='ollama/hf.co/DevQuasar/Salesforce.Llama-xLAM-2-8b-fc-r-GGUF:Q8_0', object='chat.completion', system_fingerprint=None, choices=[Choices(finish_reason='tool_calls', index=0, message=Message(content=None, role='assistant', tool_calls=[ChatCompletionMessageToolCall(function=Function(arguments='{\"query\": \"newest Salesforce model\"}', name='tavily-search'), id='call_a6a7ed64-1eab-486c-ad08-48b3143c11ff', type='function')], function_call=None, provider_specific_fields=None))], usage=Usage(completion_tokens=22, prompt_tokens=1840, total_tokens=1862, completion_tokens_details=None, prompt_tokens_details=None))"
Exception ignored in: <function BaseSubprocessTransport.__del__ at 0x0000012D73FB04A0>
Traceback (most recent call last):
File "C:\Users\Fractal\AppData\Roaming\uv\python\cpython-3.11.11-windows-x86_64-none\Lib\asyncio\base_subprocess.py", line 125, in __del__
_warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Fractal\AppData\Roaming\uv\python\cpython-3.11.11-windows-x86_64-none\Lib\asyncio\base_subprocess.py", line 78, in __repr__
info.append(f'stdout={stdout.pipe}')
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Fractal\AppData\Roaming\uv\python\cpython-3.11.11-windows-x86_64-none\Lib\asyncio\proactor_events.py", line 80, in __repr__
info.append(f'fd={self._sock.fileno()}')
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Fractal\AppData\Roaming\uv\python\cpython-3.11.11-windows-x86_64-none\Lib\asyncio\windows_utils.py", line 102, in fileno
raise ValueError("I/O operation on closed pipe")
ValueError: I/O operation on closed pipe
Exception ignored in: <function _ProactorBasePipeTransport.__del__ at 0x0000012D73FB1D00>
Traceback (most recent call last):
File "C:\Users\Fractal\AppData\Roaming\uv\python\cpython-3.11.11-windows-x86_64-none\Lib\asyncio\proactor_events.py", line 116, in __del__
_warn(f"unclosed transport {self!r}", ResourceWarning, source=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Fractal\AppData\Roaming\uv\python\cpython-3.11.11-windows-x86_64-none\Lib\asyncio\proactor_events.py", line 80, in __repr__
info.append(f'fd={self._sock.fileno()}')
^^^^^^^^^^^^^^^^^^^
File "C:\Users\Fractal\AppData\Roaming\uv\python\cpython-3.11.11-windows-x86_64-none\Lib\asyncio\windows_utils.py", line 102, in fileno
raise ValueError("I/O operation on closed pipe")
ValueError: I/O operation on closed pipe
Are you a ML Ops Team?
No
What LiteLLM version are you on ?
v1.69.2
Twitter / LinkedIn details
No response
This looks like an issue with the MCP SDK, and I have made a ticket with them there. This is still a litellm issue with the experimental mcp client because if i leave the function empty nothing happens.
Anecdotally, I've seen this without anything MCP-related (i.e., on a straight call to an LLM). Unfortunately, attempts to consistently reproduce it have been challenging...
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}