Choose to run an `mcp tool_call` from specific `mcp` server

[zigelboim-misha]

trafficstars

Choose to run an mcp tool_call from specific mcp server

Allowing the customers to choose on what mcp server to run their tool_calls.

Why

In case when you use the same mcp server but with different parameters (like having 2 GitHub mcp servers - one for each organization):

mcp_servers:
    {
        "ben-github": {
            "url": "http://github-mcp-server-a.mcp.svc.cluster.local:8000/sse",
        },
        "misha-github": {
            "url": "http://github-mcp-server-b.mcp.svc.cluster.local:8000/sse",
        }
    }

This is how it looks in the ui:

When I fetch all tools i get the following expected result:

[
    {
        "name": "get_me",
        "description": "Get details of the authenticated GitHub user. Use this when a request include \"me\", \"my\"...",
        "inputSchema": {
            "type": "object",
            "properties": {
                "reason": {
                    "description": "Optional: reason the session was created",
                    "type": "string"
                }
            }
        },
        "mcp_info": {
            "server_name": "ben-github"
        }
    },
    {
        "name": "get_me",
        "description": "Get details of the authenticated GitHub user. Use this when a request include \"me\", \"my\"...",
        "inputSchema": {
            "type": "object",
            "properties": {
                "reason": {
                    "description": "Optional: reason the session was created",
                    "type": "string"
                }
            }
        },
        "mcp_info": {
            "server_name": "misha-github"
        }
    }
]

Before this PR

And when I want to call one of these tools, it only uses the latest mcp server:

curl -X 'POST' "http://localhost:4000/mcp/tools/call" \
     -H "Content-Type: application/json" \
     -H "Accept: */*" \
     -d '{
            "name": "get_me",
            "arguments": {"reason":"test"},
        }'

After this PR

I want to allow the following curl command:

curl -X 'POST' "http://localhost:4000/mcp/tools/call" \
     -H "Content-Type: application/json" \
     -H "Accept: */*" \
     -d '{
            "name": "get_me",
            "arguments": {"reason":"test"},
            "mcp_server": "misha-github"
        }'

Relevant issues

• Fixes https://github.com/BerriAI/litellm/issues/10404

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• [ ] I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• [ ] I have added a screenshot of my new test passing locally
• [ ] My PR passes all unit tests on make test-unit
• [ ] My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

🆕 New Feature 🐛 Bug Fix 🧹 Refactoring 📖 Documentation 🚄 Infrastructure ✅ Test

Changes

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
litellm	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 8, 2025 11:29am

[CLAassistant]

All committers have signed the CLA.

[zigelboim-misha]

I built it locally and ran it with our 2 GitHub mcp servers and it worked:

Not breaking

curl -X 'POST' "http://localhost:4000/mcp/tools/call" \
     -H "Content-Type: application/json" \
     -H "Accept: */*" \
     -d '{
            "name": "get_me",
            "arguments": {"reason":"test"},
        }'

Its using the last mcp server as it used to do before the update:

[{"type":"text","text":"{\"login\":\"Mishaxxx\",\"id\":xxxxx,\"node_id\":\"xxxx\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/xxxxx?v=4\",\"html_url\":\"https://github.com/Mishaxxx\",\"gravatar_id\":\"\",\"name\":\"Mishaxxx\",\"blog\":\"\",\"bio\":\"xxx\",\"public_repos\":11,\"public_gists\":5,\"followers\":1,\"following\":3,\"created_at\":\"2017-09-11T18:15:14Z\",\"updated_at\":\"2025-04-16T17:57:36Z\",\"type\":\"User\",\"site_admin\":false,\"url\":\"https://api.github.com/users/Mishaxxx\",\"events_url\":\"https://api.github.com/users/Mishaxxx/events{/privacy}\",\"following_url\":\"https://api.github.com/users/Mishaxxx/following{/other_user}\",\"followers_url\":\"https://api.github.com/users/Mishaxxx/followers\",\"gists_url\":\"https://api.github.com/users/Mishaxxx/gists{/gist_id}\",\"organizations_url\":\"https://api.github.com/users/Mishaxxx/orgs\",\"received_events_url\":\"https://api.github.com/users/Mishaxxx/received_events\",\"repos_url\":\"https://api.github.com/users/Mishaxxx/repos\",\"starred_url\":\"https://api.github.com/users/Mishaxxx/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Mishaxxx/subscriptions\"}","annotations":null}]

New parameter

curl -X 'POST' "http://localhost:4000/mcp/tools/call" \
     -H "Content-Type: application/json" \
     -H "Accept: */*" \
     -d '{
            "name": "get_me",
            "arguments": {"reason":"test"},
            "mcp_server": "ben-github"
        }'

[{"type":"text","text":"{\"login\":\"Benxxx\",\"id\":xxx,\"node_id\":\"xxx\",\"avatar_url\":\"https://avatars.githubusercontent.com/u/xxx?v=4\",\"html_url\":\"https://github.com/Benxxx\",\"gravatar_id\":\"\",\"name\":\"Benxxx Edgar\",\"blog\":\"\",\"public_repos\":0,\"public_gists\":0,\"followers\":0,\"following\":0,\"created_at\":\"xxx-01-22T20:05:34Z\",\"updated_at\":\"xxx-04-02T06:44:40Z\",\"type\":\"User\",\"site_admin\":false,\"url\":\"https://api.github.com/users/Benxxx\",\"events_url\":\"https://api.github.com/users/Benxxx/events{/privacy}\",\"following_url\":\"https://api.github.com/users/Benxxx/following{/other_user}\",\"followers_url\":\"https://api.github.com/users/Benxxx/followers\",\"gists_url\":\"https://api.github.com/users/Benxxx/gists{/gist_id}\",\"organizations_url\":\"https://api.github.com/users/Benxxx/orgs\",\"received_events_url\":\"https://api.github.com/users/Benxxx/received_events\",\"repos_url\":\"https://api.github.com/users/Benxxx/repos\",\"starred_url\":\"https://api.github.com/users/Benxxx/starred{/owner}{/repo}\",\"subscriptions_url\":\"https://api.github.com/users/Benxxx/subscriptions\"}","annotations":null}]

Its using the correct mcp server.

[zigelboim-misha]

@wagnerjt @matiasdev30 @ishaan-jaff Is there anything else I need to do?

I'm going to deploy my version on our cluster.

[wagnerjt]

@wagnerjt @matiasdev30 @ishaan-jaff Is there anything else I need to do?

I'm going to deploy my version on our cluster.

I'm working on adding the initial MCP Server operations including tying the db stored and config servers, and believe this can work via the alias pattern than the server id. I think @ishaan-jaff needs to review

[zigelboim-misha]

@wagnerjt Cool so could you please let me know in this thread when you have a solution and how to use it? I'm currently using my version in prod and it looks great as all the data I need to decide what mcp to use is available using your existing apis.

[krrishdholakia]

cc: @ishaan-jaff can you review this when you get a chance?

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.