Support for minimal browser-based LLM?

[kentcdodds]

Is your feature request related to a problem? Please describe.

Testing out manual requests in the inspector is fun, but filling out forms for input params and sampling responses can be a pain.

Additionally, I'm preparing educational material for MCPs and plan to use the inspector to test things out, but it helps solidify what we're doing if we can interact with what we're building using natural language. I can require attendees to pay for LLM host apps to play around with things, but that adds friction, cost, and reduces the accessibility of the material.

Describe the solution you'd like

There are some solutions (like https://www.npmjs.com/package/@mlc-ai/web-llm) for simple LLMs that can execute tools right from the browser (check this LLM tool calling demo). I think it'd be cool to have the inspector have a page for interacting with the tool with a little more natural language. Even if it's not quite what the real experience is like with a real LLM host application because it's not as powerful, it would probably make people more productive building and testing their MCPs.

Describe alternatives you've considered

I already run my MCP with Cursor and Claude Desktop, but the workflow isn't awesome and those cost money and require accounts.

Additional context

Here's a quick mock-up of what I'm imagining

[ibuildthecloud]

This would be super nice. Then the inspector could have a very basic chat interface. Clearly the models would not be great, but if you pick up a small LLM tuned for tool calling, they are nice because they are overly aggressive at calling tools, which is specifically what you'd want in a testing use case.

[QuantGeekDev]

Neat. I'm adding something similar for a hosting service - can add it to the inspector. How do we handle providers? Users will have to select a llm provider, a model, and input their api keys. Since it's a simple llm chat for testing purposes - it would be overkill to add many. maybe 2 or 3? We can have a picklist for model names but someone will have to update them every now and then as new models come out. Maybe we can solve this with a library? Anyone have a preference here?

[kentcdodds]

Actually I think we can just run this directly in the browser without any providers. It doesn't need to be a very comprehensive model for this use case.

As mentioned, I think this would handle it without external providers necessary: npmjs.com/package/@mlc-ai/web-llm

[QuantGeekDev]

I tested out web-llm. It seems to download the entire model into the browser, and supports mostly small language models which don't all support accurate tool calling. I can give local a try - but I think users might want to try it with an llm that they're more likely to use in production and without large downloads. Maybe we can get both?

[kentcdodds]

As mentioned before this actually does work: https://llm-tool-calling.nico.dev/

I'm not opposed to doing both, but I really think sticking with a simple setup that does not require providing tokens is sufficient for the inspector. If people want to test things out with a real model they can just use Cursor or Claude.

[olaservo]

Thanks for the suggestion, a few thoughts:

• Since this is largely a community-maintained project, I want to make sure that this really belongs here and won't distract too much from making the core functionality both a.) stable and b.) up to date with the latest available MCP spec and features.

• CLI mode is one way that models can already interact with Inspector, assuming they can run terminal commands. In my mind, that is the best interface for more 'automated' scenarios. @nbarraud correct me if I'm wrong but I think that the main reason why you added CLI mode is to streamline testing without filling in forms in a UI? So you could still use a local model, but keep it decoupled from this app. CLI mode doesn't do everything yet but we could focus some attention on expanding it.

• Inspector is already getting larger and more complex. To me, it makes the most sense for that complexity to be focused on its primary use case as I understand it. Otherwise I foresee issues related to the model behavior adding a lot of noise here. (Although one exception to that might be integrating a tiny LLM into handling Sampling requests.)

• Reference MCP client implementations are on the roadmap. So I could see a scenario where we could use those for demonstrating a full set of real LLM -> Server interactions, while Inspector stays focused on its current uses.

[nbarraud]

Yes that is exactly what the CLI is for.

If you have access to a coding assistant like Cursor, Windsurf, Cline, Claude Code... you can tell it to use the CLI and do whatever you need (testing an MCP server, using MCP servers to accomplish tasks, etc.)

If you don't have access to or don't want to use a coding assistant, you can go full inception mode and use a general chat assistant like Claude Desktop with a "command line" MCP tool which spawns the inspector CLI to interact with MCP servers. But that's too far-fetched if the goal is to learn about MCP. Best is to get down and dirty with the protocol directly via the web and CLI interfaces (and the docs).

[QuantGeekDev]

I feel like having an implementation of the mcp client within an actual chat in this codebase would be nice. I'm thinking that downloading several GB of model data for a test into your browser is not a good user / developer experience. People usually have OLLAMA or connect to a provider via API. I believe in OSS local llms being the future, but they're not there yet and most devs will just use openai / claude / gemini via API to mirror a production environment. I feel like "debugging with Cursor" is insufficient because their client implementation isn't complete and we can't see the code it's running so it's not that useful. Ultimately, there will be a crowd for each use-case. Taking into account what's being discussed in this issue, it seems like we'd need to support direct API for LLM providers, browser downloaded LLMs as per web-llm example, and local ollama api if we truly want this to be a useful debugging inspector

[fredericbarthelet]

In case somebody is looking for browser-based LLM support and stumbles upon this issue, we created a fork of this project to include LLM-specific features (playground, tool poisoning detection, tool confusion benchmark....) : https://github.com/alpic-ai/grizzly

[kentcdodds]

This is awesome. I was going to post an issue, but you don't have issues enabled on that repository. I'm curious what your overall plans are for your fork of the inspector.

[fredericbarthelet]

Indeed, just activated issues on the repo. Thanks for letting me know @kentcdodds :) Happy to discuss the fork direction over there or in our discord - https://discord.gg/d4Nv3JN2

[kentcdodds]

Doesn't look like anyone's watching the repo so I don't think you'll get notified of this issue: https://github.com/alpic-ai/grizzly/issues/5

[olaservo]

I created a separate issue for 'integrating LLMs' in general here since this specific feature request is related to supporting browser-based LLMs.

[kentcdodds]

Thanks @olaservo!

[olaservo]

Closing as not planned and labeling with v2 in case we want to review for next version.