[Enhancement] - Supporting LLM routing by integrating Not Diamond.

[acompa]

Reference Issues

None yet.

Summary

Hi there! I am a big fan of kotaemon and would love to integrate Not Diamond into it. In case you’re unfamiliar with Not Diamond, it automatically routes your prompts to the AI model best-suited to respond to that prompt. By using Not Diamond as an AI “meta-model,” you can achieve state-of-the-art performance that beats every foundation model on every major benchmark.*

For kotaemon we would like to add our router as a component in your library, so that users can automatically route their prompts without relying on complex business logic.

I believe Not Diamond might fit best into the LLMManager logic, but what do you think? See below for an idea.

Basic Example

Here's an example implementation:

class NotDiamondLLMManager(LLMManager):
    def __init__(self, *args, **kwargs) -> None:
        from notdiamond import NotDiamond
        super(...).init(...)
        self._ndclient = NotDiamond(...)

    def get(self, key: str, default: Optional[ChatLLM] = None) -> Optional[ChatLLM]:
        # ignore provided key - interface preserved
        session_id, best_llm = self._ndclient.chat.completions.model_select(...)

        chat_llm = None
        for model_name, model in self._models:
            if model_name == best_llm:
                chat_llm = model
                break

        return chat_llm

Drawbacks

This proposed implementation might not fit your theories for automated routing of LLM prompts.

Additional information

*If you’d like to learn more about our routing, check out this example walkthrough video with Tomas, our CEO. You can also

• see how we work by using our Chat app,
• learn more at our website, or
• read our docs.

[taprosoft]

Thanks for the suggestion. We will look at this as a potential extension of LLMManager.

[acompa]

Thanks for the suggestion. We will look at this as a potential extension of LLMManager.

That's exciting to hear. I'm happy to take on this work if you're open to it. If you'd prefer to develop it yourself, I can also support your integration work when the time comes.