Add configurable knowledgebase management

[3coins]

Description

This issue proposes new components for adding and managing Knowledgebases in Jupyter AI, allowing users to attach external knowledgebases and target questions to specific datasets or files. This includes creating configurable KnowledgeBases with retriever classes, a selection UI knowledgebases for targeting questions, and a knowledgebases manager for creating and editing knowledgebases.

Problem

Knowledgebases are a common component used in augmenting and grounding the LLM with contextual data, for example local files, enterprise data, wikis etc. Jupyter AI currently implements this in the form of a /learn slash command that allows adding the file data to a vector store, so that users can /ask relevant questions based on the loaded files. There are several limitations with the current /learn feature:

1. Users cannot attach an external knowledgebase that might already have the data available. Common examples of these are enterprise or company data data that might be available readily to be consumed and is quite large in size to be loaded in Jupyter AI individually by users.
2. Users cannot ask a question targeting a specific dataset or files. For example, users might be only interested in asking questions about a specific paper. This is currently not possible as raised in https://github.com/jupyterlab/jupyter-ai/issues/907 without completely deleting the existing learned files and re-learning with the target file(s).
3. Users have to find the location of the files/folders they want Jupyter AI to learn, copy their path, and type their location with the /learn command.

Proposed Solution

• [ ] A new configurable KnowledgeBases that provides a list of knowledge base classes to add. This will be in addition to the default knowledge base Jupyter AI provides.
• [ ] A selection UI for knowledge bases within the chat interface so the users can select one or more knowledge bases to target the question to. As suggested by @dlqqq, this can eliminate the /ask command, as we can infer the behavior if a selection is made and switch to /ask implicitly.
• [ ] A knowledge bases manager that allows users to create, edit and manage knowledge bases. This will provide users with features like drag-n-drop files to create/add to a knowledge base. This will improve the current /learn experience in chat, where users have to find and type the location of the files/folders they want Jupyter AI to learn.