LangBot icon indicating copy to clipboard operation
LangBot copied to clipboard
verified publisher icon RockChinQ

Add external knowledge base support for HTTP-based RAG retrieval

Open Copilot opened this issue 2 months ago • 2 comments

概述 / Overview

Extends RAG capabilities with external knowledge bases via HTTP API. Built-in knowledge bases had limited retrieval quality; external KBs allow integration with superior third-party retrieval services (e.g., Dify-compatible APIs).

实现细节 / Implementation Details

Backend Architecture:

  • Abstract KnowledgeBaseInterface for polymorphic KB handling (internal/external)
  • ExternalKnowledgeBase implementation with async HTTP client, Bearer auth support
  • ExternalKnowledgeBase persistence table (uuid, name, api_url, api_key, top_k)
  • Full CRUD endpoints at /api/v1/knowledge/external-bases
  • RAGManager loads both KB types; pipelines retrieve from mixed sources seamlessly

Frontend UI:

  • Tab navigation: Built-in | External (styled to match plugins page design)
  • External KB card list with create/edit dialog
  • Form fields: name, description, API URL, API key (optional), top_k
  • HTTP client methods for external KB operations
  • i18n: EN, ZH-Hans, ZH-Hant, JA

API Contract:

POST {api_url}
Authorization: Bearer {api_key}
Content-Type: application/json

{"query": "text", "top_k": 5}

Response: {"records": [{"content": "...", "score": 0.95, "title": "...", "metadata": {}}]}

Backward Compatibility:

  • Existing internal KBs unchanged
  • Pipeline knowledge-bases array field already supported
  • Legacy single knowledge-base string still works
  • Auto-creates tables on startup

检查清单 / Checklist

PR 作者完成 / For PR author

  • [x] 阅读仓库贡献指引了吗？ / Have you read the contribution guide?
  • [x] 与项目所有者沟通过了吗？ / Have you communicated with the project maintainer?
  • [x] 我确定已自行测试所作的更改，确保功能符合预期。 / I have tested the changes and ensured they work as expected.

项目维护者完成 / For project maintainer

  • [ ] 相关 issues 链接了吗？ / Have you linked the related issues?
  • [ ] 配置项写好了吗？迁移写好了吗？生效了吗？ / Have you written the configuration items? Have you written the migration? Has it taken effect?
  • [ ] 依赖加到 pyproject.toml 和 core/bootutils/deps.py 了吗 / Have you added the dependencies to pyproject.toml and core/bootutils/deps.py?
  • [ ] 文档编写了吗？ / Have you written the documentation?
Original prompt

我们内置的知识库（RAG）检索算法比较简单，效果质量并不好；我们希望能通过“外部知识库”来扩展能力，外部知识库即通过http来检索文档，只需要使用到检索接口；我们假设外部知识库会提供与：https://docs.dify.ai/zh-hans/guides/knowledge-base/api-documentation/external-knowledge-api-documentation 文档定义的格式一样的接口，请你设计支持外部知识库；

请在知识库页面左上角加一个tab list，可以参考插件页的用法；分为两个tab：内置、外部知识库；当切换到 外部知识库时，显示外部知识库的列表，最左上角的卡片是 添加卡片 （可以参考其他页面的组件），点击添加卡片即弹出对话框，供用户输入外部知识库的地址；

后端中，请你做好抽象，在内置知识库和外部知识库上方加一个抽象层？在流水线配置的选择知识库时，需要支持同时选择多个内置知识库或外部知识库；流水线检索时，内部知识库就是使用内部的rag逻辑，外部知识库则直接访问外部知识库的检索api；

注意：上方那个文档仅供参考，请根据 LangBot 内部所需的字段和具体情况来对外部知识库的接口做出定义；

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Copilot avatar Nov 16 '25 14:11 Copilot

@copilot 知识库页面的左上角tab list样式很难看，请参考插件页面的那个tab list样式实现

RockChinQ avatar Nov 16 '25 15:11 RockChinQ

@copilot 知识库页面的左上角tab list样式很难看，请参考插件页面的那个tab list样式实现

Updated the tab list styling to match the plugins page. Changes in commit abe9795:

  • Added flex container with justify-between and px-[0.8rem] padding
  • Applied shadow-md, py-5, and bg-[#f0f0f0] dark:bg-[#2a2a2e] to TabsList
  • Added px-6 py-4 cursor-pointer to each TabsTrigger

The knowledge base tabs now have the same visual style as the plugins page tabs.

Copilot avatar Nov 16 '25 15:11 Copilot