[RFC] Knowledge base in OpenSearch

[yuye-aws]

Problem statement

Users in ml-commons can use RAG (Retrieval Augmented Generation) service either from an OpenSearch index or a bedrock knowledge base. The RAG service mainly consists of two steps. First, retrieve documents from the knowledge base. Second, generate answer given retrieved documents.

Despite that, both OpenSearch index and bedrock knowledge base is accessible. The configuration steps are quite different. The only method to support bedrock knowledge base is ML Model tool. It is not straightforward for users to configure their bedrock RAG service because they are required to configure some specialized functions or parameters in connector. We are considering how to provide users with a more convenient method to configure the knowledge base workflow. It would be better if user could have a unified experience to use different knowledge bases.

In the scope

• Users can customize the prompt when querying the RAG service
• Users can choose whether the knowledge base is an OpenSearch index or hosted on bedrock
• For OpenSearch index, users can specify search with dense or sparse embedding

Out of the scope

• Register or deploy the LLM model
• Register or deploy the dense or sparse embedding model
• Retrieve documents alone and not generate answers
• Configure the OpenSearch index or the bedrock knowledge base
• Search OpenSearch index with BM25 query
• Search bedrock knowledge base with non-bedrock embedding model
• Allow users to specify specify how to concatenate retrieved documents

Options

This RFC lists a few options along with their pros and cons. We expect to solicit feedbacks on or other potential better options.

Option 1

Instruct users to use bedrock knowledge base service via MLModel Tool or Connector Tool.

Pros

1. No coding effort. All we need to do is to publish some tutorials and documentation.

Cons

1. It is weird for user to use RAG service with MLModel tool.
2. More configuration effort for the users. Users need to configure some post processing steps through painless script in connector.
3. The painless script in OpenSearch may be impacted by throttling mechanism.

Option 2

Integrate bedrock knowledge base into RAGTool.

Pros

1. Unified experience. User can query index or bedrock knowledge base with a single tool.
2. No post process effort for user.

Cons

1. The parameter configuration in RAG tool would be quite different. It would be a breaking change to RAG tool in previous versions.

Option 3

Implement a new tool like RemoteRAG Tool (feel free to tell me if you can think of any better name.

Pros

1. Easy for users to distinguish between RemoteRAG tool and LocalRAG tool.
2. No post process effort for user.

Cons

1. Code duplication among RemoteRAG tool and LocalRAG tool.