feat: Enhance Memory System with Milvus VectorDB's Hybrid Search Capa…

[jinhonglin-ryan]

Description

This PR introduces native full-text hybrid search support to the Milvus integration in the memory system. By combining semantic vector search with native BM25-based full-text search, the system now supports more accurate and context-aware memory retrieval with configurable search weights.

Changes

• Added hybrid_search_config configuration dictionary to MilvusStorage class
• Implemented BM25 text search capability alongside vector similarity search
• Created configurable weighted hybrid search with customizable balance between vector and text relevance
• Updated collection schema to support sparse (text) indexing
• Improved variable naming for better code clarity

Key Benefits

• Configurable hybrid search: Fine-tune the balance between vector similarity and text relevance
• Native full-text search: Utilizes Milvus's built-in BM25 capability
• Improved memory retrieval quality: Combines vector similarity with text relevance for more accurate results
• Flexible configuration: Simple dictionary-based configuration for all hybrid search settings
• Milvus-specific optimization: Automatically falls back to vector-only mode when hybrid search is disabled

Usage Example

Users can enable and configure hybrid search with a simple configuration:

memory = LongtermAgentMemory(
    context_creator=ScoreBasedContextCreator(
        token_counter=OpenAITokenCounter(ModelType.GPT_4O_MINI),
        token_limit=1024,
    ),
    chat_history_block=ChatHistoryBlock(),
    vector_db_block=VectorDBBlock(
        embedding=embed,
        storage=MilvusStorage(
            vector_dim=embed.get_output_dim(),
            url_and_api_key=("your-milvus-uri", "your-milvus-token"),
            collection_name="your_collection_name",
            hybrid_search_config={
                'enable': True,               # Enable hybrid search
                'vector_weight': 0.7,         # Weight for vector similarity (70%)
                'text_weight': 0.3            # Weight for text relevance (30%)
            }
        ),
    ),
)

Technical Notes

• Uses Milvus-native BM25 for sparse text indexing
• Hybrid scoring performed via weighted ranking with configurable weights
• Default configuration: 50% vector similarity, 50% text relevance when enabled
• Simple dictionary-based configuration allows for future extensions
• Improved internal variable naming for better code clarity and maintainability

[jinhonglin-ryan]

@X-TRON404 @Wendong-Fan Hi, I accidentally lost the previous PR after force-pushing master while resolving a rebase conflict. I’m opening a new PR with the updated commits — apologies for the confusion! Could you take at this again real quick? Thanks!

[X-TRON404]

@X-TRON404 @Wendong-Fan Hi, I accidentally lost the previous PR after force-pushing master while resolving a rebase conflict. I’m opening a new PR with the updated commits — apologies for the confusion! Could you take at this again real quick? Thanks!

Hey @jinhonglin-ryan , no worries! I will review the PR.