VectorDBBench icon indicating copy to clipboard operation
VectorDBBench copied to clipboard

feat(oss-opensearch): Add KNN derived source configuration option

Open Akhil-Pathivada opened this issue 1 month ago • 3 comments

Problem

OpenSearch 3.x+ enables knn.derived_source.enabled by default, which allows OpenSearch to derive vectors from the vector values file instead of storing them in the _source field. However, users need a way to control this behavior for their specific use cases:

  • When working with different OpenSearch versions because this feature was not be available (< 3.x)
  • When they want to disable this feature to store vectors in _source for specific requirements

Solution

Added a new configurable option knn_derived_source_enabled allowing users to:

  • "None" (default) - Skip setting for OpenSearch < 3.x compatibility
  • "True" - Explicitly enable (3.x+ default behavior)
  • "False" - Disable for better performance on 3.x+

Changes

  • Added knn_derived_source_enabled: str | None = None field to OSSOpenSearchIndexConfig
  • Conditional setting in _create_index() - only added when explicitly set
  • UI dropdown with 3 options and version guidance
  • Added enum to CaseConfigParamType

Benefits

  • ✅ Version-compatible (OpenSearch < 3.x and 3.x+)
  • ✅ User-friendly UI with clear guidance
  • ✅ No breaking changes - default behavior unchanged

Files Changed

  • vectordb_bench/backend/clients/oss_opensearch/config.py
  • vectordb_bench/models.py
  • vectordb_bench/backend/clients/oss_opensearch/oss_opensearch.py
  • vectordb_bench/frontend/config/dbCaseConfigs.py

Screenshots

Screenshot 2025-11-07 at 17 04 59

Akhil-Pathivada avatar Nov 07 '25 11:11 Akhil-Pathivada

/assign @alwayslove2013

Akhil-Pathivada avatar Nov 07 '25 12:11 Akhil-Pathivada

Hi @alwayslove2013 I've opened a foundational PR https://github.com/zilliztech/VectorDBBench/pull/635, which introduces a new version-aware system for OpenSearch settings.

I'm converting this PR to a draft as it depends on the foundational system in https://github.com/zilliztech/VectorDBBench/pull/635. Once that merges, I'll update this branch with the new design and mark it as ready for review.

Akhil-Pathivada avatar Nov 08 '25 22:11 Akhil-Pathivada

@alwayslove2013 Can you help reviewing this?

Akhil-Pathivada avatar Dec 04 '25 15:12 Akhil-Pathivada

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Akhil-Pathivada, alwayslove2013 To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed. You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot avatar Dec 05 '25 08:12 sre-ci-robot