[Enhancement]: Materialized View: Improve Filtered Search on Partition Key

[PwzXxm]

Is there an existing issue for this?

• [X] I have searched the existing issues

What would you like to be added?

Speed up and improve recall in the case of filtered search while partition key is enabled and the expression is partition_key_field == xxx or partition_key_field in [xx, xxx, xx, ...].

Functionality

Build

• [x] Knowhere able to check whether building with additional scalar information is supported
  • https://github.com/zilliztech/knowhere/pull/346
  • https://github.com/zilliztech/knowhere/pull/476
• [x] Milvus pass this information to Knowhere while building
  • #29931
• [x] Cardinal supports (not shown)

Search

• [x] Knowhere supports search with additional config value
  • https://github.com/zilliztech/knowhere/pull/415
  • https://github.com/zilliztech/knowhere/pull/446
• [x] Milvus supports providing queried field ids and the number of touched column, also is_pure_and and is_not to knowhere
  • https://github.com/milvus-io/milvus/pull/30888
  • https://github.com/milvus-io/milvus/pull/31996
• [ ] Cardinal supports (not shown)

Metrics

• [ ] metrics

Why is this needed?

No response

Anything else?

No response

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[PwzXxm]

/reopen

[xiaofan-luan]

One other interesting idea from my mind is we can name this feature: materialized view.

Like tranditional database, user can create view based on an expresssion. Milvus will create special index on the expression, make sure that certain filtering condition is faster.

For example, we can create a view of country = "Brazil", and by creating the view we can:

1. pre generate bitset
2. build a vector index to accelerate this query

[PwzXxm]

One other interesting idea from my mind is we can name this feature: materialized view.

Like tranditional database, user can create view based on an expresssion. Milvus will create special index on the expression, make sure that certain filtering condition is faster.

For example, we can create a view of country = "Brazil", and by creating the view we can:

1. pre generate bitset
2. build a vector index to accelerate this query

Great name! In a traditional database, Materialized View creates a temporary table so that users may read data from multiple tables faster. In our case, we add more info so that users may read scalar X vector faster.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.