[Feature]: Support HNSW SQ

[xiaofan-luan]

Is there an existing issue for this?

• [X] I have searched the existing issues

Is your feature request related to a problem? Please describe.

SQ8 and PQ are widely used in ANN search. If you want to understand more about quantization, Faiss is probably one of the best code bases to explore.

HNSW is the fastest index in the open source world, so why not make it work together with SQ and PQ to accelerate it further?

Let me know if anyone is interested and we can offer more help on it

Describe the solution you'd like.

No response

Describe an alternate solution.

No response

Anything else? (Additional Context)

No response

[noble-8]

Hi @xiaofan-luan I am interested in contributing! and would like to know how to help! let me know how to get started.

[xiaofan-luan]

cool man! I though @liliu-z could offer you some help

[xiaofan-luan]

you have any experience on cpp and some any idea about HSNW algorithm yet?

[noble-8]

Hi I had taken a cpp course in college, I have primarily worked as a Java developer ~3 ish years, so i feel that I can onboard quickly. I feel that I am comfortable working on cpp. I am really new to this algorithm but getting up to speed. I am going through this documentation here:https://www.pinecone.io/learn/hnsw/ feel free to point me to other resources. I hope this is not a dealbreaker!

[xiaofan-luan]

1. you can start from read code from HNSW in knowhere https://github.com/milvus-io/knowhere/tree/c05c8767f43eaa855c13654804d0bea9cc42c7de/src/index/hnsw
2. Once you understand hnsw, then the next step would be under stand how to do PQ, SQ. Faiss document will give your general ideas and milvus has all faiss code you can utilize https://github.com/facebookresearch/faiss/wiki.
3. Add index parameters for milvus to support PQ, SQ, which will be a trivial task

[liliu-z]

Hi @noble-8 this is on our roadmap and please feel free to make a PR for https://github.com/milvus-io/knowhere . I suggest we can start from SQ8 which is easier to implement. And more than welcome to open another issue in Knowhere for further detailed communication.

[liliu-z]

/assign @liliu-z

[noble-8]

Sounds good. Will do!

[xiaofan-luan]

@noble-8 any progress on it?

[noble-8]

i could not make any progress. i shall try again, however feel free to reassign this if i do not make a commit

[xiaofan-luan]

i could not make any progress. i shall try again, however feel free to reassign this if i do not make a commit

Sure, still thanks for the interest! I would also like to help if you are intersted

[LaPetiteSouris]

I just saw that https://github.com/milvus-io/knowhere is now archived. Wondering if this issue is still open ?

Or should the PR be addressed to https://github.com/zilliztech/Knowhere instead from now on ?

To summarize to make sure that I understand this correctly:

• (1) Knowhere already implements HNSW, which runs on top of high-dimensional vectors
• (2) The PR needs to implement SQ8 using FAISS to quantize high-dimensional vectors into compressed forms.
• (3) Somehow, make HNSW to be able tow work with quantized/compressed vectors given at (2)

Am I correct ?

[xiaofan-luan]

https://github.com/milvus-io/knowhere

It has been archived and moved to https://github.com/zilliztech/knowhere, sorry for the misunderstanding.

You are correct my man. we want to add quantization support for HNSW index and integrate with Milvus

[LaPetiteSouris]

Thanks. How urgent do you folks need this ? My C++ is rusty 😭 so it may take a while ( I have CoPilot so that helps 😭 ).

But I love this challenge.

[LaPetiteSouris]

@xiaofan-luan ~~if you folks have patience to spare, then assign this to me~~

Edit: I tried to hack around and it seems that it's a bit too much for me to take this time. I'll pick another good first issue to ramp up.

[jiaoew1991]

@xiaofan-luan I think the issue is not easy for beginners, it needs lots of knowledge 😅

[xiaofan-luan]

@xiaofan-luan I think the issue is not easy for beginners, it needs lots of knowledge 😅

Agreed you might be correct.

For SQ might be ok?

[xiaofan-luan]

But true it has to be fully understand milvus

[xiaofan-luan]

remove the good first issue

[zaobao]

I sound that HNSW-SQ8 has been available on Ziili Cloud. Is that true?

[xiaofan-luan]

Zilliz cloud don't use HSNW. we have an internal index named Cardinal~

[Monster880]

whether milvus can use hnsw pq index now ? @xiaofan-luan

[xiaofan-luan]

/assign @liliu-z

[xiaofan-luan]

@liliu-z do we have plan to support hnsw pq and sq index?

[Monster880]

@xiaofan-luan Can you provide some guidance on where modifications are needed to support HNSW PQ index??

[xiaofan-luan]

NP, I thought Li @liliu-z can help on that.

[Monster880]

@liliu-z @xiaofan-luan emmm.... where is liliu-z

[liliu-z]

@liliu-z @xiaofan-luan emmm.... where is liliu-z

Sure, there are two ways to support HNSW + Quantization:

1. Make it a new index type for Milvus
2. Treat it as HNSW with a special config.

We are adopting the first way. So the work including:

1. Add the quantization support in algorithm side and expose it as a new index type. Code should be in Knowhere
2. Let Milvus know this new Index Type

Here is an example PR for the first step. It support SQ8 for HNSW in Knowhere side.

[Monster880]

@liliu-z it seems HNSW_PQ is not using faiss but using hnswlib after quantization...

[xiaofan-luan]

Now we prefer to use hnswlib rather than faiss for hnsw, so we need to backport pq and sq feature