Discussion on Future Directions

[maumueller]

Dear all,

<tl;dr> Please add your thoughts on the future of this benchmark!

Thank you very much for participating in our NeurIPS'21 competition. The competition will end with an event on Dec 8, and you can find the timeline for this event on https://big-ann-benchmarks.com/. We hope many of you will be able to participate!

The last part of the event will be an open discussion among the participants for future directions of this competition. As organizers we have already identified some points we would like to discuss and potentially include in a future version of the benchmark.

1. Filtered ANNS: can you support ANNS queries which allow filters like date range, author or some combination of attributes. This would look like a simple SQL + ANNS query.

2. Streaming ANNS: Can algorithms be robust to insertions and deletions. Here we have a strong baseline (fresh-diskann: https://arxiv.org/abs/2105.09613)

3. Out of distribution queries: this is already a problem with T2I and we can imagine various variations

4. Better vector compression: Most approaches use some variant of product quantization as vector compression, but can we get more accurate estimation, maybe at the price of more expensive decoding?

Please let us know what you think about these topics, and add your own!

Thanks!

[fuchun-wang]

Track1 is a general, classic setting track. It's easy to follow for some fresh learner except the huuuuuuuge dataset, and loooooong running time. Maybe we can build a mini-track1 and benchmark tools to verify ideas.

Really thank you for the great event.

[NJU-yasuo]

I wonder if it's free to use any algorithm in T1/T2 instead of being confined to FAISS/DISKANN ?

[fuchun-wang]

Use any algorithm you want is ok.

[fuchun-wang]

我觉得是你想用啥就用啥，不用必须用 faiss。

NJU-yasuo @.***> 于2021年12月7日周二 18:18写道：

I wonder if it's free to use any algorithm in T1/T2 instead of being confined to FAISS/DISKANN ?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/harsha-simhadri/big-ann-benchmarks/issues/90#issuecomment-987775466, or unsubscribe https://github.com/notifications/unsubscribe-auth/AU3IZOB2UVJGL4KWKHOPIFLUPXNOPANCNFSM5JMQTPWQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[markwhen]

Hi @maumueller , is there future competition for these new topics?