raft icon indicating copy to clipboard operation
raft copied to clipboard

[FEA] Support index building from host dataset for IVF-FLAT

Open tfeher opened this issue 2 years ago • 1 comments

Is your feature request related to a problem? Please describe. Currently only device datasets are allowed for extending the index (which is part of the build process). The goal is to accept input data from the host, so that we have more free device memory for building the index.

Describe the solution you'd like Similar to IVF-PQ, one could process the input data batch wise, and copy that to GPU memory in smaller batches.

tfeher avatar Jan 25 '23 17:01 tfeher

Just ran into this issue with the knowhere integration, as well. Would be a good thing to fix for consistency across ANN methods.

wphicks avatar Jan 08 '24 21:01 wphicks