raft
raft copied to clipboard
[FEA] Support index building from host dataset for IVF-FLAT
Is your feature request related to a problem? Please describe. Currently only device datasets are allowed for extending the index (which is part of the build process). The goal is to accept input data from the host, so that we have more free device memory for building the index.
Describe the solution you'd like Similar to IVF-PQ, one could process the input data batch wise, and copy that to GPU memory in smaller batches.
Just ran into this issue with the knowhere integration, as well. Would be a good thing to fix for consistency across ANN methods.