RedisAI
Provide GPU-accelerated vector indexes with RAFT
This PR is a replacement for #377, which fell out of date with changes on main. This PR provides access to GPU-accelerated index types provided by the RAPIDS RAFT library. It introduces two new index types: IVF and Tiered IVF, which provide access to both IVF Flat and IVFPQ indexes depending on how they are configured.
The intention of this initial integration is to provide GPU-based performance improvements for building indexes and batched searches. A later PR will introduce another index type which will offer perf improvement (relative to CPU HNSW) for single-query searches as well.
Main objects this PR modified
This PR does not substantively modify existing objects. A clear method is added to the brute force index to allow it to be reset with a single call.
This PR is still a work in progress. Remaining tasks:
- [ ] Update index-to-index transfer in tiered index to make use of asynchronous job infrastructure (Work on this was started but not completed)
- [ ] Add vector deletion once enabled by RAFT (https://github.com/rapidsai/raft/issues/1565)
- [ ] Add tests
- [ ] Add source files instantiating the index types provided by the current headers
- [ ] Switch back to mainline RAFT after merge of https://github.com/rapidsai/raft/pull/1716
Tasks to be tackled in later PRs:
- Integrating CAGRA indexes
- Multivalue support
- Vector update support
Questions for this PR:
- Should the IVF indexes be named in such a way as to indicate that they are GPU indexes in order to allow for possible CPU IVF support in the future?
- Based on initial feedback for 377, I have merged IVF Flat and IVF PQ into a single index type, with the choice of algorithm determined by the configuration parameters. Does this design meet the needs mentioned, or do we prefer the earlier separation?
Mark if applicable
- [ ] This PR introduces API changes
- [ ] This PR introduces serialization changes
