big-ann-benchmarks icon indicating copy to clipboard operation
big-ann-benchmarks copied to clipboard

Support for remote servers

Open wahajali opened this issue 1 month ago • 1 comments

As I understand big-ann-benchmark is used for benchmarking algorithms. Looking at ann-benchmarks, they also have support for databases, such as postgres (pgvector) and redis. I have a few questions:

  1. Can something similar be done with the big-ann-benchmark. For example, we add a docker container for pgvector, and have ann-benchmark test against it?
  2. As a follow up to [1] - in case I want to test out a server running remotely, would that be possible with big-ann-benchmark. For example, if I have a database server running remotely that I want to benchmark. Per my understanding, all test are currently run in the same hardware where the benchmark is.
  3. For each test/track there is a baseline algo defined. I'm trying to understand what is intended by the baseline. Is this the index that candidates are supposed to build on top of? For example, for the Neurips 23 ood track, the basaline is DiskAnn, why is it that the Dockerfile for pinecode-ood also includes the DiskAnn py lib to build the index?

wahajali avatar May 20 '24 11:05 wahajali