Benchmark for vector indexing and searching

[PragmaTwice]

Search before asking

• [X] I had searched in the issues and found no similar issues.

Motivation

We can try to use this repo: https://github.com/qdrant/vector-db-benchmark

After some simple patching:

diff --git a/engine/clients/redis/configure.py b/engine/clients/redis/configure.py
index a5e6fe8..b2b8af0 100644
--- a/engine/clients/redis/configure.py
+++ b/engine/clients/redis/configure.py
@@ -7,6 +7,7 @@ from redis.commands.search.field import (
     TextField,
     VectorField,
 )
+from redis.commands.search.indexDefinition import IndexDefinition, IndexType
 
 from benchmark.dataset import Dataset
 from engine.base_client.configure import BaseConfigurator
@@ -81,7 +82,7 @@ class RedisConfigurator(BaseConfigurator):
                 name="vector",
                 algorithm="HNSW",
                 attributes={
-                    "TYPE": "FLOAT32",
+                    "TYPE": "FLOAT64",
                     "DIM": dataset.config.vector_size,
                     "DISTANCE_METRIC": self.DISTANCE_MAPPING[dataset.config.distance],
                     **self.collection_params.get("hnsw_config", {}),
@@ -97,8 +98,13 @@ class RedisConfigurator(BaseConfigurator):
             ]
         for conn in conns:
             search_namespace = conn.ft()
+            idef = IndexDefinition(
+                prefix=[''],
+                index_type=IndexType.HASH
+            )
+            idef.args = idef.args[:-2]
             try:
-                search_namespace.create_index(fields=index_fields)
+                search_namespace.create_index(fields=index_fields, definition=idef)
             except redis.ResponseError as e:
                 if "Index already exists" not in str(e):
                     raise e
diff --git a/engine/clients/redis/upload.py b/engine/clients/redis/upload.py
index cd4b888..0e29533 100644
--- a/engine/clients/redis/upload.py
+++ b/engine/clients/redis/upload.py
@@ -57,7 +57,7 @@ class RedisUploader(BaseUploader):
             cls.client.hset(
                 str(idx),
                 mapping={
-                    "vector": np.array(vec).astype(np.float32).tobytes(),
+                    "vector": np.array(vec).astype(np.float64).tobytes(),
                     **payload,
                     **geopoints,
                 },

We can start a kvrocks instance and run this benchmark:

python3 run.py --engines redis-default --datasets 'glove-25-angular'

cc @Beihao-Zhou

Solution

Currently we'll get some coredumps in the vector indexing phase, e.g.:

E20240809 14:55:31.865105   134 signal_util.h:36] Stack trace (most recent call first):
#0  0x0000555f0502cac9 in SegvHandler at /usr/bin/kvrocks
#1  0x00007f2e599c004f in __sigaction at /lib/x86_64-linux-gnu/libc.so.6
#2  0x00007f2e59a0ee2c in pthread_key_delete at /lib/x86_64-linux-gnu/libc.so.6
#3  0x00007f2e599bffb1 in gsignal at /lib/x86_64-linux-gnu/libc.so.6
#4  0x00007f2e599aa471 in abort at /lib/x86_64-linux-gnu/libc.so.6
#5  0x0000555f04f77917 in __gnu_cxx::__verbose_terminate_handler() [clone .cold] at vterminate.o
#6  0x0000555f05b16c09 in __cxxabiv1::__terminate(void (*)()) at /usr/bin/kvrocks
#7  0x0000555f05b16c74 in std::terminate() at /usr/bin/kvrocks
#8  0x0000555f05b16dc7 in __cxa_throw at /usr/bin/kvrocks
#9  0x0000555f04f7908d in std::__throw_bad_alloc() at /usr/bin/kvrocks
#10 0x0000555f04f77509 in handleOOM(unsigned long, bool) [clone .cold] at /kvrocks/build/_deps/jemalloc-src/src/jemalloc_cpp.cpp:90
#11 0x0000555f051d6e4a in redis::VectorItem::VectorItem(redis::VectorItem const&) at /usr/bin/kvrocks
#12 0x0000555f051e561a in redis::IndexUpdater::UpdateHnswVectorIndex(std::basic_string_view<char, std::char_traits<char> >, kqir::Value const&, kqir::Value const&, redis::SearchKey const&, redis::HnswVectorFieldMetadata*) const at /usr/bin/kvrocks
#13 0x0000555f051ec7be in redis::IndexUpdater::Update(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, kqir::Value, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, kqir::Value> > > const&, std::basic_string_view<char, std::char_traits<char> >) const at /usr/bin/kvrocks
#14 0x0000555f05228549 in redis::Connection::ExecuteCommands(std::deque<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::allocator<std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > >*) at
 /usr/bin/kvrocks
#15 0x0000555f0505db30 in EvbufCallbackBase<redis::Connection, true, true, true>::readCB(bufferevent*, void*) at /usr/bin/kvrocks
#16 0x0000555f05978a0d in bufferevent_run_deferred_callbacks_unlocked at bufferevent.c
#17 0x0000555f059808a3 in event_process_active_single_queue at event.c
#18 0x0000555f05980f3e in event_process_active at event.c
#19 0x0000555f05982cf8 in event_base_loop.constprop.0 at event.c
#20 0x0000555f0523d46d in std::thread::_State_impl<std::thread::_Invoker<std::tuple<util::CreateThread<WorkerThread::Start()::{lambda()#1}>(char const*, WorkerThread::Start()::{lambda()#1})::{lambda()#1}> > >::_M_run() at /usr/bin/kvrocks
#21 0x0000555f05b96312 in execute_native_thread_routine at thread.o
#22 0x00007f2e59a0d133 in pthread_condattr_setpshared at /lib/x86_64-linux-gnu/libc.so.6
#23 0x00007f2e59a8ca3f in __clone at /lib/x86_64-linux-gnu/libc.so.6

Maybe we can try to solve it first.

Are you willing to submit a PR?

• [x] I'm willing to submit a PR!