node-redis
node-redis copied to clipboard
Unclear how to retrieve vector
Description
I'm struggling to understand how to transform the vector output from Redis into a JavaScript array of numbers.
My application code is similar to the KNN search example https://github.com/redis/node-redis/blob/master/examples/search-knn.js
Here's the relevant snippet that stores vectors for similarity search:
function float32Buffer(arr) {
return Buffer.from(new Float32Array(arr).buffer);
}
// Add some sample data...
// https://redis.io/commands/hset/
await Promise.all([
client.hSet('noderedis:knn:a', { v: float32Buffer([0.1, 0.1]) }),
client.hSet('noderedis:knn:b', { v: float32Buffer([0.1, 0.2]) }),
client.hSet('noderedis:knn:c', { v: float32Buffer([0.1, 0.3]) }),
client.hSet('noderedis:knn:d', { v: float32Buffer([0.1, 0.4]) }),
]);
Now here is my issue: I want to get the vector by its key and be able to see the original vector as a JavaScript Array.
I tried the following:
const redisResult = await redisClient.hGet(id, 'v');
console.log(redisResult)
The output is a cryptic string like "�����'<�n;�h�?�Vٍ���/<7c��" that I believe may be [RESP](https://redis.io/docs/reference/protocol-spec/) format. I want to transform that result into its corresponding array, e.g. [0.1, 0.3]`, however I'm not sure how to do so. Any guidance would be greatly appreciated. Thanks!
I wonder if I should try using https://www.npmjs.com/package/redis-parser ...
Related StackOverflow thread https://stackoverflow.com/questions/20732332/how-to-store-a-binary-object-in-redis-using-node
You can do that using commandOptions.returnBuffers:
await redisClient.hGet(client.commandOptions({ returnBuffers: true}), id, 'v'); // { v: Buffer }
NOTE: this API will change a bit in v5 to make it clearer and more powerful (see https://github.com/redis/node-redis/blob/v5/docs/v4-to-v5.md#command-options and https://github.com/redis/node-redis/blob/v5/docs/command-options.md)
Nice, thanks! That's what I found and it does work, but there's also an additional step to transform the buffer into an array. This is what I ended up doing in the PR #2568 :+1:
// The inverse of tobytes, for parsing results out of hget.
const frombytes = (buffer) => {
const array = new Float32Array(buffer.length / 4);
for (let i = 0; i < array.length; i++) {
array[i] = buffer.readFloatLE(i * 4);
}
return array;
};
// An example of how to get a particular vector back out of Redis.
const vector = await client.hGet(
commandOptions({ returnBuffers: true }),
'noderedis:knn:a',
'v'
);
console.log(frombytes(vector));
I'm not 100% sure that the frombytes approach is the ideal solution / best practice here.