node-redis icon indicating copy to clipboard operation
node-redis copied to clipboard

Unclear how to retrieve vector

Open curran opened this issue 2 years ago • 2 comments

Description

I'm struggling to understand how to transform the vector output from Redis into a JavaScript array of numbers.

My application code is similar to the KNN search example https://github.com/redis/node-redis/blob/master/examples/search-knn.js

Here's the relevant snippet that stores vectors for similarity search:

function float32Buffer(arr) {
  return Buffer.from(new Float32Array(arr).buffer);
}

// Add some sample data...
// https://redis.io/commands/hset/
await Promise.all([
  client.hSet('noderedis:knn:a', { v: float32Buffer([0.1, 0.1]) }),
  client.hSet('noderedis:knn:b', { v: float32Buffer([0.1, 0.2]) }),
  client.hSet('noderedis:knn:c', { v: float32Buffer([0.1, 0.3]) }),
  client.hSet('noderedis:knn:d', { v: float32Buffer([0.1, 0.4]) }),
]);

Now here is my issue: I want to get the vector by its key and be able to see the original vector as a JavaScript Array.

I tried the following:

const redisResult = await redisClient.hGet(id, 'v');
console.log(redisResult)

The output is a cryptic string like "�����'<�n;�h�?�Vٍ���/<7c��" that I believe may be [RESP](https://redis.io/docs/reference/protocol-spec/) format. I want to transform that result into its corresponding array, e.g. [0.1, 0.3]`, however I'm not sure how to do so. Any guidance would be greatly appreciated. Thanks!

I wonder if I should try using https://www.npmjs.com/package/redis-parser ...

Related StackOverflow thread https://stackoverflow.com/questions/20732332/how-to-store-a-binary-object-in-redis-using-node

curran avatar Jul 12 '23 14:07 curran

You can do that using commandOptions.returnBuffers:

await redisClient.hGet(client.commandOptions({ returnBuffers: true}), id, 'v'); // { v: Buffer }

NOTE: this API will change a bit in v5 to make it clearer and more powerful (see https://github.com/redis/node-redis/blob/v5/docs/v4-to-v5.md#command-options and https://github.com/redis/node-redis/blob/v5/docs/command-options.md)

leibale avatar Jul 18 '23 17:07 leibale

Nice, thanks! That's what I found and it does work, but there's also an additional step to transform the buffer into an array. This is what I ended up doing in the PR #2568 :+1:

// The inverse of tobytes, for parsing results out of hget.
const frombytes = (buffer) => {
  const array = new Float32Array(buffer.length / 4);
  for (let i = 0; i < array.length; i++) {
    array[i] = buffer.readFloatLE(i * 4);
  }
  return array;
};

// An example of how to get a particular vector back out of Redis.
const vector = await client.hGet(
  commandOptions({ returnBuffers: true }),
  'noderedis:knn:a',
  'v'
);
console.log(frombytes(vector));

I'm not 100% sure that the frombytes approach is the ideal solution / best practice here.

curran avatar Jul 18 '23 18:07 curran