faiss icon indicating copy to clipboard operation
faiss copied to clipboard

How to create an IVF index on `uint8`/`int8` data?

Open ShikharJ opened this issue 2 years ago • 0 comments

I usually use the following script for generating FAISS-IVF indices on floating point data:

import numpy as np
import faiss

A = np.fromfile('base.bin', dtype=np.int32)
shape = A[:2]
A = np.fromfile('base.bin', dtype=np.float32)
A = A[2:].reshape(shape)
print(shape)
dim = A.shape[1]
index = faiss.IndexIVFFlat(faiss.IndexFlatL2(dim), dim, 3200, faiss.METRIC_L2)
index.train(A)
index.add(A)
faiss.write_index(index, 'my_index.index')

But on using a dataset with uint8/int8 vectors, the following script gives an error:

import numpy as np
import faiss

A = np.fromfile('base.bin', dtype=np.int32)
shape = A[:2]
A = np.fromfile('base.bin', dtype=np.uint8)
A = A[8:].reshape(shape)
print(shape)
dim = A.shape[1]
index = faiss.IndexIVFFlat(faiss.IndexFlatL2(dim), dim, 3200, faiss.METRIC_L2)
index.train(A)
index.add(A)
faiss.write_index(index, 'my_index.index')
File "ivf_runner.py", line 12, in <module>
    index.train(A)
  File "/usr/local/lib/python3.6/dist-packages/faiss/__init__.py", line 280, in replacement_train
    self.train_c(n, swig_ptr(x))
  File "/usr/local/lib/python3.6/dist-packages/faiss/swigfaiss.py", line 4610, in train
    return _swigfaiss.IndexIVF_train(self, n, x)
TypeError: in method 'IndexIVF_train', argument 3 of type 'float const *'

How can I create an index with uint8/int8 vector data? Thanks in advance.

ShikharJ avatar May 24 '22 09:05 ShikharJ