faiss
faiss copied to clipboard
How to create an IVF index on `uint8`/`int8` data?
I usually use the following script for generating FAISS-IVF
indices on floating point data:
import numpy as np
import faiss
A = np.fromfile('base.bin', dtype=np.int32)
shape = A[:2]
A = np.fromfile('base.bin', dtype=np.float32)
A = A[2:].reshape(shape)
print(shape)
dim = A.shape[1]
index = faiss.IndexIVFFlat(faiss.IndexFlatL2(dim), dim, 3200, faiss.METRIC_L2)
index.train(A)
index.add(A)
faiss.write_index(index, 'my_index.index')
But on using a dataset with uint8
/int8
vectors, the following script gives an error:
import numpy as np
import faiss
A = np.fromfile('base.bin', dtype=np.int32)
shape = A[:2]
A = np.fromfile('base.bin', dtype=np.uint8)
A = A[8:].reshape(shape)
print(shape)
dim = A.shape[1]
index = faiss.IndexIVFFlat(faiss.IndexFlatL2(dim), dim, 3200, faiss.METRIC_L2)
index.train(A)
index.add(A)
faiss.write_index(index, 'my_index.index')
File "ivf_runner.py", line 12, in <module>
index.train(A)
File "/usr/local/lib/python3.6/dist-packages/faiss/__init__.py", line 280, in replacement_train
self.train_c(n, swig_ptr(x))
File "/usr/local/lib/python3.6/dist-packages/faiss/swigfaiss.py", line 4610, in train
return _swigfaiss.IndexIVF_train(self, n, x)
TypeError: in method 'IndexIVF_train', argument 3 of type 'float const *'
How can I create an index with uint8
/int8
vector data? Thanks in advance.