corrupted size vs. prev_size while consolidating

[skriser]

Summary

when i use this code

faiss.normalize_L2(embeddings)
quantizer = faiss.IndexFlatIP(self.dim)
index2 = faiss.IndexIVFFlat(quantizer, self.dim, self.nlist, faiss.METRIC_INNER_PRODUCT)
index2.add_with_ids(embeddings, ids)

I used jmeter for performance test and saved several times. An error happend

Platform

NAME="CentOS Linux" VERSION="8 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="8" PLATFORM_ID="platform:el8" Faiss version: faiss-cpu:1.7.2 Installed from: pip install faiss-cpu Faiss compilation options: No Running on:

• [ ✔️] CPU

Interface:

• [✔️ ] Python

Reproduction instructions

mycode about use faiss index type:

faiss.normalize_L2(embeddings)
quantizer = faiss.IndexFlatIP(self.dim)
index2 = faiss.IndexIVFFlat(quantizer, self.dim, self.nlist, faiss.METRIC_INNER_PRODUCT)
if not index2.is_trained:
    index2.train(embeddings)

mycode about save vector :

         if self.faiss_index:
                logger.debug("has faiss_index in ")
                if self.idx_type == "cosine":
                    faiss.normalize_L2(embeddings)
                if not self.faiss_index.is_trained:
                        self.faiss_index.train(embeddings)
                self.faiss_index.add_with_ids(embeddings, ids)
                if is_save:
                    faiss.write_index(self.faiss_index, self.index_file)
                return self.faiss_index

logs:

2022-10-19 10:28:49.060 | DEBUG    | src.deeprecall.vsearch.faiss_service:set_index:39 - has faiss_index in
2022-10-19 10:28:49.060 | INFO     | src.deeprecall.web.views.vector_save_view:post:80 - finall wright tmp request params
2022-10-19 10:28:49.060 | INFO     | src.deeprecall.web.views.vector_save_view:post:73 - execute_result: <faiss.swigfaiss.IndexIVFFlat; proxy of <Swig Object of type 'faiss::IndexIVFFlat *' at 0x7fe3dc330240> >
2022-10-19 10:28:49.061 | INFO     | src.deeprecall.web.views.vector_save_view:post:83 - finall clean tmp request params
INFO:werkzeug:192.168.11.201 - - [19/Oct/2022 10:28:49] "POST /v1/dssm/vector/save HTTP/1.1" 200 -
2022-10-19 10:28:49.061 | INFO     | src.deeprecall.web.views.vector_save_view:post:83 - finall clean tmp request params
INFO:werkzeug:192.168.11.201 - - [19/Oct/2022 10:28:49] "POST /v1/dssm/vector/save HTTP/1.1" 200 -
2022-10-19 10:28:49.061 | INFO     | src.deeprecall.web.views.vector_save_view:post:73 - execute_result: <faiss.swigfaiss.IndexIVFFlat; proxy of <Swig Object of type 'faiss::IndexIVFFlat *' at 0x7fe3dc330240> >
2022-10-19 10:28:49.062 | INFO     | src.deeprecall.web.views.vector_save_view:post:83 - finall clean tmp request params
INFO:werkzeug:192.168.11.201 - - [19/Oct/2022 10:28:49] "POST /v1/dssm/vector/save HTTP/1.1" 200 -
2022-10-19 10:28:49.064 | INFO     | src.deeprecall.web.views.vector_save_view:post:43 - {'dataset': {'10002': [0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.5, 0.4], '10003': [0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.5, 0.4], '10004': [0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.5, 0.4]}, 'vector_model': 'user'}
2022-10-19 10:28:49.065 | INFO     | src.deeprecall.web.views.vector_save_view:post:43 - {'dataset': {'10002': [0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.5, 0.5], '10003': [0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.5, 0.5], '10004': [0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.5, 0.5]}, 'vector_model': 'user'}
2022-10-19 10:28:49.065 | INFO     | src.deeprecall.web.views.vector_save_view:post:58 - judge if faiss_user_index in self config: True
2022-10-19 10:28:49.066 | INFO     | src.deeprecall.web.views.vector_save_view:post:43 - {'dataset': {'10002': [0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.5, 0.6], '10003': [0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.5, 0.6], '10004': [0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.3, 0.1, 0.2, 0.5, 0.6]}, 'vector_model': 'user'}
2022-10-19 10:28:49.066 | INFO     | src.deeprecall.web.views.vector_save_view:post:58 - judge if faiss_user_index in self config: True
2022-10-19 10:28:49.066 | INFO     | src.deeprecall.web.views.vector_save_view:post:61 - ids.shape: (3,), embeddings.shape: (3, 16)
2022-10-19 10:28:49.067 | INFO     | src.deeprecall.web.views.vector_save_view:post:58 - judge if faiss_user_index in self config: True
2022-10-19 10:28:49.067 | INFO     | src.deeprecall.web.views.vector_save_view:post:61 - ids.shape: (3,), embeddings.shape: (3, 16)
2022-10-19 10:28:49.067 | DEBUG    | src.deeprecall.vsearch.faiss_service:set_index:39 - has faiss_index in
2022-10-19 10:28:49.067 | INFO     | src.deeprecall.web.views.vector_save_view:post:61 - ids.shape: (3,), embeddings.shape: (3, 16)
2022-10-19 10:28:49.067 | DEBUG    | src.deeprecall.vsearch.faiss_service:set_index:39 - has faiss_index in
2022-10-19 10:28:49.067 | DEBUG    | src.deeprecall.vsearch.faiss_service:set_index:39 - has faiss_index in
double free or corruption (out)

[skriser]

[skriser]

anyone else help me

[mdouze]

This is a memory corruption error. Could you try making a small reproduction script based on random data to make sure it is indeed due to Faiss?