Faiss assertion 'blasStatus == CUBLAS_STATUS_SUCCESS' failed in virtual void faiss::gpu::StandardGpuResourcesImpl::initializeForDevice(int) at /project/faiss/faiss/gpu/StandardGpuResources.cpp:335

Open ShuxunoO opened this issue 11 months ago • 3 comments

Summary

I used model to extract features and saved them in .npy format, but I encountered an exception when using faiss-gpu to create an index. It seemed to be a conflict with the cuda driver version.

Platform

Ubuntu 18.04.6 LTS

Faiss version: faiss-gpu 1.7.2

Installed from: pypi

Running on:

[ ] CPU
[x] GPU +-----------------------------------------------------------------------------+ | NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 CUDA Version: 11.3 | |-------------------------------+----------------------+----------------------+

Interface:

[ ] C++
[x] Python

ldd _swigfaiss.cpython-310-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffee91fd000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f24fdacb000)
        libgfortran-040039e1.so.5.0.0 => /data/wupeixuan/anaconda3/envs/nft_search/lib/python3.10/site-packages/faiss/./../faiss_gpu.libs/libgfortran-040039e1.so.5.0.0 (0x00007f24fd635000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f24fd2ac000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f24fcf0e000)
        libgomp-a34b3233.so.1.0.0 => /data/wupeixuan/anaconda3/envs/nft_search/lib/python3.10/site-packages/faiss/./../faiss_gpu.libs/libgomp-a34b3233.so.1.0.0 (0x00007f24fcce4000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f24fcacc000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f24fc8ad000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f24fc4bc000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2507017000)
        libquadmath-96973f99.so.0.0.0 => /data/wupeixuan/anaconda3/envs/nft_search/lib/python3.10/site-packages/faiss/./../faiss_gpu.libs/libquadmath-96973f99.so.0.0.0 (0x00007f24fc27f000)
        libz-745e0a09.so.1.2.7 => /data/wupeixuan/anaconda3/envs/nft_search/lib/python3.10/site-packages/faiss/./../faiss_gpu.libs/libz-745e0a09.so.1.2.7 (0x00007f24fc068000)

Reproduction instructions

import torch
import faiss
import numpy as np

def create_gpu_IPindex_use_n_gpu(datas_embedding, gpus=[7]):
    index_flat = faiss.IndexFlatIP(datas_embedding.shape[1])  
    gpu_index_flat = faiss.index_cpu_to_gpus_list(index_flat, gpus=gpus)
    gpu_index_flat.add(datas_embedding)  
    return gpu_index_flat


if __name__ == "__main__":

    img_features_path = "/data/wupeixuan/NFT_Search/NFT_data/features/clip-l-14-ft/top100_all_img_features.npy"
    img_features = np.load(img_features_path)
    index_gpu = create_gpu_IPindex_use_n_gpu(img_features, gpus=[4, 5, 6, 7])