Faiss-gpu with torch.distributed

[doctrymtk]

Summary

I want to use faiss-gpu with torch.distrubuted on the same machine. However, when I launch the program, it complains that there is no available gpu even if I the gpu memories on the machine are nearly empty. What can I do with it?

Running on:

• [ ] CPU
• [X] GPU

Interface:

• [ ] C++
• [X] Python

Reproduction instructions

[mdouze]

Maybe you could give a bit more detail ? + code ?

[GlennCGL]

Summary

I want to use faiss-gpu with torch.distrubuted on the same machine. However, when I launch the program, it complains that there is no available gpu even if I the gpu memories on the machine are nearly empty. What can I do with it?

Running on:

• [ ] CPU
• [x] GPU

Interface:

• [ ] C++
• [x] Python

Reproduction instructions

Did you solve it?

[eugene-yang]

I'm experiencing similar issues with allocating memory when running KMeans using FAISS. The problem for me seems to originate from trying to allocate memory on multiple GPUs from a child process. I successfully let FAISS to work with me by initiating everything from the main python process instead of from a child one.

This makes me suspect that there are some issues with multiprocessing with Python. I'm running with cudatoolkit=11.3.1.