The pretrained feature extractor doesn't use multi GPU

[JSP-ywu]

Hi, I have an issue about using GPU during tutorial. I try to benchmark the NAVERLABS dataset. I set the parameter of gpu of dirtorch.extract_kapture to 0 1 2 3, but the extractor still use only 1 gpu(cuda:0 indeed). I didn't change anything except the path of data. I tried to modify the code about multi gpu(in dirtorch), but it is not resolved. How can I use multi GPU during tutorial? It is too slow to extract feature from the dataset.

[JSP-ywu]

3) Extract global features (we will use AP-GeM here)
  cd ${WORKING_DIR}/deep-image-retrieval
  ${PYTHONBIN} -m dirtorch.extract_kapture --kapture-root ${WORKING_DIR}/${DATASET}/map_plus_test/ --checkpoint dirtorch/data/Resnet101-AP-GeM-LM18.pt --gpu 0 1 2 3
  mkdir -p ${WORKING_DIR}/${DATASET}/global_features/Resnet101-AP-GeM-LM18/global_features
  mv ${WORKING_DIR}/${DATASET}/map_plus_test/reconstruction/global_features/Resnet101-AP-GeM-LM18/* ${WORKING_DIR}/${DATASET}/global_features/Resnet101-AP-GeM-LM18/global_features/
  rm -rf ${WORKING_DIR}/${DATASET}/map_plus_test/reconstruction/global_features/Resnet101-AP-GeM-LM18

 4) Extract local features (we will use R2D2 here)
  cd ${WORKING_DIR}/r2d2
  ${PYTHONBIN} extract_kapture.py --model models/r2d2_WASF_N8_big.pt --kapture-root ${WORKING_DIR}/${DATASET}/map_plus_test/ --min-scale 0.3 --min-size 128 --max-size 9999 --top-k ${KPTS} --gpu 0 1 2 3
  mkdir -p ${WORKING_DIR}/${DATASET}/local_features/r2d2_WASF_N8_big/descriptors
  mv ${WORKING_DIR}/${DATASET}/map_plus_test/reconstruction/descriptors/r2d2_WASF_N8_big/* ${WORKING_DIR}/${DATASET}/local_features/r2d2_WASF_N8_big/descriptors/
  mkdir -p ${WORKING_DIR}/${DATASET}/local_features/r2d2_WASF_N8_big/keypoints
  mv ${WORKING_DIR}/${DATASET}/map_plus_test/reconstruction/keypoints/r2d2_WASF_N8_big/* ${WORKING_DIR}/${DATASET}/local_features/r2d2_WASF_N8_big/keypoints/

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA RTX A6000    Off  | 00000000:1D:00.0 Off |                  Off |
| 54%   79C    P2   260W / 300W |  41931MiB / 48685MiB |     83%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA RTX A6000    Off  | 00000000:20:00.0 Off |                  Off |
| 30%   27C    P8    11W / 300W |      2MiB / 48685MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA RTX A6000    Off  | 00000000:21:00.0 Off |                  Off |
| 30%   27C    P8    11W / 300W |      2MiB / 48685MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA RTX A6000    Off  | 00000000:24:00.0 Off |                  Off |
| 30%   27C    P8     7W / 300W |      2MiB / 48685MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    489977      C   .../envs/kapture/bin/python3    41929MiB |
+-----------------------------------------------------------------------------+

[yocabon]

Hi, the feature extraction scripts (deep-image-retrieval / r2d2) do not support multi-gpu (neither does the matching scripts). If your dataset is too big, you can always split the kapture data in 4 (you would have to create 4 kapture folders each with a fourth of records_camera.txt, and using symlinks for the rest, to avoid copying the images) and launch it on each individual gpu (then move the extracted features back to the main folder).

[JSP-ywu]

Thanks. I'll check it!