can only use one cuda on local executor

[PeterYang-niubidehen]

somehow I could only use one cuda on a local executor, and it's always cuda 0 no matter which GPU I specify it to use. Does anybody know why this happens?

[patricklabatut]

I believe we would need information on what you are trying to do to possibly help.

[Luoxsh6]

@patricklabatut if I use the commond like this: ''' export CUDA_VISIBLE_DEVICES=1 python dinov2/run/eval/linear.py --xxxx ''' it will still use the GPU:0 for linear trainning. I have try to change the local_rank in def _set_from_local(self) from init.py, but it raise another error. Any solution?

[junlinguo]

@patricklabatut if I use the commond like this: ''' export CUDA_VISIBLE_DEVICES=1 python dinov2/run/eval/linear.py --xxxx ''' it will still use the GPU:0 for linear trainning. I have try to change the local_rank in def _set_from_local(self) from init.py, but it raise another error. Any solution?

python xxx.py, this will only set up the 'local run', which is single node and gpu run. You can check this in dinov2/distributed/init _set_from_local()

try using torchrun and set the --nproc_node as your number of gpu. Also, export CUDA_VISIBLE_DEVICES first