wei-tim
YOWO not using GPUs?
Hello
First of all, thanks a lot for making this available.
I am using YOWO to train with a custom dataset. I prepared my data in the same format than AVA and used the ava.cfg configuration to run.
It is running OK but very slowly (almost 3 hours for each epoch, with total number of trainable parameters equal to 121381770). The machine where I am running has 4 GPUs but I have several reasons to think that YOWO is not using them:
- It makes no difference if I set the variable CUDA_VISIBLE_DEVICES to '0,1,2,3' or '0' or '', the speed is the same
- I am monitoring GPU usage with nvidia-smi and most of the time I see that GPU usage is 0
- If I check CPU usage, python is using 100% all the time
I added code to main.py to see if the GPUs are accesible and I can confirm that they are. The following
print(f" cuda available {torch.cuda.is_available()}") print(f" current device {torch.cuda.current_device()}") print(f" device 0 {torch.cuda.device(0)}") print(f" device count {torch.cuda.device_count()}")
Produces
cuda available True current device 0 device 0 <torch.cuda.device object at 0x7fdcf4a2ae80> device count 4
My question is, do I need to change something else to make YOWO use the GPUs?
