yolov2-yolov3_PyTorch icon indicating copy to clipboard operation
yolov2-yolov3_PyTorch copied to clipboard

Why is training so slow

Open lilslsls opened this issue 5 years ago • 8 comments

4900 images and batchsize:64 gpu:2080TI

lilslsls avatar May 23 '20 05:05 lilslsls

Because the workers in dataloader is 0. I denote this in my train code. If I add workers, it will report some errors when I use multiscale training trick. I have no idea how to solve this problems. 

---Original--- From: "lilslsls"<[email protected]> Date: Sat, May 23, 2020 13:54 PM To: "yjh0410/yolov2-yolov3_PyTorch"<[email protected]>; Cc: "Subscribed"<[email protected]>; Subject: [yjh0410/yolov2-yolov3_PyTorch] Why is training so slow (#34)

4900 images and batchsize:64 gpu:2080TI

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

yjh0410 avatar May 23 '20 05:05 yjh0410

And if you don't plan to try multi scale training, you'd better set num_workers as 8 or more to accelerate training.

---Original--- From: "lilslsls"<[email protected]> Date: Sat, May 23, 2020 13:54 PM To: "yjh0410/yolov2-yolov3_PyTorch"<[email protected]>; Cc: "Subscribed"<[email protected]>; Subject: [yjh0410/yolov2-yolov3_PyTorch] Why is training so slow (#34)

4900 images and batchsize:64 gpu:2080TI

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

yjh0410 avatar May 23 '20 06:05 yjh0410

I used yolov2 training, not multi-scale, and a batch spent nearly one minute in training. An epoch has many batches. So I feel strange. Is it because the code parses XML.

lilslsls avatar May 23 '20 06:05 lilslsls

No multi scale?Aha!You can set num_workers as 8(default=8). 

Do you set --cuda to use GPU? Its default value is False.

---Original--- From: "lilslsls"<[email protected]> Date: Sat, May 23, 2020 14:14 PM To: "yjh0410/yolov2-yolov3_PyTorch"<[email protected]>; Cc: "Comment"<[email protected]>;"Jianhua Yang"<[email protected]>; Subject: Re: [yjh0410/yolov2-yolov3_PyTorch] Why is training so slow (#34)

I used yolov2 training, not multi-scale, and a batch spent nearly one minute in training. An epoch has many batches. So I feel strange. Is it because the code parses XML.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

yjh0410 avatar May 23 '20 06:05 yjh0410

Sorry, I didn't read the code carefully. Now it's fast to use CUDA. thank you

lilslsls avatar May 23 '20 06:05 lilslsls

You're welcome!!!(๑>؂<๑)

---Original--- From: "lilslsls"<[email protected]> Date: Sat, May 23, 2020 14:27 PM To: "yjh0410/yolov2-yolov3_PyTorch"<[email protected]>; Cc: "Comment"<[email protected]>;"Jianhua Yang"<[email protected]>; Subject: Re: [yjh0410/yolov2-yolov3_PyTorch] Why is training so slow (#34)

Sorry, I didn't read the code carefully. Now it's fast to use CUDA. thank you

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

yjh0410 avatar May 23 '20 06:05 yjh0410

SET CUDA TRUE To use GPU!

envs

  • 2080ti x 1
  • ubuntu 16.04
  • python 3.6
  • pytorch 1.5.0
  • cpu cores :8
  • cpu processor num :16

without GPU

when i use cpu to train the model ,the training is sooooooo slow, such as the blow info... [Epoch 1/250][Iter 0/517][lr 0.000000][Loss: obj 210.70 || cls 4.88 || bbox 8.88 || total 224.46 || size 416 || time: 47.35]

use GPU

but when i set cuda True, the training speed up obivous! Epoch 1/250][Iter 10/517][lr 0.000000][Loss: obj 212.88 || cls 7.12 || bbox 9.07 || total 229.07 || size 416 || time: 2.20]

bruce1408 avatar Sep 04 '20 13:09 bruce1408

SET CUDA TRUE To use GPU!

envs

  • 2080ti x 1
  • ubuntu 16.04
  • python 3.6
  • pytorch 1.5.0
  • cpu cores :8
  • cpu processor num :16

without GPU

when i use cpu to train the model ,the training is sooooooo slow, such as the blow info... [Epoch 1/250][Iter 0/517][lr 0.000000][Loss: obj 210.70 || cls 4.88 || bbox 8.88 || total 224.46 || size 416 || time: 47.35]

use GPU

but when i set cuda True, the training speed up obivous! Epoch 1/250][Iter 10/517][lr 0.000000][Loss: obj 212.88 || cls 7.12 || bbox 9.07 || total 229.07 || size 416 || time: 2.20]

Thanks for your advice!HAHA!

The reason why I didn't set cuda as True is that I wanna indirectly remind everyone who plans to try this project to use GPU to train the model. If no GPU,it's a little hard to study object detection.

yjh0410 avatar Sep 04 '20 13:09 yjh0410