pytorch_Realtime_Multi-Person_Pose_Estimation icon indicating copy to clipboard operation
pytorch_Realtime_Multi-Person_Pose_Estimation copied to clipboard

Published 20 hours ago • verified publisher icon tensorboy

Memory error

Open tomercohen11 opened this issue 6 years ago • 8 comments

Hi, during inference, i get the following error:

" RuntimeError('cuda runtime error (2) : out of memory at /pytorch/torch/lib/THC/THCTensorCopy.cu:204',), )"

Which usually happens when using very large sized images. Even when I resize the images to the paper's (368, 654) this error still happens.

Any ideas on why this occurs?

Thanks!

tomercohen11 avatar Sep 04 '18 11:09 tomercohen11

with torch.no_grad() : output= model(image) may help.

tensorboy avatar Sep 04 '18 15:09 tensorboy

Or you should buy a GPU with large memory. :)

tensorboy avatar Sep 04 '18 15:09 tensorboy

I've tried volatile=true (using pytorch<0.4) and it's still taking about 4 second per image which is a far cry from real time performance... I am using a k80 so memory shouldn't be an issue.

Any other ideas?

Thanks.

tomercohen11 avatar Sep 04 '18 16:09 tomercohen11

It's strange, it is super quick on 1080ti...

tensorboy avatar Sep 04 '18 16:09 tensorboy

I noticed the bottleneck happens in the line: "heatmaps = output2.cpu().data.numpy().transpose(0, 2, 3, 1)" (in coco_eval.py)

Specifically output2.cpu() takes 2 seconds.

Does this help you help me? :)

tomercohen11 avatar Sep 05 '18 08:09 tomercohen11

It is weird that operation takes so much time......

tensorboy avatar Sep 05 '18 17:09 tensorboy

A question about memory...

I have a big computer with 2x 1080 GTX Ti. I can not run the training with more than 32 images in a batch... As you mentioned your are using a 1080, what is the batch size you used ?

Thank you.

Vaufreyd avatar Sep 14 '18 18:09 Vaufreyd

@Vaufreyd Do you have solved it ? I meet the same problem. Could you help me? @tensorboy

yllgl avatar Jul 26 '19 08:07 yllgl