convolutional-pose-machines-release icon indicating copy to clipboard operation
convolutional-pose-machines-release copied to clipboard

error == cudaSuccess (2 vs 0) out memory

Open gxdai opened this issue 9 years ago • 11 comments

Hi Shihen,

I tried to retrain your model, my GPU version is K80, 24GB, but I got the following error: error == cudaSuccess (2 vs 0) out of memory

gxdai avatar Jun 16 '16 12:06 gxdai

I got this error as well! Did you solve it and how? Thanks!

BrightFeather avatar Jul 09 '16 04:07 BrightFeather

Hi, could you check the batch size? I was using 4 or 8 on 1 GPU.

shihenw avatar Jul 09 '16 17:07 shihenw

Are you using batchsize = 1? in file cpm_data_layer.cpp:106, there are lines like: if (batch_size == 1 && crop_size == 0) { Datum& datum = *(reader_.full().peek()); if (datum.encoded()) { if (force_color) { DecodeDatum(&datum, true); } else { DecodeDatumNative(&datum); } } batch->data_.Reshape(1, datum.channels(), datum.height(), datum.width()); this->transformed_data_.Reshape(1, datum.channels(), datum.height(), datum.width()); } The code will resize the data_ to the origin image size.

MudeLin avatar Aug 04 '16 07:08 MudeLin

No I using batch size at least 4 on each GPU. You are right, that code has problems with batch size 1. Will update.

shihenw avatar Aug 04 '16 19:08 shihenw

I got the same error when i forgot to set the USE_CUDNN := 1 in Makefile.config

aronfothi avatar Sep 09 '16 08:09 aronfothi

aronfothi's setting in the building of Caffe is crucial. I tried several times w/o USE_CUDNN on various computers (even one w/ a GPU card of 12 GB)--you must set USE_CUDNN to 1!

Btw, use CUDNN v. 3 and DO NOT use OpenCV 3 per Shih-En's recommendation.

schelian avatar Sep 29 '16 22:09 schelian

Hello everyone, In gpu mode, one nvidia gtx 1080 which has 8 g graphics memory can use to train the network and real-time forecasting?

If anyone knows, please tell me, much thanks!

mindcont avatar Dec 12 '16 00:12 mindcont

Yes, you should be able to train it but testing will not be realtime. Issue #14 talks about realtime testing with 4 GPUs (!).

schelian avatar Dec 12 '16 17:12 schelian

Hi, @schelian Thanks for you reply, it`s a great help to me. Sir, may i ask a another question? I am confused with ‘multiplier ’in applyModel.m line 26

multiplier = 2.^(log2(starting_scale):(1/octave):log2(ending_scale));

% data container for each scale and stage score = cell(nstage, length(multiplier)); pad = cell(1, length(multiplier)); ori_size = cell(1, length(multiplier));

net = caffe.Net(model.deployFile, model.caffemodel, 'test'); % change outputs to enable visualizing stagewise results % note this is why we keep out own copy of m-files of caffe wrapper

colors = hsv(length(multiplier)); for m = 1:length(multiplier) scale = multiplier(m);

and 'scale_search = 0.7:0.1:1.3; % fit training' in run_benchmark.m line 4

one is by hand drag a bounding box ,one is by pre-setting , i notice that this parameter affects the size of the detection scale, how should I adjust this parameter to fit the size of my photo(e.g 640*480 )?

mindcont avatar Dec 14 '16 13:12 mindcont

I don't think you have to adjust the parameter. multiplier is derived from starting_scale (line 26); starting_scale is derived from middle_range (line 21); middle_range is derived from the size of the image (line 20). If you happen to know your testing images have people within a certain scale then you can change the parameter. This will increase speed but could introduce false negatives / misses.

schelian avatar Dec 14 '16 17:12 schelian

@schelian Sir,i get your point, thank you again for your timely reply.

mindcont avatar Dec 15 '16 00:12 mindcont