denseflow icon indicating copy to clipboard operation
denseflow copied to clipboard

GPU Memory error when running DenseFlow

Open cdrwolfe opened this issue 4 years ago • 4 comments

Running denseflow using the mmaction 'rawframes.py' script works for the RGB extraction but fails when tryign to extract flow images, throwing the following error:

terminate called after throwing an instance of 'cv::Exception' what(): OpenCV(4.2.0) /home/******/opencv-4.2.0/modules/core/src/cuda_host_mem.cpp:95: error: (-217:Gpu API call) out of memory in function 'allocate'

I only use 1 gpu (eGPU), but set it as such when calling script:

python build_rawframes.py /mnt/d/Data/ava/videos_15min/ /mnt/d/Data/ava/rawframes/ --task flow --level 1 --mixed-ext --flow-type tvl1 --num-gpu 1 --resume

Any ideas?

cdrwolfe avatar Dec 27 '20 16:12 cdrwolfe

~~See https://github.com/open-mmlab/denseflow/issues/18~~

innerlee avatar Dec 28 '20 00:12 innerlee

Please post the full stacktraces

innerlee avatar Dec 28 '20 00:12 innerlee

Try making this number smaller

https://github.com/open-mmlab/denseflow/blob/master/include/dense_flow.h#L95

innerlee avatar Dec 28 '20 00:12 innerlee

I've tried to set the batch size to 32, but unfortunately to no avail. This is a bit of a differant beast when it comes to setup, as it is a WSL ubuntu VM on Win 10 bootcamp (Macbook Pro) using an Aurous RTX 2080ti 11Gb GPU. So it might be a result of such a daisy chain of connections, but it is a bit odd given that at 11Gb I would have though it would be sufficient. Or possibly does the DenseFlow script not run through the GPU?

Unfortunately I can't seem to get a full stack trace from error, as it never really kills the process / script, and terminal / PyCharm just prints out as below.

flow 36 bhlFavrh7WU.mp4 tvl1 done push frames gray, video_flow_idx 127, batch_size 32 frames_gray_queue full, waiting... "/mnt/d/Data/ava/videos_15min/3IOE-Q3UWdA.mp4", frames ≈ 30600 "/mnt/d/Data/ava/videos_15min/z3kgrh0L_80.mp4", frames ≈ 30600 "/mnt/d/Data/ava/videos_15min/_-Z6wFjXtGQ.mp4", frames ≈ 30600 push frames gray, video_flow_idx 0, batch_size 32 push frames gray, video_flow_idx 0, batch_size 32 push frames gray, video_flow_idx 0, batch_size 32 push frames gray, video_flow_idx 31, batch_size 32 push frames gray, video_flow_idx 31, batch_size 32 push frames gray, video_flow_idx 31, batch_size 32 push frames gray, video_flow_idx 63, batch_size 32 push frames gray, video_flow_idx 63, batch_size 32 push frames gray, video_flow_idx 63, batch_size 32 push frames gray, video_flow_idx 95, batch_size 32 push frames gray, video_flow_idx 95, batch_size 32 push frames gray, video_flow_idx 95, batch_size 32 terminate called after throwing an instance of 'cv::Exception' what(): OpenCV(4.2.0) /home/*****/opencv-4.2.0/modules/core/src/cuda_host_mem.cpp:95: error: (-217:Gpu API call) out of memory in function 'allocate'

cdrwolfe avatar Dec 28 '20 13:12 cdrwolfe