DeepStack icon indicating copy to clipboard operation
DeepStack copied to clipboard

Published 20 hours ago • verified publisher icon johnolafenwa

Memory Issues

Open 7thstorm opened this issue 4 years ago • 6 comments

System Specs:

  • Ubuntu 18.04
  • Proc: xeon 2.7 Ghz
  • Ram: 48 GB
  • GPU: Quadro P3200 6 GB
  • Cuda 11

Deepstack process does not release memory after processing.

Memory usage upon starting Deepstack in docker (927mb) image

memory @ 3.2 GB after registering 27 faces image

This high memory blockage prevents me from performing other functions including registering faces image

Either I'm missing something or memory management need some serious improvement. The only way around this is to keep on stopping the docker container after each action Thoughts?

7thstorm avatar Feb 20 '21 03:02 7thstorm

I have the same problem with my Jetson Nano / GTX 1650 I haven't reached the limit yet but I am guessing I will only be able to train about 30ish faces per person before I run out of memory. The GTX1650 only has 4GB of RAM and I don't have access to an Nvidia card with more.

I have maxed out at 3869MB so far on the GTX. So between teaching people I have to restart the container. Once it's fully loaded it floats around 1181 - 1185MB

wills106 avatar May 08 '21 16:05 wills106

is there any update on this?

7thstorm avatar May 19 '21 17:05 7thstorm

Just wondering if there has been any progress on the memory issue / memory leak where it doesn't release memory after training in faces?

wills106 avatar Jul 11 '21 16:07 wills106

NOPE afaik image deepstack cpu on xeonE6-1650v2 3.5GHz 48 images, 79% of 5GB memory

manalishi70 avatar Jan 04 '22 19:01 manalishi70

I think I'm having the same on the Windows/GPU version. I can't tell how much memory is being used per process, either due to a limitation of nvidia-smi on Windows, my card (P400) or some combination. However, there is a definite correlation between GPU memory, and DeepStack entering an unrecoverable (without a restart) state where all requests result in a 100/Timeout error.

Without better logs (https://github.com/johnolafenwa/DeepStack/issues/142) I can't be sure where exactly what is going on, but there's a strong correlation.

I've been tracking troubleshooting and the steps I've taken here: https://ipcamtalk.com/threads/deepstack-gpu-memory-issue-error-100-timeout-after-several-hours.60827/

BeanBagKing avatar Jan 07 '22 17:01 BeanBagKing

I trained 4 face images and it's using more than half of the memory on jetson nano. If I train two more images it runs out of memory.

bogdanr avatar Feb 27 '22 16:02 bogdanr