darknet icon indicating copy to clipboard operation
darknet copied to clipboard

Published 20 hours ago • verified publisher icon prabindh

After 10 Minutes Arapaho program getting killed

Open shaikhibrahim951 opened this issue 6 years ago • 15 comments

Hi,

I trained my images with tiny yolo and output is coming good but after 10 to 12 minutes in terminal I am getting killed message. I checked my memory usage using top command then I find that available memory getting reduced by some amount and at some point system is getting hanged. I think some memory leak problem is there and program is not clearing that memory after allocation. Note : I am working on NVIDIA TX2 board having 8GB of RAM. Kindly provide some help regarding this issue. Thanks in advance

shaikhibrahim951 avatar Jul 16 '18 12:07 shaikhibrahim951

Please share the source code being used. Have you confirmed other components (like camera, opencv, or) do not leak memory as well ?

prabindh avatar Jul 17 '18 04:07 prabindh

My camera and opencv is not resposible for memory leak. I checked it. I will share my source code if you find the problem then it will be good.

test.txt

Attached file is test.cpp in txt form

Thanks

shaikhibrahim951 avatar Jul 17 '18 05:07 shaikhibrahim951

This may be platform specific behaviour. To be on the safe side, I have pushed a long pending patch for a known memory leak (https://github.com/prabindh/darknet/commit/f6f41a1f3a61b801c109ed62643bcfeb891447a9) - please take a look at latest master.

prabindh avatar Jul 21 '18 14:07 prabindh

hey, in my case the leak problem is because of the resize function in ArapahoV2::Detect. in every frame it is leaking around 60KB of memory.

SinghiDivyanshu avatar Jul 24 '18 06:07 SinghiDivyanshu

@SinghiDivyanshu Which platform (ARM or x64, Windows or Linux) are you seeing this on ?

prabindh avatar Jul 25 '18 00:07 prabindh

@prabindh That problem is resolved as i change the width and height in .cfg file so that resize function is not required.

but still their is a 18KB of definite leak per frame and i'm using linux.

SinghiDivyanshu avatar Jul 25 '18 05:07 SinghiDivyanshu

18 kB seems small for a 3-ch image frame. Will have to check further.

prabindh avatar Jul 25 '18 08:07 prabindh

@prabindh ok thanks for the help.

SinghiDivyanshu avatar Jul 25 '18 10:07 SinghiDivyanshu

Do we have a solution for this?

ArpitaSTugave avatar Aug 27 '18 10:08 ArpitaSTugave

Havent found time to deep debug, but to rule out CV itself, wrote a modified function with separate Mats for each operation. Could you please give it a try and let me know if it helps ?

  • Replace the function bool ArapahoV2::Detect( const cv::Mat & inputMat, float thresh, float hier_thresh, int & objectCount)

, with the one in the gist below. https://gist.github.com/prabindh/b7875557e736f75a576972de5618aecf

prabindh avatar Aug 27 '18 12:08 prabindh

@ArpitaSTugave In my case main leak (350KB memory) was due to cv::imshow() function for every frame.

SinghiDivyanshu avatar Aug 27 '18 20:08 SinghiDivyanshu

@SinghiDivyanshu - What version of OpenCV dll is being used in your cases ?

prabindh avatar Aug 28 '18 02:08 prabindh

openCV 3.3

SinghiDivyanshu avatar Aug 28 '18 09:08 SinghiDivyanshu

So I think there are 2 issues here:

  1. Leak in resize - Can we have the gist I posted above tested ?
  2. Leak in imshow() and cap.read() - @SinghiDivyanshu How did you resolve that leak ? Create a new image object every time ?

prabindh avatar Aug 28 '18 11:08 prabindh

leak due to resize: I change the width and height in cfg file.

and for imshow() leak: I'm not able to find any solution, so i remove that function.

SinghiDivyanshu avatar Aug 29 '18 06:08 SinghiDivyanshu