Memory Leak in CSRT Tracker?

[Tomas1337]

System information (version)

• OpenCV => opencv-contrib-python 4.5.1.48
• Operating System / Platform => Windows 10 64 Bit
• Compiler => VS Code 1.53.2

Memory leak observed in CSRT Tracker. Each call to .update(frame) function causes peak memory to go up while no garbage collection is observed.

Steps to reproduce

from imutils.video import VideoStream
from imutils.video import FPS
import argparse
import imutils
import time
import cv2
import sys, os
import gc
PACKAGE_PARENT = '..'
SCRIPT_DIR = os.path.dirname(os.path.realpath(os.path.join(os.getcwd(), os.path.expanduser(__file__))))
sys.path.append(os.path.normpath(os.path.join(SCRIPT_DIR, PACKAGE_PARENT)))

def get_video_path():
    test_path = "test/"
    for file in os.listdir(test_path):
        if file.endswith(".mp4"):
            video_path = os.path.join(test_path, file)
            yield video_path

def test_trackers(tracker):
    video = cv2.VideoCapture(list(get_video_path())[0])
    end = 2000
    start=0
    target_coordinates = (100,100)   
    f_tracker = None
    (x,y,w,h) = (285, 73, 48, 62) #Test Track coordinates for short_video

    lost_tracking_count = 0
    
    while video.isOpened():
        ret, frame = video.read()
        resize_frame_shape = (640,360)
        frame = cv2.resize(frame, resize_frame_shape)

        cv2.imshow('frame',frame)
        if cv2.waitKey(1) & 0xFF == ord('q'): # wait for 1 millisecond
            break

        if ret:
            # if start% 200 == 0:
            #     #Reset Trigger
            #     probe.add_marker(f'Garbage collecting')
            #     f_tracker =  None
            #     print(f'GC before : {gc.get_count()}')
            #     gc.collect()
            #     print(f'GC after: {gc.get_count()}')

            if not f_tracker is None:
                ok, position = f_tracker.update(frame)
                if ok:
                    pass
                    #print(f'Position: {position}')-
                else:
                    lost_tracking_count += 1
                    #print('Tracking failed')        
            else: 
                f_tracker = tracker()
                f_tracker.init(frame, (x,y,w,h))

            start +=1 

        if start == end:
            print('Ended')
            break

    print(f'Lost Tracking count for Tracker is: {lost_tracking_count}')
            

OPENCV_OBJECT_TRACKERS = {
        "csrt": cv2.TrackerCSRT_create,
    	"kcf": cv2.TrackerKCF_create,
	}


if __name__ == '__main__':
    from blackfire import probe
    probe.initialize(
        # Client credentials. If None, they are read from environment variables, or config_file.
        client_id='<client_id>',
        client_token='<client_token>',
        log_level=1,
        log_file=None,
        config_file='~/.blackfire.ini',
        query=None,
        agent_socket=None,
        agent_timeout=None,
        endpoint=None,
    )

    probe.enable()
    for (key, value) in OPENCV_OBJECT_TRACKERS.items():
        probe.add_marker(f'Tracker: {key}')
        print(f'Tracker: {key}')
        test_trackers(value) #No Tracking
    probe.end()

The blue graph in the beginning is applicaton memory with CSRT Tracking. KCF Tracker is then initialized afterwards which gives a flat memory peak.

To reproduce add a test_video mp4 file in a folder named 'test'

[Tomas1337]

Adding Information:

Stopped the tracker midway in the Video to test if the openCV video reading is the problem. Screenshot proves further that its the CSRT tracker. Last marker stops the tracker. Also, using memory_profiler, cant seem to reproduce this graph.

[Tomas1337]

Uploading the test video: https://drive.google.com/file/d/1Otr21tZHwCBOmL6n6EO8YwE4-yeTvJ-L/view?usp=sharing

First frame you can pass the coordinates (x,y,w,h) = (285, 73, 48, 62) to trach the face

[Tomas1337]

For some reason, tracemalloc and memory_profiler on python don't seem to catch the memory leak and I can only see the memory leak using the blackfire probe. Maybe because the increments are really low?

Added a 30min~ test to see what happens in longer videos: Still increasing consumption and RAM leak is about 10Mb/hour

[alalek]

Without labeling of y axis these graphics look really useless. Are these graphs reproducible (do they have the same structure)?

Your code snippet uses these components:

1. blackfire (where is this tool collect/store data?)
2. VideoCapture (may queue several "pending" frames, 1 HD frame is about 3Mb)
3. imshow
4. tracking

At first we should determine the "leaked" component properly. Why do you think that this is tracking part?

Disable components one-by-one. For example, VideoCapture can be disabled through pre-fetching frames into the memory (array of frames).

tracemalloc and memory_profiler on python

OpenCV is C++ library. Python part is just wrapper over C++ native code. So C++ tools should be used for proper analysis (on Linux there are great valgind/massif tools).

It would be nice to have minimal C++ reproducer.

[Tomas1337]

HI, Alalek.

Here's better reproducible code:

from imutils.video import VideoStream
from imutils.video import FPS
import argparse
import imutils
import time
import cv2
import sys, os
import psutil
import gc

#from memory_profiler import profile

def get_video_path():
    return list(Path("test/").glob("*.mp4"))

def test_trackers(tracker):
    #Get single frame from Video Object
    video = cv2.VideoCapture(list(get_video_path())[0])
    while video.isOpened():
        ret, frame = video.read()
        if ret:
            break
        else:
            pass
    f_tracker = tracker()
    (x,y,w,h) = (285, 73, 48, 62) 
    f_tracker.init(frame, (x,y,w,h))
    start = 0
    end = 1000

    while start != end:
        ok, position = f_tracker.update(frame)
        if ok:
            pass
        else: 
            lost_tracking_count +=1
        
        start +=1 

    print ('ended')

OPENCV_OBJECT_TRACKERS = {
        "csrt": cv2.TrackerCSRT_create,
        #"mil": cv2.TrackerMIL_create,
    	"kcf": cv2.TrackerKCF_create,
	}


if __name__ == '__main__':
    from blackfire import probe
    probe.initialize(
        # Client credentials. If None, they are read from environment variables, or config_file.
        client_id='eda8c24f-7937-4890-acc0-fe46483fa320',
        client_token='db841bf47a62da43a4c0cddc0b4dd36b971e81635d45078251661daa176f8924' ,
        log_level=1,
        log_file=None,
        query=None,
        agent_socket=None,
        agent_timeout=None,
        endpoint=None,
    )

    probe.enable()
    for (key, value) in OPENCV_OBJECT_TRACKERS.items():
        probe.add_marker(f'Tracker: {key}')
        print(f'Tracker: {key}')
        test_trackers(value) #No Tracking

    probe.end()

I've only activated cv2 Video function for getting one frame. Then reusing that frame into the update function of the tracker object. The reason that I think it is the CSRT Tracker in particular is because the issue is resolved with a tracker such as KCF. The graph shows that the memory (y-axis light blue graph) increase stops when we switch over to KCF Tracker. It's only during CSRT where the memory increases.

I too would like to have it in C++ but its taking me time to get it running since installing contrib is hard

[rokopi-byte]

I have the same issue, but on ARM platform. I'm using a raspberry pi 4 with opencv cross-compiled (4.5.2). It's a c++ application for people counting and using CRST to track people causes a memory leak. I attach a valgrind output here, let me know if I can further help. valgrind_out.txt

[kevinlinxc]

I also have this problem. Was this ever resolved in any later versions?