Memory leak and endless gigabytes of the character '\' written to logs.

Describe the bug: I've started trying to build a project like I always have before but in 2.5.1 I'm hitting a memory leak similar to this older issues: https://github.com/OpenShot/openshot-qt/issues/2237 and https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=955975 where just adding a few clips to the project causes OpenShot to steadily consume all 128GB of my RAM and then crash. Using strace I've also identified that after some mutex events it starts to just write constant '\' character to ~/.openshot_qt/openshot-qt.log* files to the tune of 9GB (Two ~5GB log files actually).

Steps to reproduce the behavior:

1. Create new project.
2. Drag a group of clips from a folder into the project.
3. Repeat as needed with more clips until the app decides to start consuming massive amounts of RAM.
4. In my case the app stops responding and consuming memory after the last clip thumbnail shows in project.
5. App eventually becomes unresponsive but continues writing \ to logs until it

Expected behavior: Clips should add smoothly, the UI should stay responsive, and RAM usage shouldn't skyrocket.

System Details: HP Z640 2x10 Xeon, 128GB RAM, Nvidia + AMD GPU. Fedora 34 - X11 (not using Wayland)

Log Files: app:INFO ------------------------------------------------ app:INFO Sat Apr 10 09:49:19 2021
app:INFO Starting new session
app:INFO ------------------------------------------------ app:INFO OpenShot (version 2.5.1)
app:INFO ------------------------------------------------ app:INFO openshot-qt version: 2.5.1 app:INFO libopenshot version: 0.2.5 app:INFO platform: Linux-5.11.12-300.fc34.x86_64-x86_64-with-glibc2.33 app:INFO processor: x86_64 app:INFO machine: x86_64 app:INFO python version: 3.9.4 app:INFO qt5 version: 5.15.2 app:INFO pyqt5 version: 5.15.0 metrics:ERROR Error determining OS version in metrics.py

(LATER filled with gigs of \ until app crashes)

Exception / Stacktrace: If you are experiencing a crash or strange output on the command line, please attach a small snippet of the exception here, but please keep it as small as possible (for readability).

Screenshots: (Optional) Screenshot of system RAM and CPU usage graph from KDE:

same to me, i also opened an issue about this because this is very annoying.... https://github.com/OpenShot/openshot-qt/issues/4103

Sorry if this was included above, but did you try this with the daily build?

Also, does this happen with all files/file types?

Sorry if this was included above, but did you try this with the daily build?

Also, does this happen with all files/file types?

mine is here https://github.com/OpenShot/openshot-qt/issues/4103#issuecomment-854088670

Here as well. Core i7 6820HK, 32GB RAM, GTX 980 GPU. Ubuntu 20.04, Nvidia Driver 440.100

         app:INFO ------------------------------------------------
         app:INFO             Fri Jun  4 16:56:30 2021            
         app:INFO               Starting new session              
         app:INFO ------------------------------------------------
         app:INFO          OpenShot (version 2.5.1-dev2)          
         app:INFO ------------------------------------------------
         app:INFO openshot-qt version: 2.5.1-dev2
         app:INFO libopenshot version: 0.2.5-dev2
         app:INFO platform: Linux-5.4.0-39-generic-x86_64-with-glibc2.29
         app:INFO processor: x86_64
         app:INFO machine: x86_64
         app:INFO python version: 3.8.2
         app:INFO qt5 version: 5.12.8
         app:INFO pyqt5 version: 5.14.1
     metrics:ERROR Error determining OS version in metrics.py
    language:INFO Qt Detected Languages: ['pt-BR', 'pt', 'en']

Also might be useful to notice: the memory grows happen when I play the timeline. If I just cut without playing anyhting the memory is stable. I have all decoders / encoders set to vdpau to use GPU.

If I keep editing video it reaches the point that kernel kills the process due out of memory.

Btw just tested one thing: I setup originally the cache to memory, with a 3GB capacity. I tried switch to disk cache, the memory usage stopped to grow. So it might be an issue with the cache memory limit.

Btw just tested one thing: I setup originally the cache to memory, with a 3GB capacity. I tried switch to disk cache, the memory usage stopped to grow. So it might be an issue with the cache memory limit.

it is simple random, i also changed the settings and have a chnace it is works or just try export-cancel export until have no memory leak, but even if it is works then next time u face with same issue...

I just imagined that the Python GC couldn't keep up with the rate, but its indeed a memory leak. About an hour after rendering a video, it still using 24.6GB of ram

So it doesn't cleanup afterwards.

I will try spending some time checking the code to see if I can find where is leaking.

Well, bad news. It looks like its not a leak from the code itself, but because how python memory allocation works.

I added a piece of code to track memory usage, and while it was using about 20GB of my RAM, the allocated stuff was barely over 70MB.

I was looking at this https://chase-seibert.github.io/blog/2013/08/03/diagnosing-memory-leaks-python.html and looks like python doesnt dealloc any OS memory it allocs until the process closes :| because the garbage collector can reuse the memory, but in the case of OpenShot that's a really bad scenario.

   thumbnail:INFO Shutting down thumbnail server: ('127.0.0.1', 48725)
         app:INFO             OpenShot's session ended            
         app:INFO             Tue Jun  8 13:57:13 2021            
         app:INFO ================================================
      logger:INFO ENDING TRACE
      logger:INFO [ Top 50 ]
      logger:INFO /usr/lib/python3.8/json/decoder.py:353: size=17.8 MiB, count=187978, average=100 B
      logger:INFO <frozen importlib._bootstrap_external>:580: size=9048 KiB, count=32763, average=283 B
      logger:INFO /usr/lib/python3.8/copy.py:230: size=5268 KiB, count=27645, average=195 B
      logger:INFO <frozen importlib._bootstrap>:219: size=3663 KiB, count=24916, average=151 B
      logger:INFO /usr/lib/python3.8/copy.py:227: size=1648 KiB, count=26361, average=64 B
      logger:INFO /usr/lib/python3.8/xml/etree/ElementTree.py:595: size=876 KiB, count=15091, average=59 B
      logger:INFO /usr/lib/python3.8/copy.py:201: size=417 KiB, count=7618, average=56 B
      logger:INFO /usr/lib/python3.8/linecache.py:137: size=277 KiB, count=2467, average=115 B
      logger:INFO /usr/lib/python3.8/copy.py:205: size=230 KiB, count=7309, average=32 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/classes/app.py:44: size=170 KiB, count=2304, average=75 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/classes/language.py:73: size=132 KiB, count=1720, average=79 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/windows/models/transition_model.py:167: size=128 KiB, count=1, average=128 KiB
      logger:INFO /usr/lib/python3.8/posixpath.py:88: size=125 KiB, count=1001, average=128 B
      logger:INFO /usr/lib/python3.8/dist-packages/PyQt5/uic/Loader/qobjectcreator.py:137: size=124 KiB, count=1405, average=90 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/classes/app.py:187: size=90.7 KiB, count=1190, average=78 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/windows/main_window.py:2350: size=75.1 KiB, count=948, average=81 B
      logger:INFO /usr/lib/python3.8/html/entities.py:264: size=72.6 KiB, count=3, average=24.2 KiB
      logger:INFO /usr/lib/python3.8/dist-packages/gi/module.py:215: size=50.4 KiB, count=275, average=188 B
      logger:INFO /usr/lib/python3.8/threading.py:801: size=44.9 KiB, count=82, average=561 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/classes/app.py:221: size=43.0 KiB, count=573, average=77 B
      logger:INFO /usr/lib/python3.8/mimetypes.py:93: size=42.1 KiB, count=439, average=98 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/windows/main_window.py:2534: size=39.2 KiB, count=472, average=85 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot.py:2677: size=38.5 KiB, count=11, average=3580 B
      logger:INFO /usr/lib/python3.8/dist-packages/gi/types.py:55: size=38.0 KiB, count=273, average=142 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/windows/models/transition_model.py:194: size=36.0 KiB, count=2, average=18.0 KiB
      logger:INFO <frozen importlib._bootstrap>:683: size=36.0 KiB, count=1, average=36.0 KiB
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/windows/views/timeline_webview.py:2969: size=35.8 KiB, count=420, average=87 B
      logger:INFO /usr/lib/python3.8/threading.py:371: size=35.1 KiB, count=68, average=528 B
      logger:INFO <frozen importlib._bootstrap_external>:1491: size=34.6 KiB, count=50, average=709 B
      logger:INFO <frozen importlib._bootstrap_external>:64: size=34.0 KiB, count=297, average=117 B
      logger:INFO <frozen importlib._bootstrap_external>:1483: size=30.9 KiB, count=497, average=64 B
      logger:INFO /usr/lib/python3.8/mimetypes.py:220: size=29.0 KiB, count=420, average=71 B
      logger:INFO /usr/lib/python3.8/dist-packages/gi/types.py:54: size=27.6 KiB, count=504, average=56 B
      logger:INFO /usr/lib/python3.8/mimetypes.py:229: size=25.4 KiB, count=480, average=54 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/classes/app.py:248: size=24.9 KiB, count=227, average=113 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/windows/main_window.py:2673: size=24.0 KiB, count=312, average=79 B
      logger:INFO <frozen importlib._bootstrap>:344: size=22.6 KiB, count=322, average=72 B
      logger:INFO /usr/lib/python3.8/dist-packages/gi/overrides/__init__.py:54: size=20.6 KiB, count=310, average=68 B
      logger:INFO /usr/lib/python3.8/html/entities.py:2507: size=20.3 KiB, count=153, average=136 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/windows/models/files_model.py:221: size=19.8 KiB, count=230, average=88 B
      logger:INFO /usr/lib/python3.8/dist-packages/gi/types.py:71: size=19.4 KiB, count=255, average=78 B
      logger:INFO /usr/lib/python3.8/dist-packages/gi/overrides/__init__.py:55: size=18.3 KiB, count=155, average=121 B
      logger:INFO /usr/lib/python3.8/mimetypes.py:92: size=18.3 KiB, count=2, average=9376 B
      logger:INFO /usr/lib/python3.8/http/server.py:881: size=18.1 KiB, count=2, average=9260 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/windows/main_window.py:42: size=18.0 KiB, count=1, average=18.0 KiB
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/windows/export.py:36: size=18.0 KiB, count=1, average=18.0 KiB
      logger:INFO /usr/lib/python3.8/dist-packages/PyQt5/uic/Compiler/qtproxies.py:464: size=18.0 KiB, count=91, average=203 B
      logger:INFO <frozen importlib._bootstrap>:36: size=17.7 KiB, count=263, average=69 B
      logger:INFO /usr/lib/python3.8/dist-packages/openshot_qt/windows/views/timeline_webview.py:2966: size=17.5 KiB, count=209, average=86 B
      logger:INFO /usr/lib/python3.8/threading.py:506: size=17.5 KiB, count=400, average=45 B
      logger:INFO Partition of a set of 632541 objects. Total size = 70159596 bytes.
      logger:INFO  Index  Count   %     Size   % Cumulative  % Kind (class / dict of class)
      logger:INFO      0 395791  63 25207200  36  25207200  36 tuple
      logger:INFO      1  88243  14 23217272  33  48424472  69 dict (no owner)
      logger:INFO      2   5719   1  5512683   8  53937155  77 bytes
      logger:INFO      3  27345   4  5462248   8  59399403  85 list
      logger:INFO      4  37604   6  3060797   4  62460200  89 str
      logger:INFO      5    761   0   824376   1  63284576  90 sip.wrappertype
      logger:INFO      6    642   0   683088   1  63967664  91 sip.enumtype
      logger:INFO      7    759   0   641384   1  64609048  92 dict of sip.wrappertype
      logger:INFO      8  21434   3   601232   1  65210280  93 int
      logger:INFO      9   2988   0   528384   1  65738664  94 types.CodeType

Thank you so much for submitting an issue to help improve OpenShot Video Editor. We are sorry about this, but this particular issue has gone unnoticed for quite some time. To help keep the OpenShot GitHub Issue Tracker organized and focused, we must ensure that every issue is correctly labelled and triaged, to get the proper attention.

This issue will be closed, as it meets the following criteria:

• No activity in the past 180 days
• No one is assigned to this issue

We'd like to ask you to help us out and determine whether this issue should be reopened.

• If this issue is reporting a bug, please can you attempt to reproduce on the latest daily build to help us to understand whether the bug still needs our attention.
• If this issue is proposing a new feature, please can you verify whether the feature proposal is still relevant.

Thanks again for your help!

Let's get this straightforward. It may still be present in later versions (2.6+), and I've seen this under Windows but the only exception is no obvious memory leak.

Let's get this straightforward. It may still be present in later versions (2.6+), and I've seen this under Windows but the only exception is no obvious memory leak.

this memory issue happen on linux too, i used Linux Mint 20.04, however it is happened with longer video which was recorded with device, with youtube videos i guess was no issue.

Here it happens with any video input thats > 30 minutes with Ubuntu 20.04. I usually edit videos for youtube.

I had switched back to Kde'n'live because it looks like even 64GB of ram isnt enough to edit with OpenShot.

Thank you so much for submitting an issue to help improve OpenShot Video Editor. We are sorry about this, but this particular issue has gone unnoticed for quite some time. To help keep the OpenShot GitHub Issue Tracker organized and focused, we must ensure that every issue is correctly labelled and triaged, to get the proper attention.

This issue will be closed, as it meets the following criteria:

• No activity in the past 180 days
• No one is assigned to this issue

We'd like to ask you to help us out and determine whether this issue should be reopened.

• If this issue is reporting a bug, please can you attempt to reproduce on the latest daily build to help us to understand whether the bug still needs our attention.
• If this issue is proposing a new feature, please can you verify whether the feature proposal is still relevant.

Thanks again for your help!

Bump

It is funny how people use that bot for just nuking out the issues and "magically solve them" instead of keeping it open so other people could help fix it.

Thank you so much for submitting an issue to help improve OpenShot Video Editor. We are sorry about this, but this particular issue has gone unnoticed for quite some time. To help keep the OpenShot GitHub Issue Tracker organized and focused, we must ensure that every issue is correctly labelled and triaged, to get the proper attention.

This issue will be closed, as it meets the following criteria:

• No activity in the past 180 days
• No one is assigned to this issue

We'd like to ask you to help us out and determine whether this issue should be reopened.

• If this issue is reporting a bug, please can you attempt to reproduce on the latest daily build to help us to understand whether the bug still needs our attention.
• If this issue is proposing a new feature, please can you verify whether the feature proposal is still relevant.

Thanks again for your help!

Bump

@JacksonRG - This still needs fixing...

This issue has been fixed in 3.0.0. A giant thread + memory leak was found a fixed. If this issue persists on 3.0.0 please re-open this issue again, and please share a test project which reproduces the memory leak. Thx!