watchdog
watchdog copied to clipboard
gorakhargosh
InotifyEvents seem to not be closed / garbage collected
Platform we're running on : Ubuntu 18.04, Python 3.6.8
Context: We have a long running service that uses watchdog to watch a log file and parse it when it's moved.
Anyways, we've been tracing a memory leak in it with tracemalloc and it appears like watchdog is the culprit, it's the top memory consumer compared to the snapshot taken when starting the app:
/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py:0: size=2470 KiB (+2388 KiB), count=47172 (+47157), average=54 B
some more showing it's one of the top mem incrementors:
/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py:243: size=1280 KiB (+704 KiB), count=2 (+0), average=640 KiB
/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py:462: size=1634 KiB (+184 KiB), count=33095 (+3379), average=51 B"
/opt/app/env/lib/python3.6/site-packages/watchdog/utils/unicode_paths.py:59: size=151 KiB (+151 KiB), count=2016 (+2016), average=77 B
/opt/app/env/lib/python3.6/site-packages/watchdog/events.py:118: size=114 KiB (+112 KiB), count=2558 (+2531), average=46 B
When looking at the tracebacks for this, these are the top memory consuming lines:
81837 memory blocks: 3994.4 KiB
File "/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py", line 462
self._wd = wd
56312 memory blocks: 3342.1 KiB
File "/opt/app/env/lib/python3.6/posixpath.py", line 90
path += b
56313 memory blocks: 3080.0 KiB
File "/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py", line 323
inotify_event = InotifyEvent(wd, mask, cookie, name, src_path)
217 memory blocks: 2573.5 KiB
File "/usr/lib/python3.6/ssl.py", line 651
return self._sslobj.peer_certificate(binary_form)
2 memory blocks: 2560.4 KiB
File "/opt/app/env/lib/python3.6/site-packages/watchdog/observers/inotify_c.py", line 243
self._moved_from_events[event.cookie] = event
So it seems to me that the issue is the InotifyEvent objects that get created are never closed / garbage collected until the Observer object stops / joins.
I've tried issuing a del event after we are finished using it but this has no perceptible effect, something else is holding on to a reference to the event.
How could we "forcibly" close or delete the InotifyEvent or FileSystemEvent when we're done with it?
Can confirm. In the project i'm currently working on we use watchdog as well. We also used the tracemalloc package to look for the top memory consumers, and using the InotifyObserver of watchdog, it was leading the charts with increasing memory usage over time. The scripts were running on a Ubuntu 19.10 machine, Python 3.7.4. The only workaround for us was to explictly use the PollingObserver instead of the Observer class.
Definitely, something may need to be improved. Do you have a clue or even a patch almost ready for that? :crossed_fingers: :)
Hey I have the problem on my embedded system. Each Observer I created on a folder with a lot of activity will increase his memory and provoke my system to kill process.
It's a big issue, does anyone has a solution ?
Polling is not an option for us
Does this problem still persists? Or has it been fixed in one of the future releases.
