HKSV Causes Orphaned FFMpeg Processes, Leading to Ever Increasing Memory Consumption

[TanoopSungha]

After the last two updates I now find that it now gobbles up RAM on the Pi. When I restart the child bridge it then resolves the issue. I am using a Pi5.

[potmat]

That's very odd, I can't account for it as there were no changes to the plugin itself, merely the arguments it passes to ffmpeg. Can you see which process is using the RAM (Homebridge or ffmpeg)?

[TanoopSungha]

The memory only goes down when I reboot or restart the child bridge to google nest sdm - anything else doesn't change it or improve the situation.

[potmat]

Hi @TanoopSungha, I've not be able to reproduce this, my Pi4 still has 95% memory available after weeks of uptime. I'll leave it open if anyone can reproduce it and submit a PR.

[hillaker]

@potmat, I am seeing the same behavior as TanoopSungha on my Raspberry Pi 5-- memory slowly disappears over the course of the day until I have less than 0.5GB available (out of 8).

Please find a screenshot of top sorted by memory-- it looks like ffmpeg is sticking around after a particular recording is complete. Happy to share anything else that might help identify a fix-- I'm running Kali linux, rather than Raspian or similar, and it's the ARM64 version. Not sure if that's relevant for this issue:

Memory after letting the plugin run for ~9 hours (after opening a stream, I left it running without opening a stream earlier in the day and didn't notice an increase in memory usage:

Memory after restarting the plugin's bridge (but not home bridge), taken ~1 minute after the previous screenshot:

If there's something I can do to change my configuration to make it easier to diagnose the issue, please let me know and I'll give it a shot and share the results. Thanks!

Edited again to add that I only have this plugin as one that an outputs video-- my other plugins are switches.

[GillesWi]

Yes, same behaviour on Raspberry Pi 4 (4Gb). After 7 days the memory increases so much that Homebridge just crashes and I need to unplug the Rpi to get it working again. If I restart the plugin within those days there are no problems...

And a week ago I lost my whole homebridge instance, everything was corrupt, so be sure to backup in time ;)!

[ShadowRegent]

Can confirm that I'm also seeing orphaned ffmpeg processes consuming high amounts of memory. I had 16 processes with just 5 hours of uptime. Killing them (with SIGKILL, they were unresponsive to SIGTERM) released memory and left the following logs after killing:

[homebridge-google-nest-sdm] Encountered unexpected error on generator Error: FFMPEG socket closed during read for 8 bytes! at Socket.endHandler (/usr/local/lib/node_modules/homebridge-google-nest-sdm/src/HksvStreamer.ts:190:24) at Socket.emit (node:events:519:28) at TCP.<anonymous> (node:net:338:12)

This was preceded by many errors that looked like this:

Recording download stream 1 is still awaiting generator although stream was closed 10s ago! This is a programming mistake by the camera implementation which prevents freeing up resources.

[potmat]

@ShadowRegent: Ok that means that HKSV is what's causing the issue.

It's starting up streams to record motion (or whatever) events and the FFMpeg process are not getting properly cleaned up.

To everyone in the thread, turning off HKSV will likely solve the issue. I know that's not great, but we can continue to investigate HKSV and hopefully find out why so many processes are being orphaned.

[JulesTKirk]

Chiming in, it also seems that I am having issues with HKSV or something related to FFMpeg :

[8/11/2024, 2:14:32 AM] [homebridge-google-nest-sdm] Encountered unexpected error on generator Error: FFMPEG socket closed during read for 8 bytes!
    at Socket.endHandler (/volume1/homebridge/node_modules/homebridge-google-nest-sdm/src/HksvStreamer.ts:190:24)
    at Socket.emit (node:events:514:28)
    at TCP.<anonymous> (node:net:337:12)

which was preceeded by a high volume of these errors :

Recording download stream 1 is still awaiting generator although stream was closed 10s ago! This is a programming mistake by the camera implementation which prevents freeing up resources.
[8/9/2024, 12:02:21 PM] [homebridge-google-nest-sdm] Encountered unexpected error on generator Error: Unexpected state!
    at HksvStreamer.generator (/volume1/homebridge/node_modules/homebridge-google-nest-sdm/src/HksvStreamer.ts:141:19)
    at DoorbellStreamingDelegate.handleRecordingStreamRequest (/volume1/homebridge/node_modules/homebridge-google-nest-sdm/src/StreamingDelegate.ts:559:24)

I'm running on a Synology DS418 NAS with about 2GB of RAM so it was filled pretty fast and left the system completely unresponsive :(