rtsp source issues in 1.9.3 and 1.9.4
another user posted this on the Frigate repo and i've been experiencing the same issue myself. confirmed everything is fine running 1.9.2 or before. both 1.9.3 and 1.9.4 have the issue alluded to below: https://github.com/blakeblackshear/frigate/discussions/12117
I can provide my own logs this evening as well to help diagnose the issue.
I scanned open issues and didnt see one that appeared to be related to this--but if I missed it and one already exists my bad.
I'm able to consistently reproduce this. Let me know if there's a certain way I should set logging to provide useful information. From what I've looked at on my own, I can't see anything that would be the culprit.
Since you are using an audio stream for webrtc, do you notice the number of passive consumers for the audio increasing and never clearing until the crash?
I am noticing a similar issue where the memory will climb as the number of consumers continues to climb until (for me) go2rtc crashes and restarts. I don't run out of memory though.
Related to this, see if you see this in your go2rtc logs
Normal audio producer stop
11:22:22.011 AM | debug | [streams] stop producer url=ffmpeg:CAMERA-1#audio=opus
11:22:22.011 AM | trace | [exec] kill with signal=9
11:22:22.017 AM | debug | [rtsp] handle error=EOF # <-- this is sometimes not there
Producer stop that keeps an audio consumer up for me
11:22:22.120 AM | debug | [streams] stop producer url=ffmpeg:CAMERA-2#audio=opus
11:22:22.120 AM | trace | [exec] kill with signal=9
11:22:22.125 AM | debug | [rtsp] handle error=read tcp 127.0.0.1:8554->127.0.0.1:45878: read: connection reset by peer
Note: 45878 is the port listed for the passive audio rtsp+tcp consumer that I see for FFmpeg receiving the aac audio from the camera.
Error that causes stream to die and not come back up for me
11:22:22.125 AM | debug | [rtsp] handle error="read tcp 127.0.0.1:8554->127.0.0.1:56678: i/o timeout"
This happens here: https://github.com/AlexxIT/go2rtc/blob/a4885c2c3abce58074d04878bba0d72105642a9b/pkg/rtsp/conn.go#L143-L147
I have found that using the direct camera connection URL instead of the go2rtc stream for the opus audio ffmpeg stream fixes my issues, but slows down the connection so there is a lot of delay when first connecting. Something happening with the go2rtc audio set up that causes timeout issues. Haven't figured out where yet.
Can confirm that there is something broken in the latest versions. Frigate + go2rtc 1.9.4:
- minor decoding errors, which didn't show in the recording
- "left" something missing, which lead to a missing chunk in the recording
- "The go2rtc service exited with code 256 (by signal 9)"
- once it even messed up the camera connection to the point where the camera was unreachable and had to be reset.
I've tried with the latest dev builds too, same problems. With go2rtc 1.9.2 everything is rock solid now and there are no more errors for over a week.
has anyone tried 1.9.8 to see if this is still broken w/ Frigate? if not I can test this weekend.