blakeblackshear
[Config Support]: Audio with Reolink RLC-520 and webRTC
Describe the problem you are having
I'm testing out the new Beta and I'm a bit confused what's the best config to use. I've followed the updated docs and the specific config for Reolink cameras
When using the webRTC card in HA I get no audio unless I set the mode: mse. However, this mode doesn't work at all on iOS so I have to set it to mode: webrtc.
I also get no audio with the Frigate card using the default camera entity. I presume this is because the integration uses the MJPEG stream?
I've also noticed when playing the restream on VLC the audio keeps breaking up with the http one. After about ten seconds it stabilises and plays OK. The rtsp one works a lot better with maybe only a few seconds of breaking up before everything plays fine.
What are the advantages of using go2rtc for the detect and recording input instead of direct from the camera? Would this increase CPU usage? I want to preserve as much CPU as possible as my system is already running between 70-90% CPU usage.
Version
0.12.0-12D51D3
Frigate config file
environment_vars:
LIBVA_DRIVER_NAME: i965
mqtt:
host: xxxx
user: xxxx
password: xxxx
detectors:
coral_pci:
type: edgetpu
device: pci
logger:
# Optional: default log level (default: shown below)
default: info
# Optional: module by module log level configuration
# logs:
# frigate.mqtt: debug
# frigate.event: debug
birdseye:
enabled: False
mode: continuous
objects:
track:
- person
- car
- cat
- dog
filters:
person:
threshold: 0.7
max_ratio: 0.6
record:
enabled: True
retain:
days: 0
events:
retain:
default: 20
mode: all
pre_capture: 2
post_capture: 10
snapshots:
enabled: True
timestamp: False
bounding_box: True
crop: False
retain:
default: 20
detect:
stationary:
interval: 10
ffmpeg:
hwaccel_args: preset-vaapi
go2rtc:
streams:
front: ffmpeg:https://192.168.xxx.xxx/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=xxxx&password=xxxx#video=copy#audio=copy#audio=opus
front_rtsp:
- rtsp://xxxx:[email protected]:554/h264Preview_01_main
- ffmpeg:front_rtsp
cameras:
###############################################################front########################################################################
front:
ffmpeg:
inputs:
- path: rtsp://ccab4aaf-frigate-fa-beta:8554/front?video=copy&audio=aac
roles:
- record
- detect
input_args: preset-rtsp-restream
detect:
max_disappeared: 100
width: 2048
height: 1536
fps: 15
mqtt:
required_zones:
- front_drive
- front_path
motion:
mask:
- 0,80,780,79,795,0,0,0
- 0,1429,0,1089,315,611,436,529,550,548,518,737,408,795
- 1272,612,1363,826,1575,782,1509,595,1515,487,1330,457
objects:
filters:
car:
mask:
- 22,0,2048,0,2048,508,1322,436,589,432,0,484,0,0
- 670,573,195,1536,0,1536,0,554
- 1629,1536,2048,1536,2048,627,1220,564
dog:
max_area: 10000
person:
max_area: 250000
min_area: 5000
mask:
- 2048,0,2048,384,1220,261,631,268,0,336,0,0
zones:
front_drive:
coordinates: 0,1536,865,1536,1705,1536,1309,720,531,710,357,950
Relevant log output
go2rtc:
2023-01-22 11:38:09.272369512 11:38:09.272 INF go2rtc version 0.1-rc.9 linux/amd64
2023-01-22 11:38:09.273466461 11:38:09.273 INF [api] listen addr=:1984
2023-01-22 11:38:09.273912790 11:38:09.273 INF [rtsp] listen addr=:8554
2023-01-22 11:38:09.274546971 11:38:09.274 INF [webrtc] listen addr=:8555
2023-01-22 11:38:09.274858581 11:38:09.274 INF [srtp] listen addr=:8443
2023-01-22 13:33:38.306759825 13:33:38.306 WRN github.com/AlexxIT/go2rtc/cmd/rtsp/rtsp.go:203 > error=EOF
Frigate stats
{"back_garden":{"camera_fps":5.0,"capture_pid":245,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":264,"pid":224,"process_fps":5.0,"skipped_fps":0.0},"back_patio":{"camera_fps":7.1,"capture_pid":250,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":263,"pid":236,"process_fps":7.1,"skipped_fps":0.0},"cpu_usages":{"%Cpu(s):":{"cpu":"id,","mem":"8.1"},"1":{"cpu":"0.0","mem":"0.0"},"101":{"cpu":"9.0","mem":"13.6"},"10721":{"cpu":"0.0","mem":"0.1"},"10725":{"cpu":"1.3","mem":"0.1"},"10726":{"cpu":"3.3","mem":"0.9"},"113":{"cpu":"0.0","mem":"0.3"},"122":{"cpu":"0.0","mem":"0.2"},"123":{"cpu":"0.0","mem":"0.1"},"15":{"cpu":"0.0","mem":"0.0"},"16":{"cpu":"0.0","mem":"0.0"},"206":{"cpu":"0.0","mem":"2.0"},"212":{"cpu":"0.3","mem":"0.2"},"213":{"cpu":"2.3","mem":"2.9"},"214":{"cpu":"4.0","mem":"2.7"},"217":{"cpu":"12.6","mem":"3.3"},"220":{"cpu":"0.7","mem":"2.7"},"224":{"cpu":"1.7","mem":"2.8"},"225":{"cpu":"0.0","mem":"0.3"},"236":{"cpu":"1.3","mem":"2.8"},"237":{"cpu":"0.0","mem":"0.3"},"238":{"cpu":"24.6","mem":"2.8"},"239":{"cpu":"0.0","mem":"0.3"},"24":{"cpu":"0.0","mem":"0.0"},"242":{"cpu":"1.7","mem":"2.6"},"243":{"cpu":"0.0","mem":"0.3"},"245":{"cpu":"8.6","mem":"2.8"},"247":{"cpu":"18.6","mem":"2.3"},"25":{"cpu":"0.0","mem":"0.0"},"250":{"cpu":"2.7","mem":"2.6"},"253":{"cpu":"2.0","mem":"0.6"},"26":{"cpu":"0.0","mem":"0.0"},"261":{"cpu":"5.0","mem":"0.5"},"263":{"cpu":"3.3","mem":"0.6"},"264":{"cpu":"9.3","mem":"1.5"},"267":{"cpu":"4.0","mem":"0.5"},"27":{"cpu":"0.0","mem":"0.0"},"270":{"cpu":"5.6","mem":"0.5"},"28":{"cpu":"0.0","mem":"0.0"},"29":{"cpu":"0.0","mem":"0.0"},"30":{"cpu":"0.0","mem":"0.0"},"31":{"cpu":"0.0","mem":"0.0"},"40":{"cpu":"0.0","mem":"0.0"},"41":{"cpu":"0.0","mem":"0.0"},"77":{"cpu":"0.0","mem":"0.0"},"78":{"cpu":"0.0","mem":"0.0"},"79":{"cpu":"0.0","mem":"0.0"},"87":{"cpu":"1.0","mem":"0.5"},"MiB":{"cpu":"1660.1","mem":"avail"},"PID":{"cpu":"%CPU","mem":"%MEM"},"Tasks:":{"cpu":"stopped,","mem":"0"},"top":{"cpu":"users,","mem":"load"}},"detection_fps":7.8,"detectors":{"coral_pci":{"detection_start":0.0,"inference_speed":10.8,"pid":213}},"front":{"camera_fps":15.0,"capture_pid":238,"detection_enabled":1,"detection_fps":7.8,"ffmpeg_pid":247,"pid":217,"process_fps":15.0,"skipped_fps":0.0},"front_door":{"camera_fps":7.3,"capture_pid":242,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":253,"pid":220,"process_fps":7.3,"skipped_fps":0.0},"gpu_usages":{"error-gpu":{"gpu":-1,"mem":-1}},"service":{"latest_version":"0.11.1","storage":{"/dev/shm":{"free":1944.2,"mount_type":"tmpfs","total":1957.1,"used":12.9},"/media/frigate/clips":{"free":205737.1,"mount_type":"ext4","total":471807.4,"used":246857.6},"/media/frigate/recordings":{"free":205737.1,"mount_type":"ext4","total":471807.4,"used":246857.6},"/tmp/cache":{"free":1917.6,"mount_type":"tmpfs","total":1957.1,"used":39.6}},"temperatures":{"apex_0":58.3},"uptime":9388,"version":"0.12.0-12d51d3"}}
Operating system
HassOS
Install method
HassOS Addon
Coral version
M.2
Any other information that may be helpful
FFPROBE:
"[\n {\n "return_code": 0,\n "stderr": {},\n "stdout": {\n "programs": [],\n "streams": [\n {\n "avg_frame_rate": "0/0",\n "codec_long_name": "H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10",\n "height": 1536,\n "width": 2048\n },\n {\n "avg_frame_rate": "0/0",\n "codec_long_name": "AAC (Advanced Audio Coding)"\n }\n ]\n }\n }\n]"
I also get no audio with the Frigate card using the default camera entity. I presume this is because the integration uses the MJPEG stream?
The frigate card can be set to use the jsmpeg stream (different from mjpeg) but it will use RTSP by default.
What are the advantages of using go2rtc for the detect and recording input instead of direct from the camera? Would this increase CPU usage? I want to preserve as much CPU as possible as my system is already running between 70-90% CPU usage.
Go2rtc reduces connections to the camera and provides a clean and stable stream to frigate. There is minimal impact on CPU especially because viewing the live view in HA will have no additional usage and will be faster.
Also for
rtsp://ccab4aaf-frigate-fa-beta:8554/front?video=copy&audio=aac
you should be using 127.0.0.1
Also if you're worried about CPU usage why are you using the full res stream at 15 fps? That seems totally unnecessary, even if you need the higher res you can lower the fps value in frigate
The frigate card can be set to use the jsmpeg stream (different from mjpeg) but it will use RTSP by default.
Sorry yes, I meant jsmpeg. This has no audio correct? I will try with RTSP, I'm guessing the benefits of using RTSP is that it will work with audio and on iphones no problem.
Go2rtc reduces connections to the camera and provides a clean and stable stream to frigate. There is minimal impact on CPU especially because viewing the live view in HA will have no additional usage and will be faster.
Gotcha.
Also for
rtsp://ccab4aaf-frigate-fa-beta:8554/front?video=copy&audio=aacyou should be using 127.0.0.1
Yes I had some problems initially with the config and I tried the host name. Have switched it back to loopback now.
Also if you're worried about CPU usage why are you using the full res stream at 15 fps? That seems totally unnecessary, even if you need the higher res you can lower the fps value in frigate
Good spot, I had upped that as I was trying to improve detection times/accuracy at night. Would you say 7 was plenty? The rest are set to that or 5.
I've been checking some recordings and some have the audio missing completely so something's not right. Also it appears to drop out for a split second occasionally on the clips that do have audio.
Do I need this output arg? record: preset-record-generic-audio-aac
Sorry yes, I meant jsmpeg. This has no audio correct? I will try with RTSP, I'm guessing the benefits of using RTSP is that it will work with audio and on iphones no problem.
Correct, jsmpeg doesn't have audio
Good spot, I had upped that as I was trying to improve detection times/accuracy at night. Would you say 7 was plenty? The rest are set to that or 5.
If you're not I'd suggest setting motion -> improve_contrast to true. 5 or 7 should be fine.
I've been checking some recordings and some have the audio missing completely so something's not right. Also it appears to drop out for a split second occasionally on the clips that do have audio.
Do I need this output arg? record: preset-record-generic-audio-aac
You should use
record: preset-record-generic-audio-copy since you've already specified aac
I'm not 100% sure this is correct as it's not what the docs are saying but it seems to have solved the issue of no audio with webRTC. I removed audio=copy from the restream config. So the example in the docs looks like this now:
go2rtc:
streams:
reolink: ffmpeg:http://reolink_ip/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=username&password=password#video=copy#audio=opus
Also, maybe this is needs to be added under the Reolink camera specific config in the docs?
output_args:
record: preset-record-generic-audio-copy
Looks as though the clips are being recorded with audio, without any dropouts now. However, I've noticed there is a sync issue with the audio being delayed from the video. Is there anything I can do to fix that?
Without #audio=copy the stream is only providing opus audio and that is being used for the recordings. It's definitely recommended to use AAC and #copy shouldn't necessarily cause any issues unless the camera itself is choosing bad audio. You could use #video=copy#audio=aac#audio=opus and see how that works
Not sure what you mean the camera itself is choosing bad audio? Isn't it a fixed codec?
audio=aac is already in the camera input, do I need to adjust this too?
- path: rtsp://127.0.0.1:8554/front?video=copy&audio=aac #http://192.168.xxx.xxx/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=xxxx&password=xxxx
roles:
- record
- detect
Not sure what you mean the camera itself is choosing bad audio? Isn't it a fixed codec?
I meant sending bad audio.
And no you want to leave ?audio=AAC as that's telling go2rtc to send the audio AAC track (instead of opus)
I got some ffmpeg crashes with audio=aac when starting the addon:
ffmpeg.front.detect ERROR : rtsp://127.0.0.1:8554/front?video=copy&audio=aac: Invalid data found when processing input
But it seems to settle down and stop crashing. However, webRTC has no audio still.
Current config is the same as in the OP apart from I changed these lines (as suggested):
go2rtc:
streams:
front: ffmpeg:https://192.168.xxx.xxx/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=xxxx&password=xxxx#video=copy#audio=aac#audio=opus
In the camera config:
output_args:
record: preset-record-generic-audio-copy
Doesn't seem like you ever posted the ffmpeg logs, those would be helpful to see. In any case, is everything working as expected right now?
Doesn't seem like you ever posted the ffmpeg logs, those would be helpful to see.
How do I get those? I added ffmpeg logging to the config but I wasn't sure what '<sorted_roles>' was for. Without it I get a validation error:
logger:
# Optional: default log level (default: shown below)
default: info
# Optional: module by module log level configuration
logs:
ffmpeg.front
In any case, is everything working as expected right now?
No. Some clips are being recorded without audio still. Ones that do have audio, it's out of sync.
WebRTC is working with audio though which was one of my goals. Now if I could only get it working on iOS outside of my LAN. I've done port forwarding and added local ip, wan ip and stun to candidates: but still nothing.
How do I get those? I added ffmpeg logging to the config but I wasn't sure what '<sorted_roles>' was for. Without it I get a validation error:
You said ffmpeg was crashing so you must have the logs for that?
No. Some clips are being recorded without audio still. Ones that do have audio, it's out of sync.
Not sure why that would be the case, no one has reported that and it doesn't do that with my reolink cams.
You may be better off just using frigate config directly with the camera stream as before.
WebRTC is working with audio though which was one of my goals. Now if I could only get it working on iOS outside of my LAN. I've done port forwarding and added local ip, wan ip and stun to candidates: but still nothing.
This is usually due to having a symmetrical NAT which causes problems, the go2rtc docs cover this in more detail.
You said ffmpeg was crashing so you must have the logs for that?
That was only when I had audio=aac in the restream config. Without it and audio=copy webRTC works with audio and I don't get any crashes.
Not sure why that would be the case
Me neither but I'm sure I won't be the only one. Maybe it's something with the older firmware of the RLC-520 and it's http stream? Maybe I'll try feeding go2rtc the rtsp stream instead. If there's any other logs or tests I can do, let me know.
I've noticed that some of the clips which don't have audio, are in fact only missing a channel. For some reason the Frigate web player won't play the audio at all. If I download the clip and play in VLC it plays but there is a channel missing. There is also some significant lag in the audio stream. So there's something up with the re-stream and this RLC-520. I have the RLC-520A which has newer firmware and it seems ok. I will go back to using the camera stream for Frigate. Maybe I should raise this with the go2rtc project?
newer firmware and it seems ok
I am noticing several seconds of audio delay on my duo 2 which is fairly new. the recordings from the low res stream have a delay but the h265 stream does not.
My rlc520 has the delay too.
So there's something up with the re-stream and this RLC-520.
Maybe just use the direct stream from the camera for frigate then. You can try raising an issue with go2rtc and see what comes of it.
this may be fixed https://github.com/blakeblackshear/frigate/pull/5231 as well
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}