go2rtc
go2rtc copied to clipboard
AlexxIT
Add audio mixer for backchannel
When multiple consumers want to send audio back to a single producer (e.g., IP camera with backchannel), only one consumer could be active at a time. The audio from additional consumers would be lost or cause conflicts.
Solution
This PR introduces an optional audio mixer that combines multiple backchannel audio streams into a single stream before sending it to the producer.
How it works
- When enabled with the
#mixflag, an FFmpeg-based audio mixer is automatically created when the first backchannel consumer connects - Additional backchannel consumers are added as parents to the mixer
- The mixer uses FFmpeg's
amixfilter to combine all audio streams in real-time - The mixed audio is then sent to the underlying producer (camera/device)
Usage
Add #mix to your stream source URL:
streams:
camera_backchannel:
- rtsp://admin:[email protected]/stream#mix
Without #mix, the behavior remains unchanged - only the first backchannel consumer will work.
Features
- Opt-in: Mixing is disabled by default
- Automatic lifecycle: FFmpeg process starts when 2+ consumers connect, stops when <2 remain
- Codec support: Works with all audio codecs
-
Visualization: Mixers are displayed in the
/api/streamsendpoint and/netgraph
Demo
GET /api/streams?src=tapo
{
"producers": [
{
"id": 2148886400,
"format_name": "tapo",
"protocol": "http",
"remote_addr": "192.168.178.128:8800",
"medias": [
"video, recvonly, H264",
"audio, recvonly, PCMA/8000",
"audio, sendonly, PCMA/8000"
],
"bytes_recv": 14068400,
"receivers": [
{
"id": 2,
"codec": {
"codec_name": "h264",
"codec_type": "video"
},
"childs": [
3,
10,
14
],
"bytes": 12457273,
"packets": 1811
},
{
"id": 4,
"codec": {
"codec_name": "pcm_alaw",
"codec_type": "audio",
"sample_rate": 8000
},
"childs": [
5,
12,
16
],
"bytes": 966656,
"packets": 944
}
],
"senders": [
{
"id": 8,
"codec": {
"codec_name": "pcm_alaw",
"codec_type": "audio",
"sample_rate": 8000
},
"parent": 7,
"bytes": 877440,
"packets": 5484
}
],
"mixers": [
{
"id": 7,
"codec": {
"codec_name": "pcm_alaw",
"codec_type": "audio",
"sample_rate": 8000
},
"parents": [
6,
11,
15
],
"childs": [
8
],
"bytes": 1532160,
"packets": 9576
}
]
}
],
"consumers": [
{
"id": 1,
"format_name": "preload",
"medias": [
"video, sendonly, ANY",
"audio, sendonly, ANY",
"audio, recvonly, ANY"
],
"bytes_send": 13423929,
"receivers": [
{
"id": 6,
"codec": {
"codec_name": "ANY",
"codec_type": ""
},
"childs": [
7
]
}
],
"senders": [
{
"id": 3,
"codec": {
"codec_name": "h264",
"codec_type": "video"
},
"parent": 2,
"bytes": 12457273,
"packets": 1811
},
{
"id": 5,
"codec": {
"codec_name": "pcm_alaw",
"codec_type": "audio",
"sample_rate": 8000
},
"parent": 4,
"bytes": 966656,
"packets": 944
}
]
},
{
"id": 9,
"format_name": "webrtc",
"protocol": "ws+udp",
"remote_addr": "[fd13:1c98:5a6b:0:1023:52e0:c779:9286]:58762 host",
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36",
"medias": [
"video, sendonly, VP8, VP9, H264, AV1, H265",
"audio, recvonly, OPUS/48000/2, G722/8000, PCMU/8000, PCMA/8000",
"audio, sendonly, OPUS/48000/2, G722/8000, PCMU/8000, PCMA/8000, L16, PCML"
],
"bytes_recv": 989647,
"bytes_send": 12297412,
"receivers": [
{
"id": 11,
"codec": {
"codec_name": "pcm_alaw",
"codec_type": "audio",
"sample_rate": 8000
},
"childs": [
7
],
"bytes": 878560,
"packets": 5491
}
],
"senders": [
{
"id": 10,
"codec": {
"codec_name": "h264",
"codec_type": "video"
},
"parent": 2,
"bytes": 11267537,
"packets": 1647
},
{
"id": 12,
"codec": {
"codec_name": "pcm_alaw",
"codec_type": "audio",
"sample_rate": 8000
},
"parent": 4,
"bytes": 879616,
"packets": 859
}
]
},
{
"id": 13,
"format_name": "webrtc",
"protocol": "ws+udp",
"remote_addr": "[fd13:1c98:5a6b:0:1023:52e0:c779:9286]:49303 host",
"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36",
"medias": [
"video, sendonly, VP8, VP9, H264, AV1, H265",
"audio, recvonly, OPUS/48000/2, G722/8000, PCMU/8000, PCMA/8000",
"audio, sendonly, OPUS/48000/2, G722/8000, PCMU/8000, PCMA/8000, L16, PCML"
],
"bytes_recv": 736574,
"bytes_send": 9217431,
"receivers": [
{
"id": 15,
"codec": {
"codec_name": "pcm_alaw",
"codec_type": "audio",
"sample_rate": 8000
},
"childs": [
7
],
"bytes": 653600,
"packets": 4085
}
],
"senders": [
{
"id": 14,
"codec": {
"codec_name": "h264",
"codec_type": "video"
},
"parent": 2,
"bytes": 8450403,
"packets": 1225
},
{
"id": 16,
"codec": {
"codec_name": "pcm_alaw",
"codec_type": "audio",
"sample_rate": 8000
},
"parent": 4,
"bytes": 654336,
"packets": 639
}
]
}
]
}
/net.html
Related: https://github.com/AlexxIT/go2rtc/issues/1899 https://github.com/AlexxIT/go2rtc/issues/1828
This looks awesome!
It fixes https://github.com/AlexxIT/go2rtc/issues/1828.
In future, this could also be reused for https://github.com/AlexxIT/go2rtc/issues/105.
