AlexxIT
Improve latency for when AAC needs to be transcoded
I just noticed this news: https://www.phoronix.com/news/FFmpeg-Lands-WHIP-Muxer
And I wonder if go2rtc can somehow leverage this to improve latency when using FFMpeg.
I don't know anything about it, please feel free to just close this if not relevant.
WebRTC is a very “expensive” protocol in terms of implementation complexity. I think wherever it is possible to use simpler protocols - it is necessary to use them.
Right now, the main protocol between FFmpeg and go2rtc is RTSP TCP. Which is much simpler than WebRTC, but still not perfect.
Where possible - I try to use simpler protocols: https://github.com/AlexxIT/go2rtc/blob/ae8145f26696ffc19a13fea4a21b4a5306fc0e3a/internal/ffmpeg/ffmpeg.go#L357-L378
In terms of latency - all the protocols in use now are good. WebRTC won't do any better. It may even make it worse.
The only concern I have are setups like these:
Fixing buggy stream:
streams:
my_cam: ffmpeg:rtsp://[email protected]/av_stream/ch0#video=copy#audio=copy
Transcoding AAC to OPUS for 2-way audio over WebRTC:
streams:
my_cam:
- rtsp://[email protected]/av_stream/ch0
- ffmpeg:my_cam#audio=opus
Which adds noticeable latency and is very detrimental for 2-way audio communication experience.
As per the buggy stream, maybe go2rtc will improve over time to better handle it and will avoid needing ffmpeg in more cases.
As per AAC -> OPUS, maybe go2rtc could add native transcoding without ffmpeg, but I am not sure this can be elegantly done. I found a good Opus library, but couldn't find any good one for AAC.
Yet another solution would be to add the low latency flags to the ffmpeg command, but that often causes other issues.
I changed this issue's title to clarify my intention.
This is useless AAC header parser. Go2rtc has it.
I'd add an AAC decoder, but no one has written one in Go yet.