go2rtc icon indicating copy to clipboard operation
go2rtc copied to clipboard

Add bidirectional audio support for HomeKit cameras

Open andersbaekgaard opened this issue 1 month ago • 2 comments

Summary

  • Add bidirectional audio support for HomeKit cameras (speaker/talkback functionality)
  • Audio path: WebRTC/Browser mic → FFmpeg transcoding (OPUS→AAC-ELD) → HomeKit speaker via SRTP
  • Implements connection tracking to prevent duplicate consumer-track pairs
  • Adds accept-audio parameter to FFmpeg for waiting on external audio sources
  • Thread-safe SRTP writes with mutex protection

Requirements

  • FFmpeg must be compiled with libfdk_aac for AAC-ELD encoding (required for HomeKit speaker audio)

Test plan

  • [x] Tested with Aqara G4 doorbell - microphone works in Home app
  • [x] Verified audio latency is optimal for the device (~3s jitter buffer is device firmware, not go2rtc)
  • [ ] Test with other HomeKit cameras that support bidirectional audio

Implemented by Anders Baekgaard in cooperation with Anthropic Claude Opus 4.5 https://claude.ai/

andersbaekgaard avatar Dec 14 '25 11:12 andersbaekgaard

Thanks! The implementation is quite complex and introduces a lot of confusion into the core. I won't accept it in this form, but I will try to understand the idea and adapt it when I have time.

And I really don't like the idea of ​​using custom ffmpeg. This is one of the block factors for implementing two-way for HomeKit cameras.

AlexxIT avatar Dec 15 '25 09:12 AlexxIT

Thank you for the prompt feedback.

I understand your concern about complexity, the PR got much more complex than I expected when first looking into it. I could not get streams matching to work for the aac-eld transcoder producer, due to archtiectural limitation. With your insight, you could maybe make it simpler.

Note that there is also a simple bug fix.

andersbaekgaard avatar Dec 15 '25 09:12 andersbaekgaard