2-Way Audio is Broken with AAC on ISAPI Cameras

[Marcoske23]

I'm trying to set up 2-way audio on my camera using ISAPI, but it just won't work if I set the camera audio to AAC. I figured AAC or Opus would be better, so I tried this configuration:

Important Context: I'm using Frigate's interface in Home Assistant, so the whole communication chain (including the microphone) is running over HTTPS, and the other cameras' 2-way audio works fine.

My go2rtc configuration was this:

streams: 
  frontal01_HD:

    #1. Main attempt with ISAPI (camera audio is AAC)
    - isapi://admin:[email protected]:80/
    - rtsp://admin:[email protected]/streaming/channels/101#video=copy#audio=copy
    #2. Tried forcing Opus too, just in case
    - ffmpeg:frontal01_HD#timeout=30#video=h264#audio=opus#hardware=vaapi

The mic icon either doesn't show up in Frigate, or when I try to talk, nothing comes out of the camera speaker. All the other streams work fine, and other cameras' 2-way audio works (tapo and thingino cameras).

The Fix (What worked)

The weird thing is, I went into the camera settings and changed the audio codec from AAC to G.711 ulaw.

Boom! The microphone appeared instantly, and 2-way audio started working perfectly.

The Question

Why does it only work with G.711? I thought AAC or Opus was better/more modern, and I even tried to transcode to Opus in the config.

Is the ISAPI/RTSP/ONVIF backchannel hardcoded to only accept G.711?

If so, maybe that should be clearly noted somewhere. Spent way too much time debugging why the "better" codec was actually the wrong one for the mic function, even though the entire HA setup is using HTTPS.

List of avaiable codecs in hikvision camera:

[AlexxIT]

1. WebRTC doesn't support AAC for both - input and output. https://github.com/AlexxIT/go2rtc?tab=readme-ov-file#codecs-madness

2. ISAPI hardcoded only to PCMA/PCMU. I could add OPUS, but looks like ISAPI doesn't support it. https://github.com/AlexxIT/go2rtc/blob/f8c88cfbe04b11d334d49055c94682e76908819d/pkg/isapi/client.go#L68-L76

[willag]

I am running into this as exact issue as well. I tried everything to get 2 way audio to work and the only thing that worked was to switch the cameras audio to G.711. G.711 sounds much worse than ACC. I understand this is an issue beyond go2rtc's control. But I would like to make sure I understand the issue completely as it may impact my choices when I purchase cameras in the future.

Is this a Hikvision specific problem with their custom ISAPI implementation? I have all Hikvision cameras and don't have anything else available to test against. Do Dahau (or other brands) cameras support ACC audio output with opus input? Does webrtc support different codecs for sending and receiving audio or do they need to match?

I find this issue to be frustrating because ACC sounds so much better than G.711. But so far with all of my Hikvision cameras I am basically stuck between choosing high quality single direction audio or low quality two way audio. If this is a limitation that all cameras share or is imposed by webrtc then that is fine with me. But if there is something I can look for when purchasing cameras that solves this issue then I would love to know what it is.

Thanks so much for all the hard work you guys put in on things like go2rtc. Its an excellent project that enabled some really great things.

[AlexxIT]

The link above about codecs answers many questions. AAC is not supported in WebRTC. This may be due to licensing issues, or perhaps due to Google's policies. Go2rtc doesn't currently support backchannel transcoding. This feature may be added in the future. There are very few cameras on the market that support OPUS. The codec is popular only in Google projects.