Audio Issues | Users are unable to hear each other speak on VPN

[william-vz]

Description

Users are unable to hear each other speak. Sometimes, person A can hear person B speak, but person B cannot hear person A speak. Other times, both people are unable to hear each other speak. This seems to only happen when at least one user is behind company VPN.

Initial Investigation

• Using Wireshark, I am able to confirm that this is not a network issue. Nothing is being blocked by our firewall and we are seeing data transmitted successfully from/to our EC2 Streaming Servers.
• Using the WebRTC Debug Panel and the presence of speaking indicators (aka mouth / head moving) on the scene, I confirmed that voice data is successfully transmitted from person A's client (speaker) to person B's client (listener), even though there is no audio output from person B's client.

Hypothesis & Current Workaround

The issue appears to be coming from the enableChromeAEC function in audio-systems.js. The local webRTC connection is not being successfully established for every client, likely due to some race conditions. After some debugging, it looks like the RTCPeerConenctions have iceConnectionState that are not being accounted for in the function.

Example 1: RTCPeerConnection has a status of closed.

• To get around this, I added additional conditions to check iceConnectionState against. Currently, lines 68 and 77.

if (outboundPeerConnection.iceConnectionState === "disconnected" || 
     outboundPeerConnection.iceConnectionState === "failed" || 
     outboundPeerConnection.iceConnectionState === "closed")

if (inboundPeerConnection.iceConnectionState === "disconnected" || 
     inboundPeerConnection.iceConnectionState === "failed" || 
     inboundPeerConnection.iceConnectionState === "closed")

Example 2: RTCPeerConnection has a status of new, even after local and remote descriptions were set on that connection. I don't know why this ever occurs, but when it does, it doesn't trigger the iceconnectionstatechanged event listener. My assumption is that because a new RTCPeerConnection was instantiated, technically nothing was changed.

• To get around this, I added the following condition above the try/catch block.

  if (
    inboundPeerConnection.iceConnectionState === "new" ||
    outboundPeerConnection.iceConnectionState === "new"
  ) 
    performDelayedReconnect(gainNode);
  }

To Reproduce Steps to reproduce the behavior:

1. Person A logs onto VPN.
2. Person B logs onto VPN.
3. Both users enter scene.
4. Person A unmutes and speaks.

Browser Version

• Chrome Version 100.0.4896.127 (Official Build) (x86_64) (This bug is not version specific as this problem has been plaguing us for over a year)

[markusTraber]

Hi @william-vz! We have experienced similar issues. Unfortunately I cannot confirm anything about the network environment of the affected people, since this only happened on public events and people were not aware of their network specifics. Nevertheless it seemed to only affect Chrome Browsers and the symptoms were similar. E.g. according to the RTC-Debug-Panel the voice data has been received by the affected users.

I will try out the code provided and hopefully report back soon. Thank you for investigating this issue and providing detailed information. 👍

[Charlesc22]

I encounter this problem too. Can not get audio in Chrome when my VPN is on, but works fine with VPN on in Firefox. If I turn off my VPN, then Audio works fine in Chrome.

[Dayk0]

Hello, after doing extensive research with a technical team regarding a problem similar to yours. After many changes to VPN, networks, etc. we came to the conclusion that the problem came from the browser (and more precisely Chrome and Edge). Despite all permissions being done, we still had issues and this was caused (on Edge Chromium) by a setting that blocked WebRTC. The "WebRtcLocalHostIpHandling" setting was at fault and after removing it on the browser, this fixed the issue.

[william-vz]

hi @Dayk0 - thanks for your response. I believe I saw your recommendation in another discussion but I was unable to find the "WebRtcLocalHostIpHandling" setting you mentioned. Would you be able to take a screenshot for me? Additionally, would that need to be configured on each client (aka every user that will use Hubs)? If so, that may not be a feasible solution for us.

[markusTraber]

Hey everyone, unfortunately, the suggested solution by @william-vz did not work for us. We still face audio issues sometimes, even without using a VPN, and I still could not figure out what the cause is. I should mention that I only have little knowledge regarding WebRTC...

As for the flag, mentioned by @Dayk0: I also was not able to find that flag on Chrome. I can find it on Edge, but no value has been set. I am on a M1 Mac Pro. Maybe this is something being set while deployment?