Video not consistently working on some networks (not bandwidth related)

[dan-steno]

• [x] I have verified that the issue occurs with the latest twilio-video.js release and is not marked as a known issue in the CHANGELOG.md.
• [x] I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
• [x] I verified that the Quickstart application works in my environment.
• [x] I am not sharing any Personally Identifiable Information (PII) or sensitive account information (API keys, credentials, etc.) when reporting this issue.

Expected behavior: As long as a network has enough bandwidth, and WebRTC can work within that network (and there are no firewall issues that are blocking connections), then any supported device/browser from that network should be able to use twilio-video.

In other words, if Google Hangouts works, then twilio-video should work (let me know if this is a bad assumption).

Actual behavior:

We have a team member (and also a customer) who when they connect from their own network, their video will not work most of the time, but it doesn't appear to be a bandwidth issue. These same people are able to use Zoom in high def, and have at least 50 Mbps down and 20 Mbps up. Additionally, they are able to use Google Hangouts without issue, so there doesn't seem to be anything blocking WebRTC for them.

We first ran into this issue when using our application (which uses twilio-video), but in order to rule out some logic in our app that was causing this issue, we tested with the Twilio twilio-not-working.txt twilio-working.txt

quickstart one (the most recent one), and the issue seemed to also happen with that.

The same device when connected via a different network or 4G appears to connect fine. Different devices on this network (or our other customer's network who was running into this issue) don't seem to work either, so we can rule out the Operating Sytem or browser (in this case Chrome).

We were able to get a session that worked with quickstart app, and one session where it didn't work. Attached are the "twilio-working" logs and the "twilio-not-working" logs.

You can see in both cases, "Greg" was the one who was in fact publishing video, and both of those rooms were being recorded, but the recording only happened on the first one, and not the second one. I compared both of the logs, and the only difference I noticed was the order in which some of the events happened in the "not-working" one vs. the "working one". Specifically, you can see in the "working" one, right before it connected to the ICE servers, there was a much larger "incoming" payload that was logged vs. the "not working" one.

Software versions:

• [x] Browser(s): Chrome 83
• [x] Operating System: Mac OS
• [x] twilio-video.js: 2.4.0 (this is what is bundled with the quickstart app right now
• [x] Third-party libraries (e.g., Angular, React, etc.): (for this test we were able to replicate with the quick start app)

[kumail-raza]

@manjeshbhargav @philnash

The same issue we experienced, except the SDK version 2.0 stable. It was a very bad experienced and embarrassing situation while I was demo my product to new customer I had that situation where the client users have very good bandwidth network, they suddenly start dropping when 5th participant joins the room.

I am using region selection to be GLL, that is I believe the first user who join/created the room the region will be the closest one to him. Here I have RoomSID and the ParticipantSID who basically when join whole room was messed up and not recover till we concluded that we are not going to have demo more right now.

RoomSID: RMb168068ef3dfd84c2d4346974a342d8b ParticipantSID: PA1ccf6e8fafb56e2be808d25fadf7ae53

Can you investigate it in the logs at server side, do share me the region from which ROOM was connected, and the region of Participant. This is something important now to us to decide whether we will move to other providers, as we investigate the code at our end its very basic with no big sciences.

--

In my product, I am using NetworkQualityAPI for LocalParticipant

https://www.twilio.com/blog/announcing-programmable-video-network-quality-api

As stated in the above link: It is important to note that the Network Quality API evaluates the network quality and not the perceived quality. Hence a quality level of 5 does not necessarily mean that the audio and video quality are good because the perceived quality may be degraded for reasons other than the network (e.g. CPU starvation, misconfiguration, etc.) In general, when the Quality Level goes down, the perceived quality will also go down. Hence, the Network Quality API can be a powerful tool for diagnosing whether the network is the root cause of audio and video quality degradation.

Note: Even with the very good bandwidth internet, we received bad & very bad network status using Network Quality API.

[dan-steno]

Also, one update for this, for one of the people where this issue was happening, everything seems to work fine if they are in firefox. However, it seems to be the combination of that network and chrome where the issue happens (the same issue happens across multiple devices on that network with Chrome). Oddly, using that same computer/browser on a different network there is no issue... This would also seemingly rule out an autoplay issue, but I don't know if there is something in that algorithm which also takes into account the network you are on.. I did also try having this person start chrome with these options to further rule out AutoPlay:

open -a Google\ Chrome --args --autoplay-policy=no-user-gesture-required

but it didn't seem to matter

[manjeshbhargav]

Hi @andersonhail ,

I took a look at the Zendesk ticket you created regarding this issue, and it seems like your network can't reach STUN candidates (srflx and prflx) based on looking at your WebRTC test results and our Room logs. The long term fix is to change your network configuration to allow STUN candidates. In the meantime, you can force media through TURN servers by setting the following ConnectOptions flag: iceTransportPolicy: 'relay'.

Please let me know if this helps.

Thanks,

Manjesh Malavalli JSDK Team

[dan-steno]

Hi @manjeshbhargav - do you know why Firefox would be able to connect without a problem, but chrome would be having issues then?

These were the errors we were seeing:

Once we connected via Firefox, then chrome worked fine. It was as if firefox was able to punch through and establish a connection with the TURN server, which then allowed communication to happen back to Chrome.

What's odd is that using Chrome, there is no other issue with any other WebRTC system we have used (including Google Hangouts), so there seems to be something specific with Twilio's implementation that's causing this issue.

From some of the research I was doing - this person seemed to run into the same issue: I also found this: https://github.com/webrtc/samples/issues/1232#issuecomment-628953911, in that case, it was due to IPv6 not being bound to the server (if you look at the screenshot I sent, it does appear that chrome is trying to connect via IPv6).

I would also add that this user just has the vanilla Spectrum networking setup - and it is their newer setup they appear to be implementing which is defaulting their home networks to use IPv6. The other customer we had who ran into this exact same issue, was also using Chrome on a Spectrum home network.

We can also reliably reproduce this now.

Steps: 1.) On this network try connecting with Chrome - it won't work 2.) Connect on the same device on the same network with firefox, and then they will both start working

[manjeshbhargav]

Hi @andersonhail ,

There seems to be an open WebRTC GitHub Issue regarding this. I'm not sure why Firefox would work with IPv6 addresses but Chrome doesn't. I also wanted to let you know that Twilio as a platform does not support IPv6 addresses. I would highly recommend that you try to force media through TURN servers by setting the following ConnectOptions flag: iceTransportPolicy: 'relay'.

Thanks,

Manjesh Malavalli JSDK Team

[dan-steno]

Thanks @manjeshbhargav - I had tried that previously at the suggestion of Twilio support (from the Zendesk ticket), but it didn't seem to affect anything. I can try it again.

[manjeshbhargav]

Hi @andersonhail ,

Did you have any luck with the iceTransportPolicy: 'relay' ConnectOptions flag? I did some more digging, and it looks like Chrome bypasses the default DNS service and uses their own Async DNS service to resolve IP addresses. I was wondering if you could do some tests on Chrome by disabling this service using a command line flag --disable-async-dns, which will force Chrome to use the default DNS service. Please let me know what you find.

Thanks,

Manjesh Malavalli JSDK Team

[dan-steno]

Hi @manjeshbhargav - I tried both of those things:

To open chrome with those settings I did this: open -a Google\ Chrome --args --disable-async-dns

And also I set the iceTransportPolicy like this:

And this is what I'm still seeing in the webRTC internals:

This is the room ID I used to test: RM807a6e5a3f334bd928347d0d4f7b3f35

After eventually adding a 3rd participant from firefox, it started working.

[dan-steno]

Also, I tested on the network that was having the issues with Twilio, and it was able to connect to Google Hangouts without a problem. Let me know if you have any other thoughts.

[kumail-raza]

@manjeshbhargav

It seems the issue is with Twilio or the client specs, as I can see many issues reported on Twilio git here, and relate them with each most of the issues facing by Twilio users are same.

Can you please share the similar logs details for this RoomSid is RM9dd6c44bbd5a4a8857748de90b15b840.

Issue was conversation was going all fine and suddenly all the users start dropping, losing connection with Twilio, that is now little bit frustrating as its now a month having these issues. Being the user having very good internet connectivity with very high bandwidth. We have seen logs against them with bad or very bad network connection even users in same region US and connected to US media server.

Kindly look into the issue and set some direction, so the issues could be resolve whether its from your side or our side.

[CyranMS]

Any results here?

[mc4rol9]

Hello, any progress on this issue?

We are facing the same problem here. Some customers can´t connect to Twilio in Chrome, only in Firefox. Both from computer and mobile. Other platforms working well.

We have a ticket opened in Zendesk (#7441792). A lot of tests were done.

Thank you.

[manjeshbhargav]

Hi @mc4rol9 ,

Can you share some room sids and console logs, if possible? Also, what is error message that you are observing?

Thanks,

Manjesh

[mc4rol9]

Hello @manjeshbhargav.

There´s no error message captured. I´m still with the ticket opened with the Twilio Support team. Is it possible for you to have access to the ticket history? Because we ran a lot of tests, including with an application hosted by Twilio. It may be helpful to enlighten on this issue.

Thanks.

[doctorpangloss]

Can you reopen this issue? I am also experiencing it.

[MatanYemini]

We are also experiencing the same.

IPv6 handoff fails

[vipulSoni458]

We are also facing the same issue, please reopen this issue.