twilio-video.js icon indicating copy to clipboard operation
twilio-video.js copied to clipboard

Published 20 hours ago • verified publisher icon twilio

Workaround from iOS 13 is breaking iOS 14+. Requesting ability to disable or remove `playPausedElementsIfNotBackgrounded`

Open marymhart opened this issue 3 years ago • 9 comments

When receiving a call on safari with iOS 14+ while on a video chat, the video freezes entirely & the audio never returns. When I provide workaroundWebKitBug1208516: false in the track constraints the video resolves as normal but the audio stays muted. I have discovered that it is due to this function https://github.com/twilio/twilio-video.js/blob/a5dcc8dc0e0aa6ecb9ca9fb8b7f8be9ab141566c/lib/media/track/mediatrack.js#L306 that I believe was implemented to fix an issue with iOS 13. It works find when I disable the function playIfPausedAndNotBackgrounded entirely so it would be nice to be able to pass this in as an option with the audio & video constraints. The reason for the failure is because when you receive a phone call on iOS 14. the document.visibilityState is still visible so playing the paused element while in this state causes a (silent) failure. Other track events are not fired when this happens so it is not possible to resolve otherwise that I have found. We originally reported this issue here https://github.com/twilio/twilio-video.js/issues/1320

  • [X] I have verified that the issue occurs with the latest twilio-video.js release and is not marked as a known issue in the CHANGELOG.md.
  • [X] I reviewed the Common Issues and open GitHub issues and verified that this report represents a potentially new issue.
  • [X] I verified that the Quickstart application works in my environment.
  • [X] I am not sharing any Personally Identifiable Information (PII) or sensitive account information (API keys, credentials, etc.) when reporting this issue.

Expected behavior:

Audio & video should return after receiving or declining call

Actual behavior:

Start a video on desktop & join as a user on mobile Safari on iOS 14+ with video enabled. Receive an incoming phone call & either decline from the notification screen or hangup on the caller side Note that video freezes OR if you provide workaroundWebKitBug1208516: false in the constraints the video resolves as normal but the audio stays muted. The logs provided are from before the call was actually declined so you can see it attempts to play the video before the audio is unmuted.

logs where it breaks:

[Warning] 2021-09-20 – "17:33:12.342Z" – "|" – "WARN" – "in" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]:" – "Unexpected state \"closed\" for handling a \"heartbeat\" message from the TCMP server." (4.chunk.js, line 212464)
[Warning] 2021-09-20 – "17:33:12.347Z" – "|" – "WARN" – "in" – "[LocalVideoTrack #2: 95e239d3-1e25-40d6-a89c-d494c36e1bc9]:" – "Unintentionally paused:" –  (4.chunk.js, line 212464)
<video id="rhinovideo__participant-stream__video__PA8b9f72262b779888ba0411043b1da234" class="rhinovideo__participant-stream " autoplay playsinline></video>

<video id="rhinovideo__participant-stream__video__PA8b9f72262b779888ba0411043b1da234" class="rhinovideo__participant-stream " autoplay playsinline></video>
[Log] 2021-09-20 – "17:33:12.356Z" – "|" – "DEBUG" – "in" – "[PeerConnectionV2 #1: a33cb8e7-cd6f-4d4d-a85a-d33db2702262]:" – "ICE connection state is \"closed\"" (4.chunk.js, line 212464)
[Info] 2021-09-20 – "17:33:13.348Z" – "|" – "INFO" – "in" – "[LocalVideoTrack #2: 95e239d3-1e25-40d6-a89c-d494c36e1bc9]:" – "Playing unintentionally paused <video> element" (4.chunk.js, line 212464)
[Log] 2021-09-20 – "17:33:13.348Z" – "|" – "DEBUG" – "in" – "[LocalVideoTrack #2: 95e239d3-1e25-40d6-a89c-d494c36e1bc9]:" – "Element:" –  (4.chunk.js, line 212464)
<video id="rhinovideo__participant-stream__video__PA8b9f72262b779888ba0411043b1da234" class="rhinovideo__participant-stream " autoplay playsinline></video>

<video id="rhinovideo__participant-stream__video__PA8b9f72262b779888ba0411043b1da234" class="rhinovideo__participant-stream " autoplay playsinline></video>
[Info] 2021-09-20 – "17:33:13.367Z" – "|" – "INFO" – "in" – "[LocalVideoTrack #2: 95e239d3-1e25-40d6-a89c-d494c36e1bc9]:" – "Successfully played unintentionally paused <video> element" (4.chunk.js, line 212464)
[Log] 2021-09-20 – "17:33:13.368Z" – "|" – "DEBUG" – "in" – "[LocalVideoTrack #2: 95e239d3-1e25-40d6-a89c-d494c36e1bc9]:" – "Element:" –  (4.chunk.js, line 212464)
<video id="rhinovideo__participant-stream__video__PA8b9f72262b779888ba0411043b1da234" class="rhinovideo__participant-stream " autoplay playsinline></video>

<video id="rhinovideo__participant-stream__video__PA8b9f72262b779888ba0411043b1da234" class="rhinovideo__participant-stream " autoplay playsinline></video>

Software versions:

  • [x] Browser(s): iOS 14.8 safari
  • [x] twilio-video.js: 2.7.3 & above

marymhart avatar Sep 20 '21 18:09 marymhart

Hello @marymhart - Thank you for reporting this issue. Looks like we need to revisit the the workarounds introduced for iOS Safari. I have filed an internal ticket to address this. (VIDEO-7011). Will keep you posted on updates on it.

Thanks, Makarand

makarandp0 avatar Sep 20 '21 18:09 makarandp0

Hello @marymhart,

We looked into this and were not able to reproduce the issue. We see that Safari workarounds, continue to help when a video call in interrupted by an audio call.

Regarding you repro, You mentioned iOS 14.8, but looking at logs from the room (RMa3f1fa4549c942364fd884e2e861fab1) I noticed that all the clients were connected from Mobile Safari 14.1.2. Can you confirm the iOS version that causes this repro?

Thanks, Makarand

makarandp0 avatar Sep 22 '21 19:09 makarandp0

@makarandp0 I have iOS 14.8 but it corresponds to 14.1.2 which was included in the 14.8 update.

This issue happens consistently for us with the demo project & our production app. I am including some more detailed steps to recreate below.

  1. The user on mobile safari has video enabled & visible on their own screen
  2. The user on mobile safari does not have do not disturb on & has their device sound turned on
  3. Phone call notifications on the iOS device show banners. Make sure to dismiss the call from the banner.

Let me know if you are still unable to reproduce. I am planning to test with 15 tomorrow. Thanks!

marymhart avatar Sep 23 '21 20:09 marymhart

Hello @marymhart, thank you for the update. We did test with the same configuration and have not been able to repro - Would it be possible for you to share link to the app that repros this? You mentioned demo project - was that https://github.com/twilio/video-quickstart-js ?

makarandp0 avatar Sep 23 '21 21:09 makarandp0

@makarandp0 I originally used the react package but I just reproduced it with the most recent version of https://github.com/twilio/video-quickstart-js. In every instance it works the first time I decline a call but I lose audio the second time every time. In the instances where it fails, the Failed to detect silence occurs immediately after the phone starts ringing (prior to declining) followed by the video returning & the Successfully played unintentionally paused <video> element message which is the same issue experienced with our own build. The attached logs show the successful first call & the second where the audio failed.

First call:

[Log] 2021-09-24T12:25:20.979Z – "debug" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]" – "Incoming: {\"type\":\"heartbeat\"}" (index.js, line 25725)
[Log] 2021-09-24T12:25:22.630Z – "debug" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]" – "Outgoing: {\"type\":\"heartbeat\"}" (index.js, line 25725)
[Log] 2021-09-24T12:25:24.981Z – "debug" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]" – "Incoming: {\"type\":\"heartbeat\"}" (index.js, line 25725)
[Info] 2021-09-24T12:25:27.071Z – "info" – "[connect #1]" – "event" – {elapsedTime: 10866, group: "quality", level: "info", …} (index.js, line 25725)
{elapsedTime: 10866, group: "quality", level: "info", name: "stats-report", timestamp: 1632486327070, …}Object
[Info] 2021-09-24T12:25:27.072Z – "info" – "[connect #1]" – "event" – {elapsedTime: 10868, group: "quality", level: "info", …} (index.js, line 25725)
{elapsedTime: 10868, group: "quality", level: "info", name: "active-ice-candidate-pair", timestamp: 1632486327072, …}Object
[Log] 2021-09-24T12:25:27.708Z – "debug" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]" – "Outgoing: {\"type\":\"heartbeat\"}" (index.js, line 25725)
[Warning] 2021-09-24T12:25:28.165Z – "warn" – "[RemoteVideoTrack #6: MT5621971d00ef9a96a7979c5eb6cb4927]" – "Unintentionally paused:" –  (index.js, line 25725)
<video autoplay playsinline muted style>…</video>

<video autoplay playsinline muted style>…</video>
[Warning] 2021-09-24T12:25:28.167Z – "warn" – "[RemoteVideoTrack #6: MT5621971d00ef9a96a7979c5eb6cb4927]" – "Unintentionally paused:" –  (index.js, line 25725)
<video autoplay muted playsinline style>…</video>

<video autoplay muted playsinline style>…</video>
[Warning] 2021-09-24T12:25:28.168Z – "warn" – "[RemoteAudioTrack #5: MT62be2a22a40201de7cc2921c2e60de88]" – "Unintentionally paused:" –  (index.js, line 25725)
<audio autoplay style></audio>

<audio autoplay style></audio>
[Warning] 2021-09-24T12:25:28.172Z – "warn" – "[LocalVideoTrack #4: c096b5ac-bfce-4d6c-8f15-7748c3e41d1f]" – "Unintentionally paused:" –  (index.js, line 25725)
<video autoplay muted playsinline style>…</video>

<video autoplay muted playsinline style>…</video>
[Info] 2021-09-24T12:25:28.174Z – "info" – "[LocalAudioTrack #3: a113d13f-e38b-41a2-85a3-1fa76b4116ce]" – "Muted" (index.js, line 25725)
[Log] 2021-09-24T12:25:28.174Z – "debug" – "[LocalAudioTrack #3: a113d13f-e38b-41a2-85a3-1fa76b4116ce]" – "LocalMediaTrack:" – LocalAudioTrack {_events: Object, _maxListeners: undefined, _instanceId: 3, …} (index.js, line 25725)
LocalAudioTrack {_events: Object, _maxListeners: undefined, _instanceId: 3, _log: Log, kind: "audio", …}LocalAudioTrack
[Log] 2021-09-24T12:25:29.091Z – "debug" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]" – "Incoming: {\"type\":\"heartbeat\"}" (index.js, line 25725)
[Log] 2021-09-24T12:25:32.624Z – "debug" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]" – "Outgoing: {\"type\":\"heartbeat\"}" (index.js, line 25725)
[Log] 2021-09-24T12:25:33.076Z – "debug" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]" – "Incoming: {\"type\":\"heartbeat\"}" (index.js, line 25725)
[Info] 2021-09-24T12:25:37.070Z – "info" – "[connect #1]" – "event" – {elapsedTime: 20866, group: "quality", level: "info", …} (index.js, line 25725)
{elapsedTime: 20866, group: "quality", level: "info", name: "stats-report", timestamp: 1632486337070, …}Object
[Log] 2021-09-24T12:25:37.179Z – "debug" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]" – "Incoming: {\"type\":\"heartbeat\"}" (index.js, line 25725)
[Log] 2021-09-24T12:25:37.426Z – "debug" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]" – "Outgoing: {\"type\":\"heartbeat\"}" (index.js, line 25725)
[Log] 2021-09-24T12:25:41.277Z – "debug" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]" – "Incoming: {\"type\":\"heartbeat\"}" (index.js, line 25725)

Second Call:

[Log] 2021-09-24T12:25:42.422Z – "debug" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]" – "Outgoing: {\"type\":\"heartbeat\"}" (index.js, line 25725)
[Warning] 2021-09-24T12:25:42.895Z – "warn" – "[LocalAudioTrack #3: a113d13f-e38b-41a2-85a3-1fa76b4116ce]" – "Failed to detect silence:" – AbortError: The operation was aborted. (index.js, line 25725)
AbortError: The operation was aborted.
[Info] 2021-09-24T12:25:42.897Z – "info" – "[RemoteVideoTrack #6: MT5621971d00ef9a96a7979c5eb6cb4927]" – "Playing unintentionally paused <video> element" (index.js, line 25725)
[Log] 2021-09-24T12:25:42.897Z – "debug" – "[RemoteVideoTrack #6: MT5621971d00ef9a96a7979c5eb6cb4927]" – "Element:" –  (index.js, line 25725)
<video autoplay playsinline muted style>…</video>

<video autoplay playsinline muted style>…</video>
[Info] 2021-09-24T12:25:42.975Z – "info" – "[RemoteVideoTrack #6: MT5621971d00ef9a96a7979c5eb6cb4927]" – "Playing unintentionally paused <video> element" (index.js, line 25725)
[Log] 2021-09-24T12:25:42.977Z – "debug" – "[RemoteVideoTrack #6: MT5621971d00ef9a96a7979c5eb6cb4927]" – "Element:" –  (index.js, line 25725)
<video autoplay muted playsinline style>…</video>

<video autoplay muted playsinline style>…</video>
[Info] 2021-09-24T12:25:43.045Z – "info" – "[RemoteAudioTrack #5: MT62be2a22a40201de7cc2921c2e60de88]" – "Playing unintentionally paused <audio> element" (index.js, line 25725)
[Log] 2021-09-24T12:25:43.046Z – "debug" – "[RemoteAudioTrack #5: MT62be2a22a40201de7cc2921c2e60de88]" – "Element:" –  (index.js, line 25725)
<audio autoplay style></audio>

<audio autoplay style></audio>
[Info] 2021-09-24T12:25:43.110Z – "info" – "[LocalVideoTrack #4: c096b5ac-bfce-4d6c-8f15-7748c3e41d1f]" – "Playing unintentionally paused <video> element" (index.js, line 25725)
[Log] 2021-09-24T12:25:43.110Z – "debug" – "[LocalVideoTrack #4: c096b5ac-bfce-4d6c-8f15-7748c3e41d1f]" – "Element:" –  (index.js, line 25725)
<video autoplay muted playsinline style>…</video>

<video autoplay muted playsinline style>…</video>
[Warning] 2021-09-24T12:25:43.163Z – "warn" – "[LocalVideoTrack #4: c096b5ac-bfce-4d6c-8f15-7748c3e41d1f]" – "Unintentionally paused:" –  (index.js, line 25725)
<video autoplay muted playsinline style>…</video>

<video autoplay muted playsinline style>…</video>
[Warning] 2021-09-24T12:25:43.165Z – "warn" – "[RemoteAudioTrack #5: MT62be2a22a40201de7cc2921c2e60de88]" – "Unintentionally paused:" –  (index.js, line 25725)
<audio autoplay style></audio>

<audio autoplay style></audio>
[Warning] 2021-09-24T12:25:43.168Z – "warn" – "[RemoteVideoTrack #6: MT5621971d00ef9a96a7979c5eb6cb4927]" – "Unintentionally paused:" –  (index.js, line 25725)
<video autoplay muted playsinline style>…</video>

<video autoplay muted playsinline style>…</video>
[Warning] 2021-09-24T12:25:43.169Z – "warn" – "[RemoteVideoTrack #6: MT5621971d00ef9a96a7979c5eb6cb4927]" – "Unintentionally paused:" –  (index.js, line 25725)
<video autoplay playsinline muted style>…</video>

<video autoplay playsinline muted style>…</video>
[Info] 2021-09-24T12:25:43.179Z – "info" – "[LocalAudioTrack #3: a113d13f-e38b-41a2-85a3-1fa76b4116ce]" – "Muted" (index.js, line 25725)
[Log] 2021-09-24T12:25:43.179Z – "debug" – "[LocalAudioTrack #3: a113d13f-e38b-41a2-85a3-1fa76b4116ce]" – "LocalMediaTrack:" – LocalAudioTrack {_events: Object, _maxListeners: undefined, _instanceId: 3, …} (index.js, line 25725)
LocalAudioTrack {_events: Object, _maxListeners: undefined, _instanceId: 3, _log: Log, kind: "audio", …}LocalAudioTrack
[Info] 2021-09-24T12:25:43.187Z – "info" – "[RemoteVideoTrack #6: MT5621971d00ef9a96a7979c5eb6cb4927]" – "Successfully played unintentionally paused <video> element" (index.js, line 25725)
[Log] 2021-09-24T12:25:43.187Z – "debug" – "[RemoteVideoTrack #6: MT5621971d00ef9a96a7979c5eb6cb4927]" – "Element:" –  (index.js, line 25725)
<video autoplay playsinline muted style>…</video>

<video autoplay playsinline muted style>…</video>
[Info] 2021-09-24T12:25:43.189Z – "info" – "[RemoteVideoTrack #6: MT5621971d00ef9a96a7979c5eb6cb4927]" – "Successfully played unintentionally paused <video> element" (index.js, line 25725)
[Log] 2021-09-24T12:25:43.189Z – "debug" – "[RemoteVideoTrack #6: MT5621971d00ef9a96a7979c5eb6cb4927]" – "Element:" –  (index.js, line 25725)
<video autoplay muted playsinline style>…</video>

<video autoplay muted playsinline style>…</video>
[Info] 2021-09-24T12:25:43.189Z – "info" – "[RemoteAudioTrack #5: MT62be2a22a40201de7cc2921c2e60de88]" – "Successfully played unintentionally paused <audio> element" (index.js, line 25725)
[Log] 2021-09-24T12:25:43.190Z – "debug" – "[RemoteAudioTrack #5: MT62be2a22a40201de7cc2921c2e60de88]" – "Element:" –  (index.js, line 25725)
<audio autoplay style></audio>

<audio autoplay style></audio>
[Info] 2021-09-24T12:25:43.191Z – "info" – "[LocalVideoTrack #4: c096b5ac-bfce-4d6c-8f15-7748c3e41d1f]" – "Successfully played unintentionally paused <video> element" (index.js, line 25725)
[Log] 2021-09-24T12:25:43.191Z – "debug" – "[LocalVideoTrack #4: c096b5ac-bfce-4d6c-8f15-7748c3e41d1f]" – "Element:" –  (index.js, line 25725)
<video autoplay muted playsinline style>…</video>

<video autoplay muted playsinline style>…</video>
[Log] 2021-09-24T12:25:45.280Z – "debug" – "[TwilioConnection #1: wss://global.vss.twilio.com/signaling]" – "Incoming: {\"type\":\"heartbeat\"}" (index.js, line 25725)
[Info] 2021-09-24T12:25:47.074Z – "info" – "[connect #1]" – "event" – {elapsedTime: 30869, group: "quality", level: "info", …} (index.js, line 25725)
{elapsedTime: 30869, group: "quality", level: "info", name: "stats-report", timestamp: 1632486347073, …}Object
[Info] 2021-09-24T12:25:47.076Z – "info" – "[connect #1]" – "event" – {elapsedTime: 30872, group: "quality", level: "info", …} (index.js, line 25725)
{elapsedTime: 30872, group: "quality", level: "info", name: "active-ice-candidate-pair", timestamp: 1632486347076, …}Object

marymhart avatar Sep 24 '21 12:09 marymhart

Also, want to note that I tried the video-quickstart project with this version of the twilio-video package which just disables those workarounds on iOS 14.5+ & it works every time on iOS 14.8 but I have not tested with any other iOS versions. https://github.com/marymhart/twilio-video/releases/tag/rg.1.0.

marymhart avatar Sep 24 '21 12:09 marymhart

Hey @marymhart, sorry for late reply - We do see that some of the iOS workarounds are not working on newer iOS versions, and there is an open webkit bug for this.

However we found that removing workarounds does not help the situation either, but we are continuing our investigations. Thanks, Makarand

makarandp0 avatar Oct 19 '21 17:10 makarandp0

@makarandp0 if you could extend the AudioTrack and VideoTrack classes to emit mute and unmute events from the underlying MediaStreamTracks that would help a bunch- in our app, restart()-ing either of these Twilio classes and rerendering the component usually fixes the issues.

Of course, we have to listen to the parent Track.started event to attach the mute and unmute listeners to the MediaStreamTrack, so if you could abstract away that bit it would be much more convenient and appreciated.

gavinlefebvre avatar Oct 19 '21 18:10 gavinlefebvre

Thank you @gavinlefebvre for the suggestion, I have included your suggestion in internal ticket (VIDEO-7421) for improving restart api.

makarandp0 avatar Oct 19 '21 20:10 makarandp0