TwilioError: Client is unable to apply a remote media description.

[PavloShchur]

Problem

We receive an error while trying to publish a local video track. Code responsible for creation and publication of a local video track:

...
Twilio.Video.connect.createLocalVideoTrack({
    name : 'Camera'
 });

twilioRoom.localParticipant.publishTrack(track, {priority : 'low'})
...

Error:

Steps to reproduce

1. Publish a screen sharing track;
2. Publish a local video track.

Investigation

Before turning on camera (see code snippet above) we publish a screen sharing track but after passing screen sharing video track to new Twilio.Video.LocalVideoTrack() it loses its dimensions. It seems to us that it is a reason for the error mentioned above.

...
const MediaStream = _canvas.captureStream();
const videoTrack = MediaStream.getTracks()[0];
const screenTrack = new Twilio.Video.LocalVideoTrack(videoTrack, {name: 'Sharing'});
...

videoTrack (height : 429, width : 1919): CanvasCaptureMediaStreamTrack {canvas: canvas#sharingScreen, kind: 'video', id: '6ec5f206-2988-43db-b478-fd33c71a7d27', label: 'OoAJrAhictfVfOcup9P17wC69OXlonOwA3AIaGbClVJtQuUDRnxDsndVkYqcqKY+MoehaQPbFDHUN+fiUtdNpg==', enabled: true, …} canvas: canvas#sharingScreen accessKey: "" ariaAtomic: null ariaAutoComplete: null ariaBusy: null ariaChecked: null ariaColCount: null ariaColIndex: null ariaColSpan: null ariaCurrent: null ariaDescription: null ariaDisabled: null ariaExpanded: null ariaHasPopup: null ariaHidden: null ariaInvalid: null ariaKeyShortcuts: null ariaLabel: null ariaLevel: null ariaLive: null ariaModal: null ariaMultiLine: null ariaMultiSelectable: null ariaOrientation: null ariaPlaceholder: null ariaPosInSet: null ariaPressed: null ariaReadOnly: null ariaRelevant: null ariaRequired: null ariaRoleDescription: null ariaRowCount: null ariaRowIndex: null ariaRowSpan: null ariaSelected: null ariaSetSize: null ariaSort: null ariaValueMax: null ariaValueMin: null ariaValueNow: null ariaValueText: null assignedSlot: null attributeStyleMap: StylePropertyMap {size: 0} attributes: NamedNodeMap {0: id, 1: width, 2: height, id: id, width: width, height: height, length: 3} autocapitalize: "" autofocus: false baseURI: "https://ocedev3--pshchur--c.visualforce.com/apex/RemotePlatformLaunchConference?id=a2I1s000000CKmcEAG" childElementCount: 0 childNodes: NodeList [] children: HTMLCollection [] classList: DOMTokenList [value: ''] className: "" clientHeight: 0 clientLeft: 0 clientTop: 0 clientWidth: 0 contentEditable: "inherit" dataset: DOMStringMap {} dir: "" draggable: false elementTiming: "" enterKeyHint: "" firstChild: null firstElementChild: null height: 429 hidden: false id: "sharingScreen" inert: false innerHTML: "" innerText: "" inputMode: "" isConnected: true isContentEditable: false lang: "" lastChild: null lastElementChild: null localName: "canvas" namespaceURI: "http://www.w3.org/1999/xhtml" nextElementSibling: video#sharingStream nextSibling: video#sharingStream nodeName: "CANVAS" nodeType: 1 nodeValue: null nonce: "" offsetHeight: 0 offsetLeft: 0 offsetParent: null offsetTop: 0 offsetWidth: 0 onabort: null onanimationend: null onanimationiteration: null onanimationstart: null onauxclick: null onbeforecopy: null onbeforecut: null onbeforematch: null onbeforepaste: null onbeforexrselect: null onblur: null oncancel: null oncanplay: null oncanplaythrough: null onchange: null onclick: null onclose: null oncontextlost: null oncontextmenu: null oncontextrestored: null oncopy: null oncuechange: null oncut: null ondblclick: null ondrag: null ondragend: null ondragenter: null ondragleave: null ondragover: null ondragstart: null ondrop: null ondurationchange: null onemptied: null onended: null onerror: null onfocus: null onformdata: null onfullscreenchange: null onfullscreenerror: null ongotpointercapture: null oninput: null oninvalid: null onkeydown: null onkeypress: null onkeyup: null onload: null onloadeddata: null onloadedmetadata: null onloadstart: null onlostpointercapture: null onmousedown: null onmouseenter: null onmouseleave: null onmousemove: null onmouseout: null onmouseover: null onmouseup: null onmousewheel: null onpaste: null onpause: null onplay: null onplaying: null onpointercancel: null onpointerdown: null onpointerenter: null onpointerleave: null onpointermove: null onpointerout: null onpointerover: null onpointerrawupdate: null onpointerup: null onprogress: null onratechange: null onreset: null onresize: null onscroll: null onsearch: null onsecuritypolicyviolation: null onseeked: null onseeking: null onselect: null onselectionchange: null onselectstart: null onslotchange: null onstalled: null onsubmit: null onsuspend: null ontimeupdate: null ontoggle: null ontransitioncancel: null ontransitionend: null ontransitionrun: null ontransitionstart: null onvolumechange: null onwaiting: null onwebkitanimationend: null onwebkitanimationiteration: null onwebkitanimationstart: null onwebkitfullscreenchange: null onwebkitfullscreenerror: null onwebkittransitionend: null onwheel: null outerHTML: "<canvas id=\"sharingScreen\" width=\"1919\" height=\"429\"></canvas>" outerText: "" ownerDocument: document parentElement: div#sharingContainer.slds-m-right_x-small parentNode: div#sharingContainer.slds-m-right_x-small part: DOMTokenList [value: ''] prefix: null previousElementSibling: null previousSibling: null role: null scrollHeight: 0 scrollLeft: 0 scrollTop: 0 scrollWidth: 0 shadowRoot: null slot: "" spellcheck: true style: CSSStyleDeclaration {accentColor: '', additiveSymbols: '', alignContent: '', alignItems: '', alignSelf: '', …} tabIndex: -1 tagName: "CANVAS" textContent: "" title: "" translate: true virtualKeyboardPolicy: "" width: 1919 [[Prototype]]: HTMLCanvasElement contentHint: "" enabled: true id: "6ec5f206-2988-43db-b478-fd33c71a7d27" kind: "video" label: "OoAJrAhictfVfOcup9P17wC69OXlonOwA3AIaGbClVJtQuUDRnxDsndVkYqcqKY+MoehaQPbFDHUN+fiUtdNpg==" muted: true oncapturehandlechange: null onended: null onmute: null onunmute: null readyState: "live" [[Prototype]]: CanvasCaptureMediaStreamTrack

screenTrack (height : null, width : null): LocalVideoTrack {kind: 'video', name: 'Sharing', processedTrack: null, …} dimensions: height: null width: null [[Prototype]]: Object id: "6ec5f206-2988-43db-b478-fd33c71a7d27" isEnabled: (...) isStarted: (...) isStopped: (...) kind: "video" mediaStreamTrack: (...) name: "Sharing" processedTrack: null processor: null _MediaStream: ƒ MediaStream() _attachments: Set(0) {size: 0} _captureTimeoutId: null _constraints: {} _didCallEnd: false _dummyEl: video _elShims: WeakMap {} _events: {disabled: Array(2), enabled: Array(2), dimensionsChanged: ƒ, message: ƒ, started: ƒ, …} _eventsCount: 6 _gUMSilentTrackWorkaround: ƒ workaround(log,getUserMedia,constraints,n,timeout) _getUserMedia: ƒ getUserMedia(constraints) _inputFrame: null _instanceId: 14 _isCapturing: false _isCreatedByCreateLocalTracks: false _isStarted: (...) _log: Log {_component: LocalVideoTrack, _logLevels: {…}, _warnings: Set(0), …} _maxListeners: undefined _outputFrame: null _playPausedElementsIfNotBackgrounded: false _processorEventObserver: VideoProcessorEventObserver {_events: {…}, _eventsCount: 6, _maxListeners: undefined, _lastStatsSaveTime: null, _lastStatsPublishTime: null, …} _shouldShimAttachedElements: false _trackSender: MediaTrackSender {_events: {…}, _eventsCount: 0, _maxListeners: undefined, id: '6ec5f206-2988-43db-b478-fd33c71a7d27', kind: 'video', …} _unmuteHandler: null _unprocessedTrack: null _workaroundSilentLocalVideo: null _workaroundSilentLocalVideoCleanup: null _workaroundWebKitBug1208516: false _workaroundWebKitBug1208516Cleanup: null get isEnabled: ƒ () get isStarted: ƒ () get isStopped: ƒ () get mediaStreamTrack: ƒ () get _isStarted: ƒ () set _isStarted: ƒ (_isStarted) [[Prototype]]: LocalVideoTrack

Technical details

• Room sid: RM1d36866d33409e3de1985a53ac128fc1, RMe18c254e86d5166be38257b512b99e1b;
• Room type: group;
• JS Version: 2.24.0;
• Browser: Chrome, 105.0.5195.102 (Official Build) (64-bit);
• Internet download/upload speed: 55.83/55.94Mbps.

Also, we have used Portable Apps to test on different Chrome versions. As a result, we could say that the issue starts to reproduce after Chrome, 103.0.5028.0. On Chrome, 102.0.5005.115 everything works fine. Here is a Chrome change log.

The issue could not be reproduced on Firefox, 103.0.2.

Workaround

The problem ("Client is unable to apply a remote media description.") disappears when preferredVideoCodecs is changed from auto to [{ codec : 'VP8', simulcast : false }] in connect options. The issue with losing screen sharing track's dimensions after passing it to new Twilio.Video.LocalVideoTrack() still remains.

We cannot change preferredVideoCodecs to [{ codec : 'VP8', simulcast : false }] because it decreases video quality for other participants.

Questions

Could you suggest something? Are there any known issues regarding the problem?

[manjeshbhargav]

@PavloShchur ,

Thanks for writing in. The Room SID you share with us has too many Participants, so its hard to debug. Can you provide a Room SID with 2 Participants where you see this issue? Also, can you share the reproduction steps?

[PavloShchur]

@manjeshbhargav,

Thanks for the quick replay.

The problem could be reproduced with one participant. Room SID: RMe18c254e86d5166be38257b512b99e1b.

Steps to reproduce:

1. Publish a screen sharing track;
2. Publish a local video track.

Also, we have noticed that new Twilio JS library (2.24.0) was released. We are using it now and the problem still could be reproduced.

[cindyloo]

I have also found that publishing a track causes this error reliably in Windows 10, Chrome

[manjeshbhargav]

@PavloShchur , @cindyloo ,

Thanks for the clarification. I will create an internal ticket and begin the investigation. Please watch this space for updates.

[manjeshbhargav]

@PavloShchur , @cindyloo ,

Our team could not reproduce the issue with the steps you provided. Is it possible that we can have access to a version of your app in a staging/test environment. That would help us in investigating further.

[cindyloo]

absolutely- please hit https://bongo-upgrade-testing.herokuapp.com you can use any email/pwd to authenticate click "room" enter a name and room name (can be anuything) click "continue" click "join" in Windows 10, using the chrome debugger, command-P to open AnonymousVideoTrack.tsx change line 231 from if (supportsCPublishing) to if (true) in AnonymousVideoTrack.tsx,

              (participant as LocalParticipant)
                .publishTrack(track)
                .then(localTrackPublication => {
                  console.log(
                    `Track overlay was published with SID ${localTrackPublication.trackSid} for ${participant.sid} ${track.name}`
                  );

                  return track;
                });
            }

you should receive the error when it tries to publish the canvas track (or any video track for that matter)

[manjeshbhargav]

Hi @cindyloo ,

I tried your app, but the regular anonymous camera track is always switched off for some reason. So, I could not get to the stage where I could try your changes. Can you deploy and share a version of the app where you do not add any video processor to the local video track, and always publish the second canvas track? That way, it will be easier for me to debug.

[cindyloo]

sure! I will do that right now -it will build in about 10 min On Monday, August 29, 2022 at 07:43:15 PM EDT, Manjesh Malavalli @.***> wrote:

Hi @cindyloo ,

I tried your app, but the regular anonymous camera track is always switched off for some reason. So, I could not get to the stage where I could try your changes. Can you deploy and share a version of the app where you do not add any video processor to the local video track, and always publish the second canvas track? That way, it will be easier for me to debug.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[cindyloo]

ok, I rolled back to an earlier version. what you will need to do is to begin the session as Participant A on the Windows/Chrome browser and send yourself an email at the PreJoinScreen/RoomName screen. Click Continue. Click join then as Participant 2, click the link and follow the prompts similarly. You will see the error in Participant A's browser

[PavloShchur]

@cindyloo, by using your app could we reproduce the issue described in the description: client is unable to apply a remote media description?

Also, could you deploy to some remote repository code used in your app and share it with us so we are sure that we are discussing the same problem?

[cindyloo]

@pavloshshur - I opened a very similar issue #1856 literally right before you - that explains and shows all the code. I receive the same error as you when publishing a videotrack. @manjeshbhargav is welcome to post to either issue - it's fine with me. You are also welcome to follow the steps above and you will also receive the same error.

[PavloShchur]

@manjeshbhargav We've built a demo app. Please let me know if there is anything what could be done from our side. https://github.com/PavloShchur/twilio-demo-app

[manjeshbhargav]

Thanks @PavloShchur , I'll play around with it today.

[manjeshbhargav]

@PavloShchur ,

I tried running your app, but it is asking for Twilio API credentials, which I am not comfortable entering on my own. Can you deploy a version of the app that does not ask for the credentials?

[PavloShchur]

@manjeshbhargav The app uses these credentials for authentication purposes only and does not store them. You could check it by yourself in the repository which we've shared with you.

Will it be more comfortable for you if we rebuild our app in such a way that you will need to enter just the access token?

[manjeshbhargav]

@PavloShchur ,

The best practice is for the application server to access the twilio credentials from the enviroment or a .env file in order to create access tokens. Sure, if the application can accept an access token as a URL parameter, that would work too.

[PavloShchur]

@manjeshbhargav, we have updated the repository. Now the app requires only access token.

[PavloShchur]

@manjeshbhargav Have you had a chance to reproduce the issue?

[manjeshbhargav]

Hi @PavloShchur ,

I think I have a potential workaround for you while we fix this in our SDK: https://github.com/PavloShchur/twilio-demo-app/pull/1. Please let me know if this works for you. I have filed a bug in Chrome which is the cause of this issue.

[PavloShchur]

Hello @manjeshbhargav,

The provided workaround removes the error (Client is unable to apply a remote media description) but the published track still does not have dimensions. You can test it by yourself: https://twilio-demo-app-workardound.herokuapp.com.

Also, we have tested the solution in our real app and the results are the same as in the demo app: error (Client is unable to apply a remote media description) disappeared but all other participants receive track without dimensions.

Are you sure that provided workaround will allow the logic to preserve the track's dimensions?

[manjeshbhargav]

Hi @PavloShchur ,

I've filed a Chromium bug for the canvas behavior. In your app, the canvas track gets muted after it has been created, and because of that, there are no dimensions. I'm not sure what you need to do to prevent it from being muted. Maybe keep drawing to the canvas continuously?

[PavloShchur]

Hi @manjeshbhargav,

We are passing a track with dimension into new Twilio.Video.LocalVideoTrack() and after that, we lose them.

Could we expect the issue to be fixed with the next Twilio JS library version?

[manjeshbhargav]

HI @PavloShchur ,

As I mentioned here, it is a Chromium bug, so the fix needs to be made in the browser source. I will track this bug and reach out to you here if they make any progress.

[manjeshbhargav]

Hi @PavloShchur ,

I have released 2.24.2, which fixes the error you are seeing. I will close the issue, but keep track of the Chromium bug and reach out to you if any development happens.