Use MediaStream as an alternative to bindVideoElement

[dathacky]

I have been using chime SDK for almost 1 year, I have a lot of problems using bindVideoElement(). In ReactJS, every time we render we always have to call the bindVideoElement() function again. This happens a lot because our UI changes frequently (grid layout, full screen, whiteboard,...), causing us to get a lot of video related errors if the UI renders but doesn't call bindVideoElement, so I have some suggestions below:

• Observer will receive MediaStream without having to call the bindVideoElement() function, after having MediaStream, we can process and pass it to <//video> tag ourselves.
• Can send to a MediaStream (camera + share screen) instead of calling functions (listAudioInputDevices, videoInputDevices, chooseVideoInputDevice)
• I noticed that the promises of chooseAudioInputDevice and videoTileDidUpdate are not in sync. The chooseAudioInputDevice function returns a promise but videoTileDidUpdate triggers a long time later, causing our button On Camera state to not actually display correctly.

• Please vote for this issue by adding a 👍 reaction to the issue. This will help the community and maintainers from the Amazon Chime SDK team to prioritize this request.
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request.

[ltrung]

Observer will receive MediaStream without having to call the bindVideoElement() function, after having MediaStream, we can process and pass it to

Sorry pass it to what?

Can send to a MediaStream (camera + share screen) instead of calling functions (listAudioInputDevices, videoInputDevices, chooseVideoInputDevice)

I am not sure I follow this. Could you clarify? listVideoInputDevices and chooseVideoInputDevice are needed to get the stream from browsers.

I noticed that the promises of chooseAudioInputDevice and videoTileDidUpdate are not in sync. The chooseAudioInputDevice function returns a promise but videoTileDidUpdate triggers a long time later, causing our button On Camera state to not actually display correctly.

When there is a change in audio/video stream, we need to update the stream subscription in the backend and thus cause some delay to videoTileDidUpdate. It is something we could try to improve in the future. However, can you workaround by waiting for videoTileDidUpdate before update the camera state?

[dathacky]

Sorry pass it to what?

Sorry about that, i mean: pass it to <//video> tag.

[dathacky]

I am not sure I follow this. Could you clarify? listVideoInputDevices and chooseVideoInputDevice are needed to get the stream from browsers.

Similar to startContentShare but for camera, so I can use getUserMedia to get stream and send instead of listVideoInputDevices and chooseVideoInputDevice

https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia

[ltrung]

@dathacky listVideoInputDevices also just get the stream from getUserMedia. Could you provide more details on why you want to call it manually yourself?

[dathacky]

@ltrung I want to work with MediaStream completely like any other SDK, it makes it easier for me to develop and find bugs. I will also easily customize the stream by adding effects to change devices, filters, backgrounds.

[ltrung]

@dathacky Currently, there is no API to access the video stream directly but you could access via meetingSession.audioVideo.deviceController.activeDevices['video'].stream.

[dathacky]

Thanks, I hope in the future chime SDK can access video streams including remote

[ltrung]

@dathacky You can actually access the remote stream too from tileState.boundVideoStream in videoTileDidUpdate, you can just skip binding video element method.

[evangeline]

You can actually access the remote stream too from tileState.boundVideoStream in videoTileDidUpdate, you can just skip binding video element method.

Hi @ltrung, I'm using boundVideoStream as described above. Is it possible to also get the remote audio stream instead of binding an audio element?