implement getDisplayMedia

[hthetiot]

Capture

navigator.mediaDevices.getDisplayMedia test result of prototype receiving screen capture from ios to chrome:

Testing

To test task/getDisplayMedia branch

cordova plugin remove cordova-plugin-iosrtc --verbose
cordova plugin add https://github.com/cordova-rtc/cordova-plugin-iosrtc#task/getDisplayMedia --verbose
cordova platform remove ios --no-save
cordova platform add ios --no-save

[hthetiot]

That was easy LOL

[hthetiot]

Need more tester feedback before release.

[aleccheah]

i have pull this branch to test getDisplayMedia. And have encountered error that was fix in 6.0.9. Unhandled Promise Rejection: TypeError: null is not an object (evaluating 'iceCandidateFields.foundation') from Safari Console while running under Xcode 11 Simulator

[hthetiot]

I will rebase on master right away @aleccheah thank you for your testing.

[hthetiot]

Just rebased on master @aleccheah if you can test again, Thank you again this help a lot.

[aleccheah]

@hthetiot thank you for the new rebased, the error is no longer exists :)

[hthetiot]

You think is ready for release @aleccheah ?

[aleccheah]

@hthetiot Sorry for the delayed response. I have encountered some issue with the following codes in the PluginRTCScreenCaptureController.swift that was implemented in 7b2da7a During the compilation from xcode deploying to iPad, please find the following error prompted.

Argument passed to call that takes no arguments at line 98 in PluginGetUserMedia.swift Value of type 'PluginRTCVideoCaptureController' has no member 'setConstraints' at line 101 in PluginGetUserMedia.swift

The above errors are referred to missing implementation in PluginRTCVideoCaptureController.swift. These errors does not appears when execute in Xcode simulator. The initiation of the code is from PluginGetUserMedia.swift where trying to create videoCaptureController object.

[aleccheah]

@hthetiot do you need any additional information with regards to the error that was encountered during the deployment of the app to iPad Devices?

[aleccheah]

@hthetiot I have done some testing on the getDisplayMedia implementation, and I am able to share my iPad Mobile Application Screen to a Browser. My question is how do i share the device screen or a specific mobile application in this case? In Desktop Safari, the getDisplayMedia automatically choose the entire screen where the browser is. In iOS mobile application it is actually capturing mobile application that invoke the getDisplayMedia. Advice is highly appreciated

[hthetiot]

@hthetiot I have done some testing on the getDisplayMedia implementation, and I am able to share my iPad Mobile Application Screen to a Browser. My question is how do i share the device screen or a specific mobile application in this case? In Desktop Safari, the getDisplayMedia automatically choose the entire screen where the browser is. In iOS mobile application it is actually capturing mobile application that invoke the getDisplayMedia. Advice is highly appreciated

I'm not sure it's possible to share another application or the whole screen. I would need more insight on what API to use.

[hthetiot]

May be some insight here:

• https://stackoverflow.com/questions/54480680/use-replaykit-to-record-any-app-on-screen
• https://developer.apple.com/documentation/replaykit/rpscreenrecorder

"Your app can record the audio and video inside of the app, along with user commentary through the microphone. "

[aleccheah]

• https://developer.apple.com/documentation/replaykit/rpscreenrecorder

Yeah... once I notice my test result did some googling as well on this... found out another approach is to use the ReplayKit Broadcast Extension to capture device screen instead of in-app screen

[hthetiot]

I will look into it when i have a moment.

[aleccheah]

@hthetiot sir, not sure if it is the behaviour of iOS itself on something else, notice on the first called of getDisplayMedia, the request actually rejected by iOS. On subsequent request called, I am able to get the stream. If it is my code issue, advice how to properly handle the invocation of method getDisplayMedia

[rex-iotum]

Can confirm what aleccheah experienced, the first call of getDisplayMedia returns the following MediaStreamError. getDisplayMedia() failed: Screen recorder is not available!

[hthetiot]

Can confirm what aleccheah experienced, the first call of getDisplayMedia returns the following MediaStreamError. getDisplayMedia() failed: Screen recorder is not available!

I have not experienced that for now.

[hthetiot]

See captures on first call to navigator.mediaDevices.getDisplayMedia({video: true}).

[hthetiot]

From the docs (https://developer.apple.com/documentation/replaykit/rpscreenrecorder):

Apps on a user’s device can share the recording function, with each app having its own instance of RPScreenRecorder. Your app can record the audio and video inside of the app, along with user commentary through the microphone

The only other way to record the screen is through a Broadcast Upload Extension, which requires the user to initiate it through Control Centre (https://github.com/StraaS/StraaS-iOS-sdk/wiki/Live-Streaming-with-Apple%27s-ReplayKit).

source: https://stackoverflow.com/questions/54480680/use-replaykit-to-record-any-app-on-screen

[wnabil]

@hthetiot tested and working fine i merged it in a fork, But after closing the streaming the app orientation seems like locked and can't be switched anymore until you restart the app

[wnabil]

ref https://github.com/wnabil/cordova-plugin-iosrtc

[hthetiot]

@wnabil thank you for testing, the current limitations is you can only capture the current application.

[samgabriel]

@hthetiot hi there long time no talk. hope u r doing well. Any idea whn this might be released. It looks great

[hthetiot]

@hthetiot hi there long time no talk. hope u r doing well. Any idea whn this might be released. It looks great

Hello there hope you doing well.

This is limited to sharing the current application screen not the full screen with app in the background.

I can may be debase on master and if it works i can release after #687

[samgabriel]

@hthetiot I think that is a good start. And once these classes in place may be we will help with the ReplayKit Broadcast extension

[igormartsekha]

Hi @hthetiot We did smth similar using our plugin https://github.com/Qbix/cordova-plugin-sharescreen We have used ReplayKit framework and can share screen outside of the app. The idea is to make local webrtc connection between main app and extension. We provide onStream callback where user can receive video stream and re-route them in any ways.

[hthetiot]

@samgabriel @igormartsekha I merged master (M87) feel free to make PR on the same branch.

[lgtm-com[bot]]

This pull request introduces 2 alerts when merging 7697788ba9a602f2468b877e52f37a35f6283400 into c4f02170c75753c65f5e5ad2f37df35d5cbbdecd - view on LGTM.com

new alerts:

• 2 for Duplicate property

[hthetiot]

PluginGetUserMedia.swift:126:59: error: value of type 'PluginRTCVideoCaptureController' has no member 'device'

Via https://api.travis-ci.com/v3/job/521179554/log.txt

Need to fix error due merge master. Will do asap, AFK.

[hthetiot]

Note that last build fail cause travis balance on ios.