getDisplayMedia() has permission issue

[devtool20000]

Describe the bug

Hi I try to call getDisplayMedia() but got permission issue, see details below, thanks!

To Reproduce

Steps to reproduce the behavior:

1. Check out the sample code from here
2. cd to src-tauri/my-app run yarn && yarn start, this start the react dev server
3. in root folder run yarn tauri dev
4. After the application starts, click the "test" button, this will trigger the getDisplayMedia() api. Click the "Inspect Element", you will see the error in console "Unhandled Promise Rejection: NotAllowedError: The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission."

Expected behavior

It should call the getDisplayMedia() successfully without error.

Screenshots

Platform and Versions (required):

Operating System - Mac OS, version 11.5.2 X64

Node.js environment
  Node.js - 14.17.2
  @tauri-apps/cli - 1.0.0-beta.10
  @tauri-apps/api - 1.0.0-beta.8

Global packages
  npm - 6.14.13
  yarn - 1.22.4

Rust environment
  rustc - 1.55.0
  cargo - 1.55.0

App directory structure
/node_modules
/src-tauri
/.idea

App
  tauri.rs - 1.0.0-beta.8
  build-type - bundle
  CSP - default-src blob: data: filesystem: ws: wss: http: https: tauri: 'unsafe-eval' 'unsafe-inline' 'self' img-src: 'self'
  distDir - my-app/build
  devPath - http://localhost:3000

Additional context

I already add the Info.plist under src-tauri with the following configuration

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>NSCameraUsageDescription</key>
	<string>Request camera access for WebRTC</string>
	<key>NSMicrophoneUsageDescription</key>
	<string>Request microphone access for WebRTC</string>
</dict>
</plist>

The getUserMedia() works properly, but getDisplayMedia() didn't. I try to run the same react app inside safari and the getDisplayMedia() works properly in it.

Stack Trace

[alectrocute]

Side effect of #2317?

[cliqer]

I am experiencing the same error.

NotAllowedError: The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission.

and on the server I get:

+++     0x60000040ee60  0       1       914797062695125

My plist is:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>com.apple.security.cs.allow-jit</key>
    <true/>
    <key>com.apple.security.cs.allow-unsigned-executable-memory</key>
    <true/>
    <key>com.apple.security.cs.disable-library-validation</key>
    <true/>
    <key>com.apple.security.cs.allow-dyld-environment-variables</key>
    <true/>
    <key>com.apple.security.automation.apple-events</key>
    <true/>
    <key>com.apple.security.device.usb</key>
    <true/>
    <key>com.apple.security.device.camera</key>
    <true/>
    <key>com.apple.security.network.server</key>
    <true/>
    <key>com.apple.security.network.client</key>
    <true/>
    <key>com.apple.security.inherit</key>
    <true/>
  </dict>
</plist>

and the request is:

  return await navigator.mediaDevices.getDisplayMedia({
    audio: false,
    video: {
      //@ts-ignore
      mandatory: {
        displaySurface: 'monitor', // tried 'desktop', 'screen', 'display'
        chromeMediaSourceId: 1,
        minWidth: 1280,
        maxWidth: 1680,
        minHeight: 720,
        maxHeight: 1050
      }
    }
  }).then(stream => console.log(stream))
    .catch(e => console.log(e))

Also window.isSecureContext returns true on all platforms. Any solutions?

[cliqer]

navigator.mediaDevices.getDisplayMedia throw NotAllowedError: The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission.

Can someone please help with this problem for Webkit on MacOS?

[daniel-abramov]

@cliqer, I'm not an expert, so not sure if what I'm going to say would make any sense, but AFAIK there is no such thing as chromeMediaSourceId in the WebRTC spec. I assume you might be using the chromeMediaSourceId as this is something you would use with Electron when calling their getUserMedia function to capture a specific window.

It seems like this is very specific to the Chromium engine and Electron even implemented an extension/helper to handle the chromeMediaSourceId inside getUserMedia().

I'm not sure these could work with getDisplayMedia and Tauri without implementing some sort of a Tauri plugin that would have a functionality similar to the desktopCapturer in Electron.

[cliqer]

@application-developer-DA you are totally right.

WKWebView does not support getDisplayMedia and currently, there is no way to share a screen on MacOS.

There are a few solutions that could be a workaround but need implementation:

• A custom getUserMedia like Electron and Cordova did.
• The new ScreenCaptureKit framework: https://developer.apple.com/documentation/screencapturekit/capturing_screen_content_in_macos

This is a mandatory feature of our application as we need webrtc screen sharing and the only problem we have as to port it to Tauri/rust from electron is the luck of capturing the video stream of the screen.

Any good soul out there with the knowledge to port electron's implementation of getUserMedia or getDisplayMedia to WKWebView or can recommend any other solutions for Tauri/rust?

Any recommendations would be very much appreciated.

[daniel-abramov]

This is a mandatory feature of our application as we need webrtc screen sharing and the only problem we have as to port it to Tauri/rust from electron is the luck of capturing the video stream of the screen.

Yeah, right. We're in the same boat.

Any good soul out there with the knowledge to port electron's implementation of getUserMedia or getDisplayMedia to WKWebView or can recommend any other solutions for Tauri/rust? Any recommendations would be very much appreciated.

I think the only "viable" (not really unfortunately) option would be using webrtc-rs, but the client stack for the WebRTC has not yet been implemented and is part of a roadmap. There is an open issue to create an example to use it with Tauri.

I think basically you're left with 2 options here:

1. Rely upon Electron (probably not a desirable option if you're choosing Tauri for the same reason we do).
2. Implement some sort of a Tauri Plugin that would bring the desired support of the WebRTC Screen Capturing. This might take a decent amount of time before it matches the feature-set of Electron's desktopCapturer though (Disclaimer: I don't have much experience with Tauri plugins, so perhaps this whole idea is also far away from being optimal).

[towe75]

I'm not a tauri user but i came across the same problem and found this post.

After quite some digging i was able to get a acceptable solution for my WKWebView based project..

a) ensure to request screen capture permissions for the app itself via CGPreflightScreenCaptureAccess() / CGRequestScreenCaptureAccess()

b) it's necessary to implement a private delegate. I found this here: https://github.com/SafeExamBrowser/seb-mac/issues/150

Try this minimal example that i derived from my own project:

@protocol WKUIDelegatePrivateTauri <NSObject>

typedef NS_OPTIONS(NSUInteger, _WKCaptureDevices) {
    _WKCaptureDeviceMicrophone = 1 << 0,
    _WKCaptureDeviceCamera = 1 << 1,
    _WKCaptureDeviceDisplay = 1 << 2,
};

- (void)_webView:(WKWebView *)webView requestUserMediaAuthorizationForDevices:(_WKCaptureDevices)devices url:(NSURL *)url mainFrameURL:(NSURL *)mainFrameURL decisionHandler:(void (^)(BOOL authorized))decisionHandler;
@end

@interface TauriDelegate : NSObject <WKUIDelegatePrivateTauri>

@end

@implementation TauriDelegate

- (void)_webView:(WKWebView *)webView
      requestUserMediaAuthorizationForDevices:(_WKCaptureDevices)devices url:(NSURL *)url
      mainFrameURL:(NSURL *)mainFrameURL
      decisionHandler:(void (^)(BOOL authorized))decisionHandler
{
    decisionHandler(true);
}

@end

With this i was able to get a media stream. Although the chooser is still missing. It shares the screen with the browser view on it.

Maybe the situation improves with a later webkit version. Seems they work on it: https://github.com/WebKit/WebKit/pull/9789

Hope this helps you! Who knows, maybe i will use tauri for a future project ;-)

[veaba]

+1 Can't record audio.

[ZhoufangErqiangu]

i got same error on ubuntu 22.04. i am sure my code works well in browsers.

[FabianLars]

This issue is about macOS. On Linux, even though it's the same error message, the cause is different == webrtc just isn't a thing in webkitgtk yet (to be exact, it is being worked on but hidden behind a build flag only distro maintainers can flip)

[stolinski]

Has anyone managed to get this working? Just calling getDisplayMedia leads to the same errors @cliqer was having in 2022.

[cliqer]

@stolinski, this has been fixed in Webkit v16.1 on Ventura. This means that getDisplayMedia only works if your Tauri app runs on Ventura with the latest Safari.

[stolinski]

Wild I'm on Sonoma and hitting permissions issues. I need to find how to reset permissions in this app if they have been denied somehow.

[cliqer]

@stolinski try the following if you haven't. Hope it helps:

// src-tauri/info.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>NSCameraUsageDescription</key>
	<string>Request camera access for WebRTC</string>
	<key>NSMicrophoneUsageDescription</key>
	<string>Request microphone access for WebRTC</string>
</dict>
</plist>

//src-tauri/Release.entitlements
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>com.apple.security.cs.allow-jit</key>
    <true/>
    <key>com.apple.security.cs.allow-unsigned-executable-memory</key>
    <true/>
    <key>com.apple.security.cs.disable-library-validation</key>
    <true/>
    <key>com.apple.security.cs.allow-dyld-environment-variables</key>
    <true/>
    <key>com.apple.security.automation.apple-events</key>
    <true/>
    <key>com.apple.security.device.usb</key>
    <true/>
    <key>com.apple.security.device.camera</key>
    <true/>
    <key>com.apple.security.network.server</key>
    <true/>
    <key>com.apple.security.network.client</key>
    <true/>
    <key>com.apple.security.inherit</key>
    <true/>
  </dict>
</plist>

in your tauri.conf.json:

{
 {
 "tauri": {
  "macOSPrivateApi": true,
  "bundle": {
      "active": true,
      "macOS": {
        "entitlements": "Release.entitlements",
        "exceptionDomain": "",
        "frameworks": [],
      },
      "resources": [],
      "targets": ["app"],
      }
    },
    "security": {
      "csp": null
    }
  }
}

[stolinski]

Thanks for the help @cliqer but sadly no dice. Still hitting the same permissions error.

[stolinski]

So I can confirm that getDisplayMedia has been added to WKWebView it does not work within Tauri still because there is no permission dialog and therefore the request for getDisplayMedia will always return a permission denied error. So this seems to be an issue with Tauri as is right now as you are unable to grant permissions to use getDisplayMedia.

[cliqer]

@stolinski it works for me and dialogue is being displayed in the latest Tauri v1 and Ventura 13.5.1 (22G90): I am requesting it simply like this:

return await navigator.mediaDevices.getDisplayMedia({
            audio: false,
            video: true
})

[stolinski]

Is this in development? How frustrating. I'm on Sonoma, latest Tauri v1 and just getting the permissions failed instantly with the same getDisplayMedia code to request.

Do you know of a way to reset permissions completely on a Tauri app? I've tried building for production and got the same permission failure immediately.

[stolinski]

Just fired up a totally fresh Tauri app, requested

navigator.mediaDevices.getDisplayMedia({
            audio: false,
            video: true
})

And got hit with the same permission denied error without a dialogue. Now wondering if this is a Sonoma related issue. (on 14.0 Beta (23A5337a))

[stolinski]

Small update on this.

I can confirm that the permissions prompt is being issued on MacOS v13 but not on v14, which is still unreleased although in rc. I'll update again once the OS is official.

[✔] Environment
    - OS: Mac OS 14.0.0 X64
    ✔ Xcode Command Line Tools: installed
    ✔ rustc: 1.71.1 (eb26296b5 2023-08-03)
    ✔ Cargo: 1.71.1 (7f1d04c00 2023-07-29)
    ✔ rustup: 1.26.0 (2023-04-05)
    ✔ Rust toolchain: stable-aarch64-apple-darwin (default)
    - node: 20.5.0
    - pnpm: 8.5.0
    - yarn: 1.22.19
    - npm: 9.8.0

[-] Packages
    - tauri [RUST]: 2.0.0-alpha.14
    - tauri-build [RUST]: 2.0.0-alpha.8
    - wry [RUST]: 0.33.0
    - tao [RUST]: 0.22.2
    - @tauri-apps/api [NPM]: 1.4.0
    - @tauri-apps/cli [NPM]: 2.0.0-alpha.14

[-] App
    - build-type: bundle
    - CSP: unset
    - distDir: ../dist
    - devPath: http://localhost:1420/
    - framework: Svelte
    - bundler: Vite

[seanaye]

I'm on tauri v2 and MacOS 13.5.1. Sometime the getDisplayMedia call is really long, like multiple seconds, when my app is not doing anything, completely idle. Sometime it the permission dialog appears instantly and sometimes it takes up to 30seconds. Should I open this as a separate issue?

[cliqer]

Upgraded to Sonoma v14 and previous implementation doesn't work anymore. I assume this has to do with Safari 17 changes in WebRTC.

[GorIvanov]

Same in the linux (Arch)!

getUserMedia error: NotAllowedError: The request is not allowed by the user agent or the platform in the current context, possibly because the user denied permission.

webkit2gtk version is 2.42.1-1 It should work fine with webRTC, but.....

Is there any thought when this will be fixed?

[FabianLars]

No idea when this will be fixed, the webkitgtk didn't gave any ETAs or hints but they are actively working on it.

Oh and to be clear, Linux does not have webrtc support yet so there isn't any "bug" preventing it, it's just not there yet.

[tarushnagpal]

Upgraded to Sonoma v14 and previous implementation doesn't work anymore. I assume this has to do with Safari 17 changes in WebRTC.

@cliqer I am not sure that is the case, I just launched a simple WebView from xcode with a target of Sonoma 14.0 and did a simple navigator.mediaDevices.getDisplayMedia() and it worked, this might be something tauri specific not sure though

[callumcox]

I've been hitting the same problem and think I've narrowed it down to the flags supplied to the webview in Wry...

Using webkit2gtk-4.0 (running on Ubuntu), if I start a WebView pointing at the same application as I'm firing up in Tauri Media Stream and Web RTC are defaulted to off. If I start a WebView with --enable-media-stream=true and --enable-webrtc=true the view correctly prompts for permissions when navigator.mediaDevices.getUserMedia({ audio: true, video: true }) is called and can then call navigator.mediaDevices.enumerateDevices().

Looking at the args supplied to the webview these flags aren't being supplied and so I'm assuming it's defaulting to webkit2gtk's defaults of off (though this is based on a cursory inspection of the code and some extrapolation based on what I've observed).

I also couldn't find a way to supply additional args to enable these on the target platform that exists for Windows.

[stolinski]

@callumcox Were you able to supply those flags in an app with Tauri apis?

[tarushnagpal]

Upgraded to Sonoma v14 and previous implementation doesn't work anymore. I assume this has to do with Safari 17 changes in WebRTC.

@cliqer Were you able to get this to work in MacOS?

[cliqer]

@cliqer Were you able to get this to work in MacOS?

@tarushnagpal Unfortunately no

[tarushnagpal]

@cliqer Were you able to get this to work in MacOS?

@tarushnagpal Unfortunately no

Damn, Launching a webview in a fresh xcode app and calling a getDisplayMedia works just fine, has to be something with how we instantiate the web view from tauri, will get time to take a look over the weekend hopefully 🤞.