SpacingBat3
Provide a way to select resolution and framerate in screenshare
Aknowledgements
-
[X] I have checked that there's no other issue describing the same or similar problem that I currently have, regardless if it has been closed or open.
-
[X] I can confirm that this is not an issue with the Discord website, but it is a problem specific to the WebCord itself. I have tested if this bug occurs on Chromium/Chrome or any other Chromium-based browser that uses unpatched/upstream Chromium engine.
-
[X] I have tried running the build from the
masterbranch and it does not have any fixes implemented according to my issue. -
[ ] My issue describes one of the unstable and/or not fully implemented features.
-
[ ] I have found a workaround to mitigate or temporarily fix this issue in affected releases (please write it in Additional context section below).
Operating System / Platform
🐧️ Linux
Operating system architecture
x64 (64-bit Intel/AMD)
Electron version
v21.3.0
Application version
v3.10.0
Bug description
Screenshare on WebCord with Wayland drops lots of frames, but this isn't WebCord's bug because it's the same issue with Firefox and Chrome as well. However, recently I discovered https://github.com/maltejur/discord-screenaudio which enables screenshare without frame drops but I'm unsure how it does it.
Upon further inspection, I found that Discord uses H264 to encode the video when sharing with Firefox/Chrome/WebCord, but switches to VP8 when using discord-screenaudio. I'm not sure what causes this or if it's even intentional on the part of discord-screenaudio, but it does make the stream not drop frames.
I'm not very familiar with getUserMedia, so I can't tell myself what it's doing differently but it'd be nice if WebCord could also stream without dropped frames
Additional context
No response
Discord uses H264 when HW encoder is available, VP8 otherwise. So not much of a mystery, you can even get WebCord to use VP8 with the optimize client for current gpu flag unchecked.
After trying out various things though, the difference seems to be that when streaming from WebCord, it won't sacrifice the quality of the frames, even if it means there aren't many of them. Whereas discord-screenaudio looks blocky and pixelated but tries to stay at the specified fps.
Discord stream won't go above 4000 bitrate for me (as well as anyone else I asked to check) though, so it makes more sense to stream 1080p30 or 720p60, as opposed to 1080p60 or 1080p144 (which is what WebCord does?).
I guess this issue is now "Provide a way to select resolution and framerate in screenshare", and figure out why WebCord would rather stream a high quality slideshow instead of 60k pixels at 60fps, and if there's a way to let the user set a preference depending on the stream content
Discord uses H264 when HW encoder is available, VP8 otherwise. So not much of a mystery, you can even get WebCord to use VP8 with the optimize client for current gpu flag unchecked.
I guess it might be more a difference between VA-API and software encoding in Chromium – I don't think Discord has any way to control shared stream encoding to their servers (surely they can process it further, but that's made server-side).
I guess this issue is now "Provide a way to select resolution and framerate in screenshare", and figure out why WebCord would rather stream a high quality slideshow instead of 60k pixels at 60fps, and if there's a way to let the user set a preference depending on the stream content
No, I won't work on it, I think it's out of the scope of WebCord as it doesn't focus on re-implementing WebRTC for Discord (at least for now) – unless either Chromium or Electron respects some kind of non-standard stream constrains to change that. And even if working on that would be in WebCord's scope so, you should definitely report that then as a feature requests (and I believe it would be then marked as a duplicate, as I saw similar report some time ago)…
Am I correct in assuming that this currently isn't possible in WebCord because the native discord client (and discord-screenaudio?) uses Electron's API for screen grabbing while Webcord/Chromium uses the browser API?
I do much prefer leaving the monitor/application selection up to the system (like we do with xdg-desktop-portal). But; if Electron's API's do have a standard way of changing res@rate then it'd likely be the better option for webcord as well.
Am I correct in assuming that this currently isn't possible in WebCord because the native discord client (and discord-screenaudio?) uses Electron's API for screen grabbing while Webcord/Chromium uses the browser API?
Nah, there's no other main process API than desktopCapturer, which is used for fetching information and indentifiers about available screens or windows on X11 (on X/-Wayland, it returns a single entry which is used for PipeWire Capturer and skips dialog window).
On renderer side (i.e. within process that manages the web content) WebCord uses navigator.mediaDevices.getUserMedia() with non-standard constrains under the hood as navigator.mediaDevices.getDisplayMedia() and spoofs some properties like function name or (in case of getUserMedia) entire function logic to make it believable it isn't running on Electron.
X11 dialog for screen sharing is also completely separated from the web page (it is drawn using BrowserViews, so even if it looks like as a part of the website, it actually isn't), so there's a lot of communication going through the Electron's IPC while negotiating the recording sources between like three processes (two renderers + main process I suppose).
As of methods used in official Discord and probably discord-screenaudio, I believe they use completely different WebRTC / screen capturing implementation, on Discord they use a proprietary module designed for their own server architecture (I believe it might not work well for every WebRTC application) while I think discord-screenaudio is based on WebKit so it definitely differs from Chromium (or as it seems to be written in C, maybe they actually handle things on their own?).
I do much prefer leaving the monitor/application selection up to the system (like we do with xdg-desktop-portal). But; if Electron's API's do have a standard way of changing res@rate then it'd likely be the better option for webcord as well.
As said before, it probably don't have or it is being undocumented; WebCord can only manage how screen is being recorded or optimized by declaring stream constrains, which are quite specific for each browser + Electron has a few custom ones to actually be able to capture screen or window. By looking at docs, it seems to have some constrains to limit window size (there's nothing about the framerate through), but I have no clue how that should be set and which value pairs are valid (I don't think any value is accepted and that we probably want to consider how that should work in case of different screen or window aspect ratio than 16:9).
discord-screenaudio injects an userscript and calls functions from it when starting a stream https://github.com/maltejur/discord-screenaudio/blob/master/assets/userscript.js#L126
@llyyr Due to the recent changes within Electron about screen sharing APIs availability, I might wait a bit to decide how to handle the switch to the new API (#353). Note that new API is a bit less complex and would avoid wrapping getDisplayMedia (removing intensive IPC communication between multiple processes), yet it is very limited so I might not be able to modify stream constrains…