Wayland support?

[zachlatta]

Does this project support Wayland? I'm running the latest on https://aur.archlinux.org/packages/weylus-bin/ and while I have input working, I'm getting a black screen on the iPad browser I'm using it from.

Input also seems to be smooth in Xwayland apps, but jumpy in Wayland native apps.

Update: See recording of this at https://www.youtube.com/watch?v=GrJXqqLKas4. All input is made via Apple Pencil on an iPad. Notice that the input is smooth when over xeyes, but jumpy over Alacritty (which is a Wayland app).

[H-M-H]

Right now screen casting is not supported for Wayland. Currently the screen is captured either via autopilot, which only supports X11, or using XShmCreateImage here: https://github.com/H-M-H/Weylus/blob/435373be5141a2b03aeab6549225a5a5790ed1b7/lib/linux/capture.c .

To get Wayland to work there are some things I still need to find out:

• check if we a running Wayland or X (this is probably easy)
• find an API to capture the screen and selected windows, that is fast enough
  • preferably similar to the X11 API that allows me to get a pointer to the RGB buffer of some window
• API to request which windows should show up in the window selection menu

If someone wants to help me here a minimal working example of taking a screenshot of a selected window in Wayland would be great, provided it is written in Rust, C or C++. Alternatively point me to a tool that already does that. I will probably write some Trait to abstract different window systems, so I would rather not want a pull request now.

[H-M-H]

About the input issues: I am not sure what's causing that, I do not do anything specific to X11. Can you tell me if libinput debug-events or evtest outputs anything strange?

Also this should probably be a separate issue.

[H-M-H]

Looks like the tool you used (https://github.com/ammen99/wf-recorder) for recording and https://github.com/emersion/grim can be used as a reference.

[zachlatta]

Hey @H-M-H, thank you for your willingness to work on Wayland support and I'm sorry I can't do more to help right now. Realistically, I don't think I'm going to be able to help you debug libinput or evtest right now. I love this project and think it's absolutely amazing. Wish I had more time to help out.

[H-M-H]

Alright, I totally understand that, thanks for your report anyways! Maybe someone else can help out.

[H-M-H]

I did some research and the situation regarding capturing the screen or even windows on wayland is pretty dire. There is no equivalent to X11's XShmGetImage or XGetImage, Wayland itself just does not support capturing the screen. Instead it is the responsibility of the compositor to expose an API to take screenshots. But unfortunately this is not standardized, there is:

• wlroots' wlr-screencopy-unstable-v1 which does not work for gnome or kde
• there is Wayland's reference compositor Weston with another way
• there is gnome's screenshot and screen cast interface via dbus, both probably too slow for proper streaming
• there is kde's dbus interface for screenshots

[pointhi]

What about PipeWire?

Capture and playback of audio and video with minimal latency.

[H-M-H]

Interesting, I did not know about PipeWire, the docs are a bit lacking but PipeWire is used by webrtc, so a look at how they do it might help. @pointhi do you know what latency one can expect using PipeWire? For reference: capturing a 1920x1080 raw bgr0 frame on my system takes about 0.8 milliseconds if the screen contents do not change too much.

[pointhi]

I never used PipeWire, but it was designed for remote desktop applications on Wayland to my knowledge.

[ilya-epifanov]

strangely enough, unfocused windows do get captured, until the very first event sent to them which focuses them and then the output becomes black

[H-M-H]

I think I have a good idea of what's going on: If a window is not focused, Weylus captures it directly by its window handle but if it is active Weylus captures the part of the root window containing the window to be captured. The reason for this is to have things like context menus that are their own windows in themselves still show up. But I guess XWayland doesn't like capturing the root window and just decides to have it black.

[H-M-H]

Please check if things work better with the latest build: https://github.com/H-M-H/Weylus/actions/runs/585393503#artifacts

[H-M-H]

I started experimenting with pipewire as this seems to be the most promising solution here: https://github.com/H-M-H/Weylus/tree/wayland

For now only requesting a screen cast via org.freedesktop.portal.ScreenCast is implemented. That means there now is a file descriptor that can be used to connect to pipewire. Unfortunately the docs on how to use pipewire directly are still very lacking. For example pw_remote_connect_fd mentioned in the docs for org.freedesktop.portal.ScreenCast has been removed in pipewire 0.3 and seems to be pw_context_connect_fd now. All I could find on video processing is: https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/master/doc/tutorial5.md

It looks like pipewire implements a gstreamer plugin, I suspect gstreamer's API is not as badly documented, so this is perhaps worth a look.

A problem that remains is that it looks like there is no way to get the position of a captured window, which means Weylus won't be able to properly map the input to the window. As it stands only capturing the full desktop will be usable under Wayland.

[H-M-H]

For anyone interested, https://github.com/H-M-H/Weylus/commit/d632a7dfdfacd27aaa0576076f9e2d3b0b32d93c implements screen casting via gstreamer. Binaries can be found here. There still are quite some issues, refer to the commit message for details.

[H-M-H]

Looks like once this is resolved, working with cursor metadata via gstreamer will make dynamically showing and hiding the cursor possible.

Regarding window titles I opened a feature request here: https://github.com/flatpak/xdg-desktop-portal/issues/569

[dmoltisanti]

Thanks a lot for working on this :)

[H-M-H]

Thanks! So far walking this path has been rather rocky and I have hit quite some bugs on the way:

Bugs

• kwin_wayland dies and tears down the entire plasma desktop
• PipeWire deadlock No.1
• PipeWire deadlock No.2
• PipeWire returns empty buffers when it shouldn't
• Missing context for position property of pipewire streams
• Inconsistency (fortunately not really a bug) between xdg-desktop-portal-gtk and xdg-desktop-portal-kde

Missing features

• PipeWire streams have no name
• Can not get position of PipeWire streams (this breaks touch input mapping)
• Can not get cursor data from PipeWire (breaks switching displaying cursor without rerequesting permission to capture the screen)

Fortunately I could workaround all PipeWire deadlocks and at least on GNOME Weylus can now capture the desktop and windows and so far it seems not too unstable.

Feel free to try the latest builds.

Considering the current situation I will definitely mark Wayland screen capture experimental and make it opt in.

[l-o-l]

Hi,

I've tried installing weylus on a Pi 4 running manjaro sway. Everything works other than desktop capture. I can capture chromium tabs and weylus itself, but the desktop capture returns the following warning:

WARN weylus::capturable: Failed to get list of capturables via dbus/pipewire: Failed to obtain screen capture.

Chromium itself is able to capture the desktop, as I am able to share the screen from web apps such as Microsoft Teams. Before running I set:

export XDG_SESSION_TYPE=wayland export XDG_CURRENT_DESKTOP=sway /usr/lib/xdg-desktop-portal -r & /usr/lib/xdg-desktop-portal-wlr

This works for chromium, and idea what could be done to get it to work in weylus?

[H-M-H]

This means that Weylus only captures windows available via XWayland. Please check if https://gitlab.gnome.org/snippets/19 also fails.

Some more questions: How exactly does the process for window/desktop selection work with chromium, do you get some kind of popup window that lets you select what to capture and does that popup window also appear for Weylus?

[dron1885]

Hi, also getting WARN Failed to get list of capturables via dbus/pipewire: Failed to obtain screen capture.. No native Wayland windows and only a desktop in the Capture list. The desktop option sets the correct aspect ratio but shows only black.

This means that Weylus only captures windows available via XWayland. Please check if https://gitlab.gnome.org/snippets/19 also fails.

Yes, this snippet works.

Some more questions: How exactly does the process for window/desktop selection work with chromium, do you get some kind of popup window that lets you select what to capture and does that popup window also appear for Weylus?

Don't know how chromium does this, but firefox (for example https://mozilla.github.io/webrtc-landing/gum_test.html) and the python snippet both change the mouse cursor to a cross and I need to click on a desktop I want to share. Actually, Weylus also does this.

Also, input forwarding to desktop does work (but there is no image).

EDIT: Arch Linux, Sway WM, AMD gpu

[mattpepin]

In Gnome/Mutter, the whole desktop works fine (Entire screen option)! It's pretty cool! Now I wish you could use Mutter's headless stuff and create a virtual screen! :)

[H-M-H]

@dron1885 @l-o-l I got it working under sway now, please check if the latest build works for you as well: https://github.com/H-M-H/Weylus/actions/runs/1012629238#artifacts

@mattpepin do you know how Mutter's headless stuff works? Perhaps I could add some examples or even an accompanying script to Weylus.

[dron1885]

@H-M-H Can confirm that it's working. Thanks! (But it seems it requires additional testing in multimonitor setup. From my quick testing, while Waylus was showing a second monitor all inputs were translated to the first one.)

[H-M-H]

Can confirm that it's working. Thanks!

Excellent, thanks for testing!

From my quick testing, while Waylus was showing a second monitor all inputs were translated to the first one.

This is a known problem and sadly not easily fixed. For monitors xdg-desktop-portal provides positions but unfortunately those are compositor dependent/the coordinate system is unspecified and there is no straightforward way to translate them into something useful. For windows the situation is even worse as there is no position given.

Unfortunately I do not see this being solved in the foreseeable future. In my opinion the whole screen recording/remote desktop situation around Wayland is most unfortunate. There have been discussions to add screen recording to Wayland for years with the result that the team behind Flatpak is now somehow responsible for this and there are no plans to extend Wayland itself or provide an official API/Protocol. This means the Flatpak devs effectively have full responsibility/control over how screen casting works. The problem now is that Flatpaks focus is not exactly to provide a nice screen cast/remote desktop interface for Wayland [1]. I can not and will not blame them for that, it is totally understandable if they deem some features required to get a nice remote desktop experience working unnecessary for Flatpak. This is why I don't quite believe in this working anytime soon and the whole discussion doesn't even touch pipewire, which is still under heavy development.

If despite this anyone knows how to obtain the positions of pipewire streams in a way useful to Weylus I very much would like to hear that though.

[1]: There actually is a remote desktop interface. But among other things it doesn't handle resizing windows or pressure sensitivity of touch input.

[raffaem]

Hello,

just a clarification. What does it mean that for Wayland there is no support for "capturing the cursor"?

If I move the stylus on my tablet, will the cursor move accordingly?

[H-M-H]

There are two problems related to this question:

• If you want to see the cursor in the video on your tablet you will need to refresh the capturable list as Wayland/pipewire does not support dynamically toggling its visibility once the video stream has been started. But be warned, the last time I tried this it ended up crashing my KDE desktop.
• Input mapping does not work. This means your cursor will move, but if you have more than one monitor connected or if you only mirror a window the position of the cursor will most likely be wrong.

[raffaem]

Hello,

I am able to move the cursor using the pen on my Fedora 34 Worstation, but the screen on the tablet is completely white.

How can I mirror the PC screen on the tablet?

[H-M-H]

You will have to make sure PipeWire with xdg-desktop-portal is correctly setup, I am not sure how this is done on Fedora. If screen sharing with your browser works, it should also work with Weylus. Perhaps this entry from the Arch Linux Wiki can help you: https://wiki.archlinux.org/title/PipeWire#WebRTC_screen_sharing

[tinywrkb]

Wayland capture doesn't work for me. The only time it sort of worked is when I created a virtual display, and selected it, but the video was frozen, so that wasn't usable.
In Sway, you can create a virtual display like this: $ swaymsg create_output && swaymsg output "HEADLESS-1" resolution 2048x1536, or alternatively via the config file.

[H-M-H]

Hmm, does this snippet: https://gitlab.gnome.org/snippets/19 work for you? Or alternatively does WebRTC screen sharing work? If one of these does work, this is most likely a problem with Weylus otherwise it's a problem with your setup.