8357584: [XWayland] [OL10] Robot.mousePress() is delivered to wrong place

[azvegint]

This changeset introduces an adapted version of the OpenJDK fix for mouse and keyboard interactions with Robot. More info about the issue itself is available in the PR description

In short, the currently used XTest for keyboard and mouse interactions may not be suitable for automated testing at some point, as it may require user confirmation to control the mouse or keyboard from time to time.

This fix adds support for the Remote Desktop XDG portal. This allows us to control the keyboard and mouse with Robot on Wayland, even outside the XWayland server (e.g., clicking on window headers and Wayland native apps).

---

• The remote desktop for Robot is enabled by default on GnomeShell 47+
• It can be enabled manually by setting the javafx.robot.screenshotMethod system property to dbusRemoteDesktop (e.g. it works on Ubuntu 24.04 with GnomeShell 46)
• The key handling might still have bugs.
• The main goal is to add this new functionality. If there are small issues that can't be solved right away, I will prefer to fix them in follow up fixes.

---

Progress

• [x] Change must not contain extraneous whitespace
• [x] Commit message must refer to an issue
• [ ] Change must be properly reviewed (2 reviews required, with at least 1 Reviewer, 1 Author)

Issue

• JDK-8357584: [XWayland] [OL10] Robot.mousePress() is delivered to wrong place (Bug - P3)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jfx.git pull/1834/head:pull/1834
$ git checkout pull/1834

Update a local copy of the PR:
$ git checkout pull/1834
$ git pull https://git.openjdk.org/jfx.git pull/1834/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 1834

View PR using the GUI difftool:
$ git pr show -t 1834

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jfx/pull/1834.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper[bot]]

:wave: Welcome back azvegint! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk[bot]]

@azvegint This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8357584: [XWayland] [OL10] Robot.mousePress() is delivered to wrong place

Reviewed-by: lkostyra, arapte

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 8 new commits pushed to the master branch:

• 0270847b85ab6d4701abb201ec4b2aeee96129dd: 8351867: No UI changes while iconified
• b9dd4dec1bf93f56e918fd60225fa7ead8dcccca: 8357393: RichTextArea: fails to properly save text attributes
• d4cb730d01998f1ab0cdc6ed07b0289c0b59d14f: 8356652: Input field ignores custom input source characters
• ... and 5 more: https://git.openjdk.org/jfx/compare/48282b1067e37092916a9f7f0edafa66a43cfb5a...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[mlbridge[bot]]

Webrevs

• 00: Full (df733e7c)

[kevinrushforth]

Reviewers: @kevinrushforth @lukostyra

/reviewers 2

[openjdk[bot]]

@kevinrushforth The total number of required reviews for this PR (including the jcheck configuration and the last /reviewers command) is now set to 2 (with at least 1 Reviewer, 1 Author).

[kevinrushforth]

@arapte Would you also be able to look into this in case I am not able to?

[azvegint]

I see most checks relate to Gnome and its version - what would be the behavior if we ran the tests on a different desktop env like KDE?

This question has already been raised on the OpenJDK PR. The short answer is that it's an unsupported configuration (I am unsure about JFX), so I didn't test it. If some tuning is required to support it, I would prefer to address that as a follow-up fix.

Please see quote from https://github.com/openjdk/jdk/pull/25265#discussion_r2093639881 below:

---

BTW how it is expected to work on KDE + wayland?

It is not a supported configuration.However, when I tested it a while ago, it only worked in a single-screen scenario for the Screencast. There was an issue with the screen streams; they did not specify their x and y coordinates. This prevented the correct arrangement of multiple screens. I don't know if this has been fixed, but the issue is addressed in our code here:

https://github.com/openjdk/jdk/blob/3dd34517000e4ce1a21619922c62c025f98aad44/src/java.desktop/unix/native/libawt_xawt/awt/screencast_portal.c#L132-L155

In this case, the Screencast is simply disabled.

The Remote Desktop uses the same streams from Screencast portal to control the mouse position.

[azvegint]

/integrate

[openjdk[bot]]

Going to push as commit 639a5950b6eae7870fd9d9e84f9ce81322aaab82. Since your change was applied there have been 8 commits pushed to the master branch:

• 0270847b85ab6d4701abb201ec4b2aeee96129dd: 8351867: No UI changes while iconified
• b9dd4dec1bf93f56e918fd60225fa7ead8dcccca: 8357393: RichTextArea: fails to properly save text attributes
• d4cb730d01998f1ab0cdc6ed07b0289c0b59d14f: 8356652: Input field ignores custom input source characters
• ... and 5 more: https://git.openjdk.org/jfx/compare/48282b1067e37092916a9f7f0edafa66a43cfb5a...master

Your commit was automatically rebased without conflicts.

[openjdk[bot]]

@azvegint Pushed as commit 639a5950b6eae7870fd9d9e84f9ce81322aaab82.

:bulb: You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.