[Wayland] The initial value of the mouse pointer in warpd normal mode is the current mouse pointer position

[tkna91]

Thanks for your cool project.

I think it would be easier to perform consecutive operations if the current mouse pointer position is the initial value of the warpd normal mode mouse pointer. The current version, which starts operations from the center of the screen, would be fine by default, but in that case, it would be nice to have the above as an option. I would appreciate your consideration.

Sorry if I am saying something strange.

[herrsimon]

Are you using wayland? On X11 the mouse position is not touched when starting warpd (either via --normal or by the activation key after daemonizing).

[tkna91]

@herrsimon Yes, I am using Wayland (sway). Could it be that Wayland cannot get the mouse pointer position? Is this behavior inevitable?

https://user-images.githubusercontent.com/102382754/177054787-53d9a3aa-0f55-4e16-b4c1-29c8c9fba55f.mp4

[herrsimon]

I checked the source code and for wayland getting the pointer position is indeed marked as a TODO. Unfortunately I have absolutely no experience with wayland so you should wait for @rvaiya to implement this. From the little I understand, it's likely that libinput provides the necessary routines.

EDIT: this looks promising.

[tkna91]

I see, I will wait. @rvaiya, I know it may be a lot of work, but if you feel up to it, please do. @herrsimon Thanks for your kindness.

[rvaiya]

At the moment I don't believe this is possible.

As far as I know, there isn't a way to obtain pointer coordinates, so warpd has to keep track of them internally.

I haven't plumbed the depths of some of the newer protocols, though I imagine this sort of thing might be considered a security risk by the wayland folks. If someone knows better please chime in.

Getting warpd to its present state of usability on wayland already required abusing some of the experimental protocols.

EDIT: this looks promising.

warpd doesn't have direct access to the input devices, so it can't make use of any libinput functionality.

[herrsimon]

warpd doesn't have direct access to the input devices, so it can't make use of any libinput functionality.

Damn, I intended to give wayland a try within the next months but this is a strong argument against doing so.

What I'm saying now is probably nonsense as I have absolutely no clue how wayland actually works: Could a (very ugly) solution be to call warpd from a wrapper script with setuid root or any other sufficiently privileged user, which queries global pointer coordinates and passes them as options to the warpd call?

[rvaiya]

Could a (very ugly) solution be to call warpd from a wrapper script with setuid root or any other sufficiently privileged user, which queries global pointer coordinates and passes them as options to the warpd call?

My understanding is that the compositor is the only component which keeps track of cursor position. Even if you had root access you would need some way to query the internal state of the compositor. X provides an API mechanism for this, I believe wayland does not (for security reasons).

[herrsimon]

I see, even though my knowledge of X and wayland still resembles a white canvas, there are more and more colored speckles appearing thanks to your useful comments.

[tkna91]

It is a naive and optimistic idea. Sorry if it doesn't work.

Screenshot method

1. register the mouse pointer image in warpd
2. take a screenshot of the entire screen
3. find and retrieve the position coordinates of the registered mouse pointer

Layer click method

1. create a layer that covers the entire screen
2. click the mouse with keyd-like object and get the coordinates

I feel that if someone could find a way to achieve such a thing and make it into a library, many people would migrate from X to Wayland.

[rvaiya]

That approach is a bit ham fisted and probably quite expensive. I don't know if the end result would be usable and it would require some non trivial changes to the wayland platform code. PRs are welcome if someone wants to invest the effort and is able to produce something usable.

Before doing that it would probably be worth taking a deeper dive into some of the protocols to see if there is in fact a way to obtain pointer coordinates. It is possible I have missed something.

[tkna91]

After doing slurp -p, I did leftmouse with keyd and got the coordinates. How about this?

• https://man.archlinux.org/man/slurp.1
• https://github.com/emersion/slurp

https://user-images.githubusercontent.com/102382754/177866267-a66a0084-95c3-45cc-a3d1-489cf6723bdd.mp4

It appears that no actual clicks have been made.

https://user-images.githubusercontent.com/102382754/177867412-78b87118-29a1-4e6b-b392-1956237cac6e.mp4

[rvaiya]

Thanks.

A cursory look at the code would suggest slurp is using a similar trick to the one you described. Namely creating a large transparent surface and then extracting coordinates (using API calls rather than image analysis).

It should be possible to employ a similar technique in warpd, but I don't have time at the moment. There are a number of other issues with the wayland implementation of warpd that I hope to remedy in my keyd implementation, to which I am more inclined to direct my efforts. PRs are welcome in the meantime.

[tkna91]

It seems that ydotool can manipulate the mouse pointer in relative coordinates https://github.com/ReimuNotMoe/ydotool

# Relatively move mouse pointer to -100,100:

ydotool mousemove -- -100 100

Maybe there is a way to figure out the current coordinates of the mouse pointer or a way to directly move the relative position of the mouse pointer. Sorry if I am missing the point.

Commands executed

# ydotoold &

/etc/keyd/test.conf

[ids]
*

[main]
tab = overload(TAB, tab)

[TAB]
j = command(ydotool mousemove --  0 -10)
k = command(ydotool mousemove --  0  10)
h = command(ydotool mousemove -- -10  0)
l = command(ydotool mousemove --  10  0)

https://user-images.githubusercontent.com/102382754/186562742-0f79dd59-7afb-44ae-9378-9772744a0dde.mp4

[rvaiya]

The internal platform API (and many core features) is designed around being able to query cursor position. Relative movement is insufficient, the correct way to achieve this is probably to take the slurp approach. I just don't have time at the moment.

[roland-5]

@rvaiya is this somehow helpful? https://github.com/flatpak/xdg-desktop-portal/pull/711

[rvaiya]

@roland-rollo

I don't believe that is relevant. As I understand it, that is a new dbus protocol to allow for things like global key bindings, and isn't universally supported. As I mentioned in my last post, this should be possible to achieve using native wayland protocols, I just haven't had the time to implement it.

[rvaiya]

I decided to invest some time on this since a keyd implementation is some time off and this seems to be a perennial issue. The latest commit should include a fix. I would appreciate it if someone with multiple screens could test this, as I don't have a multi head setup at the moment.

[tkna91]

I checked in a multi-display environment. The pointer's starting point in normal mode appears to be fine! thanks!

$ paru -Q warpd-wayland-git
warpd-wayland-git r205.731f533-1
$

https://user-images.githubusercontent.com/102382754/194062826-04c72bea-8179-4627-9f6d-e9a9d06d4f15.mp4

[rvaiya]

Thanks for the confirmation.