Mouse and keyboard events as in Xlib

[VictorGimenez]

Is there available any resource on pywayland to control mouse and keyboard events like the method fake_input() in Xlib?! I am trying to implement a version of PyAutoGUI for use in my company where be possible to automate activities from a Debian's based distro using Wayland protocol

[m-col]

If it's all client-side, where your client tool is requesting the compositor simulate key and mouse events, possibly into other clients, your best bet would be using the wlroots virtual keyboard and virtual pointer protocols:

• https://gitlab.freedesktop.org/wlroots/wlroots/-/blob/master/protocol/virtual-keyboard-unstable-v1.xml
• https://gitlab.freedesktop.org/wlroots/wlroots/-/blob/master/protocol/wlr-virtual-pointer-unstable-v1.xml

You can access these from python using pywlroots:

• https://github.com/flacjacket/pywlroots/blob/main/wlroots/wlr_types/virtual_pointer_v1.py
• https://github.com/flacjacket/pywlroots/blob/main/wlroots/wlr_types/virtual_keyboard_v1.py

[VictorGimenez]

I found one example around the net on how to implement a script using pywayland to simulate mouse pointer motion, I ran one example and I tried to did some updates but I am in doubt about where could I get this WlCompositor and id_ that are commented after display.dispatch().

Here is the code to go to the positions x = 26 and y = 665:

import pywayland.client
#import os

sx=26
sy=665

def handle_pointer_enter(pointer, surface, sx, sy):
    print("Mouse entered surface at", sx, sy)

def handle_pointer_leave(pointer, surface):
    print("Mouse left surface")

def handle_pointer_motion(pointer, time, sx, sy):
    print("Mouse moved to", sx, sy)

def handle_pointer_button(pointer, time, button, state):
    print("Mouse button", button, "was", "pressed" if state else "released")

def move_mouse(pointer, x, y):
    # Move the mouse to the specified position
    pointer.set_cursor(pointer.get_serial(), None, x, y)
    
    
def main():
    #display = pywayland.client.Display(os.environ['DISPLAY'])
    display = pywayland.client.Display()
    display.connect()

    registry = display.get_registry()
    #registry.dispatch()
    display.dispatch()

    #pointer = registry.bind(pywayland.client.core.WL_POINTER_INTERFACE, pywayland.client.wl_pointer, version=3)
    pointer = registry.bind(pywayland.protocol_core.interface, pywayland.protocol.wayland.wl_pointer, version=3)  #I have tried to modify 
    #these parameters but the error persists, I saw that registry.bind receives (id_, WlCompositor and version)
    pointer.on_enter = handle_pointer_enter
    pointer.on_leave = handle_pointer_leave
    pointer.on_motion = handle_pointer_motion
    pointer.on_button = handle_pointer_button

    move_mouse(pointer, 26, 661)
    
    while True:
        display.dispatch()

if __name__ == "__main__":
    main()

[m-col]

To be honest I'm not really sure. Some of that looks vaguely like code that a server would run rather than a client like this, such as the while True event loop handling. Also I'm not sure whether WL_POINTER_INTERFACE is actually what you're after... is that for controlling the pointer? Unless this snippet is already working to move the pointer, it doesn't look to me like this would work.

But actually it looks like the wlroots examples folder has an example client that moves the pointer: https://gitlab.freedesktop.org/wlroots/wlroots/-/blob/master/examples/virtual-pointer.c. It's in C but hopefully translating it to Python (using pywlroots) is straight forward.

[VictorGimenez]

To be honest I'm not really sure. Some of that looks vaguely like code that a server would run rather than a client like this, such as the while True event loop handling. Also I'm not sure whether WL_POINTER_INTERFACE is actually what you're after... is that for controlling the pointer? Unless this snippet is already working to move the pointer, it doesn't look to me like this would work.

But actually it looks like the wlroots examples folder has an example client that moves the pointer: https://gitlab.freedesktop.org/wlroots/wlroots/-/blob/master/examples/virtual-pointer.c. It's in C but hopefully translating it to Python (using pywlroots) is straight forward.

I have tried to execute this script that I presented to you but it doesn't run, I gave a pseudocode to ChatGPT with the aim to generate a small code snippet to make the mouse pointer go to the coordinates x,y = 26,661 and it returned me this script above, but it fails at the line: pointer = registry.bind(pywayland.client.core.WL_POINTER_INTERFACE... which is commented so I have tried to substitute the line above with this one: pointer = registry.bind(pywayland.protocol_core.interface, pywayland.protocol.wayland.wl_pointer, version=3) and nothing.

Also I saw that test/test_registry_bind.py the method registry.bind receives three parameters id_, WlCompositor, version and these id_ and WlCompositor I don't know where they came from.

I am gonna try to execute this example that you sent, study it and convert it to python! Thank you for the tip @m-col!!

[VictorGimenez]

At Xlib (X11/X Window System) library I use to do this script in python which works to automate pointer events:

from Xlib.display import Display
from Xlib.ext.xtest import fake_input
from Xlib import X
import os

#Go to x and y coordinates and click with left button (1):
x = 26
y = 665
button = 1

myDisplay = Display(os.environ['DISPLAY'])

fake_input(myDisplay, X.MotionNotify, x=x, y=y)
myDisplay.sync()
fake_input(myDisplay, X.ButtonPress, button)
myDisplay.sync()
fake_input(myDisplay, X.ButtonRelease, button)
myDisplay.sync()

And now I am studying how X11 x Wayland deal with the screen to allow mouse and keyboard automation.