Hyprland support

[jheroy]

Original issue title: left, right and middle click error

wl_display@1: error 0: invalid object 17

system: arch linux, hyprland-git

[moverest]

This bug has been fixed in the main branch (commit ee21ec7a37e8f7327f3f0052a8dacf6a3bdd8514) but I haven't made release yet.

I've just tried wl-kbptr on Hyprland 0.40 and it's main branch (commit 4cdddcfe) and it seems to be working but for some reason the compositor sometimes crashes when the program tries to move the mouse cursor.

There's also the issue that Hyprland moves the cursor back to the center of the active window when wl-kbptr exits. I've just tried a fix that I've pushed to the main branch (d87630ef88a377024946526841cce12d3d142615) which should address that. I tested it a bit and it seems to be working however sometimes the cursor doesn't move to the expected location (and it's not the center of the active window this time).

[jheroy]

I am using the latest code, and my experience aligns with yours; it appears that running programs through Xwayland is causing the crash. I have two monitors set up in the following configuration:

monitor=DP-1,highres,auto,1
monitor=DP-2,highres,-2160x-1190,1,transform,1

When I click on DP-1, it actually clicks on the corresponding area of DP-2.

[moverest]

I tried without any Xorg application and it still crashes randomly unfortunately. There's definitely an issue on Hyprland's side as even assuming a client behaves erratically it shouldn't crash.

As for the multi-display issue, Hyprland just doesn't support output selection with the wlr-virtual-pointer-unstable-v1 interface as I've found in Hyprland/src/protocols/VirtualPointer.cpp:139.

[moverest]

For reference, here's an issue I found where the implementation of the wlr-virtual-pointer-unstable-v1 interface was discussed: https://github.com/hyprwm/Hyprland/issues/67.

[jheroy]

When I use wlrctl pointer move xx xx in Hyprland, it also crashes, but using hyprctl dispatch movecursor xxx xxx does not.

[moverest]

That makes sense, wlrctl uses the same wlr-virtual-pointer-unstable-v1 protocol as wl-kbptr and Hyprland's implementation is broken. hyprctl is most likely not using that.

[jheroy]

Perhaps wl-kbptr could be designed to solely provide the functionality of selecting coordinates, with the action of clicking being implemented through a shell script. For instance, once a coordinate is selected, a script can be invoked, which is responsible for determining how to move to the corresponding coordinate and how to perform the click. For Hyprland, it would suffice to simply call a command hyprctl dispatch movecursor xxx xxx . As for simulating mouse clicks for the left, middle, and right buttons, that should be relatively straightforward to achieve.

[jheroy]

As for coordinate selection, it could be designed to target the current program rather than the entire desktop. Here's an idea for selecting coordinates within the current program: take a screenshot of the program, then use OpenCV to recognize and label the content of the image. This approach should theoretically work for all interfaces and provide an accurate match to the elements. Below are sample images and code:

import cv2
import numpy as np

image_path = 's.jpg'
image = cv2.imread(image_path)

gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

edges = cv2.Canny(gray, 50, 150, apertureSize=3)

kernel = np.ones((5, 5), np.uint8)

dilated_edges = cv2.dilate(edges, kernel, iterations=1)

contours, _ = cv2.findContours(dilated_edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

for contour in contours:
    hull = cv2.convexHull(contour)
    
    x, y, w, h = cv2.boundingRect(hull)

    cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)

cv2.imshow('Image with Text Line Boxes', image)
cv2.waitKey(0)
cv2.destroyAllWindows()

[moverest]

Sure, we could have the program only output the selected area and the intended mouse click and not actually try to control it. This would also enable the support of other compositors such as KWin though a wrapper script.

As for you second point, I kinda like the idea. Initially I was looking for ways to do that through an accessibility protocol but this doesn't exist on Wayland unfortunately (I think GTK has something for this but that would limit us to few programs). I would be interested in trying the OpenCV approach. This could also be accomplished with a helper script which would feed the coordinates of the selectable areas to the program. It would require the addition of a new selection mode. Since this is extending the scope of this issue, could you please open an issue for this point?

[moverest]

I've just pushed commits to be able to select the output (--output) and only print the result and not move the pointer or simulate a click (--only-print). It should be pretty straightforward now to write a wrapper for Hyprland.

[moverest]

I've just pushed the wl-kbptr-hyprland helper script that uses the hyprctl command to move the cursor. I haven't found the command to use to actually click yet — granted I haven't looked for more than five minutes. So far, from my tests it seems to be working pretty well.

[matejdro]

If anyone is interested, here is my hyprland config for mouse control. It uses submap, similar to the mode in the README's sway config:

bind=$mainMod,Backspace,exec,hyprctl keyword cursor:inactive_timeout 0; hyprctl keyword cursor:hide_on_key_press false; hyprctl dispatch submap cursor
bind=$mainMod,plus,exec,wl-kbptr && (hyprctl keyword cursor:inactive_timeout 0; hyprctl keyword cursor:hide_on_key_press false; hyprctl dispatch submap cursor)

submap=cursor

binde=,j,exec,wlrctl pointer move 0 10
binde=,k,exec,wlrctl pointer move 0 -10
binde=,l,exec,wlrctl pointer move 10 0
binde=,h,exec,wlrctl pointer move -10 0

bind=,m,exec,wlrctl pointer click left
bind=,comma,exec,wlrctl pointer click middle
bind=,period,exec,wlrctl pointer click right

binde=,e,exec,wlrctl pointer scroll 10 0
binde=,r,exec,wlrctl pointer scroll -10 0
binde=,d,exec,wlrctl pointer scroll 0 -10
binde=,f,exec,wlrctl pointer scroll 0 10

bind=,escape,exec,hyprctl keyword cursor:inactive_timeout 3; hyprctl keyword cursor:hide_on_key_press true; hyprctl dispatch submap reset 

submap = reset

I have hide_on_key_press and inactive_timeout settings enabled for cursor, so those settings have to be disabled first, otherwise cursor will be invisible when moving.

Works great on

Hyprland, built from branch  at commit 918d8340afd652b011b937d29d5eea0be08467f5  (flake.lock: update).
Date: Tue Jun 25 12:06:02 2024
Tag: v0.41.2, commits: 4886

wl-kbptr v0.2.1

[moverest]

That doesn't look too bad. Feel free to open an MR to add this to the README. :slightly_smiling_face:

Out of interest, @matejdro do you have a single monitor setup?

[matejdro]

Do you want me to change the shortcuts to match existing Sway shorcuts in the README?

I'm using two monitors.

[moverest]

It seems like everything in working with Hyprland now — they fixed all their bugs. Closing this issue.