Distinguish emulated input and user input

[raphaelmenges]

Hello, thanks for this library!

I am wondering whether it would be possible to add information for mouse and key events whether they have been caused by emulation (e.g., virtual keyboard) or by a user (e.g., physical keyboard). At least on Windows, there appears to be a flag that specifies this information per input event. I have not found a similar flag for macOS or Unix systems, but perhaps someone else has more insights.

[kwhat]

We can probably do something like this around line src/windows/input_hook.c:L215

event.data.keyboard.keycode = keycode_to_scancode(kbhook->vkCode, kbhook->flags);
event.data.keyboard.rawcode = (uint16_t) kbhook->vkCode;
event.data.keyboard.keychar = CHAR_UNDEFINED;

if (kbhook->flags & LLKHF_INJECTED) {
    event.reserved = 0x02;
}

logger(LOG_LEVEL_DEBUG, "%s [%u]: Key %#X pressed. (%#X)\n",
        __FUNCTION__, __LINE__, event.data.keyboard.keycode, event.data.keyboard.rawcode);

The same thing probably needs to go on the key typed and key release and all of the mouse events as well.

[raphaelmenges]

I investigated whether macOS / Darwin offers a similar indicator as Windows does. However, I could not find anything. At least the CGEventFlags do not contain any information whether an event was caused by a real device or programmatically emitted: https://developer.apple.com/documentation/coregraphics/cgeventflags

[raphaelmenges]

It looks like under macOS hardware events are initialized with a reference to a specific event source. Perhaps the event source can provide indication whether an event is caused by emulation or by user input?

[raphaelmenges]

Hello @kwhat: I finally got time to work with libuiohook in more depth. I am already using the code from 1.3 branch and would like to know whether you have any estimation when you will implement the feature of this issue - distinguishing between emulated and user input. If you are too busy, I would consider to fork libuiohook and "hack" the above approach into it for my purposes!

[kwhat]

you have any estimation when you will implement the feature of this issue

Unfortunately, there is a lot of work that needs to be done. I am currently 2/3 of the way though a refactor for 1.3 to try and organize the code a bit better. Linux needs to move to evdev sooner than later. I need to figure out what evdev support means for BSD. And finally I need to implement this feature.

You can definitely start hacking on the 1.3 branch and I will gladly accept pull requests. IIRC Windows and Linux already provide this information and we just need to check for it and add it to an event. As usual, OSX will be more work.

We should probably widen uint16_t mask; to uint32_t mask; and remove uint16_t reserved; as evdev will render that obsolete. Create a few new modifier masks for preventing event propagation and an emulated flag.

[steffen-liersch]

It is important to recognize emulated events. This is the only way to perform a special operation on a certain key combination and finally emulate the same key combination. It is not possible to find a reliable workaround if there is no property to detect emulated events. It would be great if this feature would be available soon. Thanks a lot!

[kwhat]

I've opened up a pull request to add this functionality to Windows and Linux. OS X is still TBD. I haven't tested any of this code so let me if if it works.

[raphaelmenges]

Thanks @kwhat for the effort!

I tested 8deef04751f03ee5f7573e1d0b4c00bc91328f47 in my environment using Windows 10 and emulated keypresses are recognized as such but emulated mouse clicks and moves are not. If required, I can also make a dedicated example. However, I am quite limited in time and perhaps it is something obvious?

Besides, is there any define of the 0x02 value to check the reserved variable nicely for that emulated-flag?

[kwhat]

@raphaelmenges I made a typo. I've updated the branch to 53c6223 which should resolve the issue with mouse events on windows.

[raphaelmenges]

I can confirm it works on my Windows 11 machine! Thanks so far. Looking forward for the macOS integration 😇

[kwhat]

macOS support has been added in c522d97. If you have a mac, please test and let me know if it seems to work correctly. If everything looks good I will probably back port this to 1.2 and move it to a non-reserved element for 1.3.

[raphaelmenges]

I can validate that key press, mouse move, and mouse click can be distinguished between being spawned by my physical input devices or emulated on my Mac (macOS 13.3.1, Apple Silicon).