libretro
[Android] RetroArch Doesn't Ignore S Pen Hover Actions
Description
RetroArch seemingly makes no distinction between S Pen hover actions and mouse actions, resulting in S Pen functionality within RetroArch being inaccurate.
Expected behavior
When using the S Pen on a supported device (e.g., Galaxy Note Series up to Note 20, Galaxy S Series for S21 Ultra and up, etc.), hovering should do nothing (or move an unseen cursor exactly to where the S Pen is hovering over), and tapping the screen should act identical to using a finger.
Actual behavior
The S Pen hover action moves an unseen cursor on screen that starts at an unknown point to the user, and following taps from the user are registered at the point where the unseen cursor is, rather than where the S Pen actually is tapping on screen.
Steps to reproduce the bug
- Use a S Pen supporting Samsung device
- Go into Samsungs settings and enable Advanced Settings > S Pen > Air view > Previews and pointer
- Open a game with the MelonDS core
- Open MelonDS's core options and enable Screen > Touch Mode > Mouse
- Observe failed behaviour of black cursor vs S Pen location
It is worth noting the bug does not just occur within MelonDS, as it persists throughout the entire UI. MelonDS just offers the best way of visualising how the bug is happening.
Visual produced via MelonDS
Here you can see the S Pen's location (as a hovering action) in actual space be indicated by the white arrow. The black square pixels is where the "unseen" cursor actually is within the emulator. The cursor will register clicks when the user taps the screen (indicated by the white arrow turning into a circle), but not at where the S Pen actually is, and instead only at wherever the black pixels are.
To clarify, the S Pen is never touching the screen in this example except for when the cursor lights up in a circle.
https://github.com/libretro/RetroArch/assets/54557728/7d76da31-07c1-4596-b241-87de618831a4
Additional Notes
Searching through the repository, I can see that stylus support was added in Version 1.7.7. I am unsure if the backend that Samsung uses for stylus's is different than Android core's. I also note that in Android's input drivers, there is a line to exclude stylus input, and to only include touch/mouse input.
https://github.com/libretro/RetroArch/blob/62e4779fee8aaa29a0fc216a79b5f4d5d13f6061/input/drivers/android_input.c#L1418-L1422
Nothing seems to indicate within RetroArch that I have a mouse connected but, nevertheless, the S Pen seems to roughly act as how one would expect a mouse to operate. My rough guess is here that the S Pen's backend is different than other stylus backend on Android, and that it misses this else/if, and gets interpreted as a mouse. That's just my guess as to what's happening however, I am unfamiliar with both the backend for stylus input and s pen input on Android.
Version/Commit
- RetroArch: 1581ece
Environment information
- OS: Android 13, One UI 5.1
- Device: Note 20 Ultra (SM-N986U1)
I am surprised that despite this issue existing for couple of years now, only now there is some effort to document it, thanks @ETJeanMachine :) I am ready to assist with testing on Galaxy Fold, because I am having exactly same issue with DOSBox core and it's sad.
@ETJeanMachine hey thanks again for giving a head start on this. I spent in this rabbit hole whole Sunday and have some good news.
Good news: after refactoring this input handler it was possible to distinguish S-Pen inputs from touch and handle them properly. See https://github.com/libretro/RetroArch/pull/15597 . I was able to use that in DeSmuME with S-Pen on Galaxy Fold:
https://www.youtube.com/watch?v=Xhjjwvs-aV4
Already available on Nightly builds: https://buildbot.libretro.com/nightly/android/
Bad news: most of the cores are using legacy mouse movement emulation that is so basic that it doesn't have any sense of absolute coordinates, just sense of motion and it's controlled by polling deltas from the driver.
It is impossible to map S-Pen coords into motion deltas (this is what I tried for the rest of the Sunday lol), due to lots of factors, but luckily for us - some of the cores like DeSmuME already ditching legacy mouse input for an absolute pointer input, as long as you have a toggle in settings.
Once this added to more cores, such as DOSBox and ScummVM, S-Pen will work fully on them as well. RetroArch itself is using absolute positioning from the driver for its menus and overlays (that's good).
S-pen became fully functional in 1.19 (mainly tested in Nintendo DS cores) , but it seems to be broken again(using hover inputs) in 1.20
@oddgoo yes it was fun to have it on my Galaxy Fold
But there is some old and tangled code inside android_input_poll_input_default that breaks in some combination of bluetooth joysticks on some devices.. very hard to debug
Somebody with better knowledge of what android_input_poll_input_default does has to step in and implement this once again. All the reference code I used is there.
Is there a toggle or ini setting to turn on S-Pen compatibility for users who are willing to risk the incompatibility?
For my part I just run 1.19 and it works fine. Not what you asked, but it's a good enough solution for my case.
No idea, sorry. 1.19 is a safe bet unless you absolutely need something newer. Otherwise you could go looking through the commit history?
What's the latest nightly build before it was reverted?
It has been reverted with this commit so you should take build before 10 June 2024.
Thank you.
Thanks! I really appreciate the responses. I tried v1.19.1 and it almost worked, and I was able to use my DualShock 4 at the same time, both wired and wireless. I was testing with Metroid Prime Hunters on an Samsung S25 Ultra.
However, the S-Pen seemed to be less sensitive than it should be, less sensitive than it's capable of. It's like I needed to press slighly harder than just touching it to the screen, and it would lose track of it sometimes. Is there some way to up the sensitivity? The issue doesn't seem to be system-side.
I noticed https://github.com/rafaelvcaetano/melonDS-android seemed to be the same way. I'm guessing its based on the same code?
Anyway, just some theorizing. Much appreciation for whoever may be able to do something about this!
Update! I realized just now that the lack of sensitivity for the S-Pen was now system-wide afterall. After some quick research I learned that strong magnetism can affect the S-Pen.
It was my magnetic phone clip! I took it off and the sensitivity instantly went back to normal!
Beware, y'all!
However, the issue of no S-Pen support in recent builds remains.
After some quick research I learned that strong magnetism can affect the S-Pen.
It was my magnetic phone clip! I took it off and the sensitivity instantly went back to normal!
Beware, y'all!
However, the issue of no S-Pen support in recent builds remains.
OMG dude you just saved me i have a mag phone case and this was throwing off my testing! thank you!
I have a (semi) working build with S pen implimentation. I expect to have a build in the pipe for everyone by monday, fingers crossed!
Help?! I can't even get the S Pen working with v1.19 on my Note 10 Lite??
Help?! I can't even get the S Pen working with v1.19 on my Note 10 Lite??
I will be pushing a test build this evening hopefully, along with a test build of SNES9X on my github. basic function has been working for a while on my build, and works 100% for melondsDS, but I had to refactor it so that I could properly build out for other cores.
I have not had as much time recently, but i am pivoting back to this project this week.