SDL
SDL copied to clipboard
libsdl-org
Cursor keys and return key under SDL_VIDEODRIVER=wayland with phosh+squeekboard (on-screen keyboard) not working
Cursor keys and the return key under SDL_VIDEODRIVER=wayland with phosh+squeekboard (on-screen keyboard) are sadly not working when pressed on the squerkboard keyboard. With SDL_VIDEODRIVER=x11 they work, letters and backspace also work just fine.
Versions:
- Phosh: 0.17.0
- squeekboard: 1.17.1
- SDL2: 2.0.22
No idea what most of these things are, but I would imagine it falls in line with text_input_v3, which lives in these two areas:
https://github.com/libsdl-org/SDL/blob/main/src/video/wayland/SDL_waylandkeyboard.c
https://github.com/libsdl-org/SDL/blob/main/src/video/wayland/SDL_waylandevents.c#L1634-L1641
Phosh is one of the two most common mobile desktops in use on Linux phones etc. right now. Squeekboard is just the on-screen keyboard it's usually packaged with. This bug renders all applications unusable on Phosh linux phones that may require cursor key input but otherwise happen to be usable on a phone screen (also #5899 may break SDL_Renderer output) - unless you use SDL2's X11 backend, but then touch might be broken beyond use for you, see #5924 .
Right now, using SDL2 for linux phones is rather frustrating due to the amount of regressions and bugs like this, even though SDL2 seems one of the most natural choices to target this kind of platform, especially for simple and fun 2d games since most linux phones sold right now are comparatively lower spec. SDL2 has a very simple and yet 3d accelerated API for all basic 2d needs via SDL_Renderer, it's very usable from C or C++ which is perfect for low spec fast games, it has a somewhat simple input API that even allows multi touch gestures, and it can do High DPI without too much fuss, and it's available widely on most linux distros whether they target mobile or not, and it tries to pioneer wayland support which is useful since many other game stuff is stuck on x11 but most of the new phone desktops are wayland only and xwayland is not fun. So, SDL2 would be really nice... if it worked 😂
I don't think we have any maintainers with that environment at all - it's either desktop Linux or iOS/Android with nothing in between, so that's probably what's up. If someone came in to regularly review and patch for that environment it would prevent a lot of this! I filed a request for a PinePhone to do SDL testing with and they said no, so that's about as close as I've ever gotten to having the relevant hardware.
I don't know who builds and ships these devices with these systems installed, but if they have any engineers I would let them know that there's a big gap in a really popular library that they could fill in!
The PinePhone is community-maintained as far as the OS side goes, there isn't even an official distro and some just happen to target it. It's essentially just a fairly standard Allwinner ARM64 board that happens to be a phone form factor. You could ask postmarketOS devs to help out with testing (which is the distro I use), but I imagine they are quite busy and low in number.
However, Fedora packages Phosh for x64. If you can get it to run in undocked mode (not sure how easy that is, could be as simple as toggling in the notification area) to get its phone mobile UI rather than the laptop/desktop convergence one and your laptop has a working touch screen you can probably reproduce most or all the issues I am filing with that. This environment also has one huge advantage to an actual PinePhone: you won't wait an hour for an SDL2 build, which is why I only test the distro versions usually.
(I actually have such a laptop but the kernel seems to have completely broken the touch drivers for it turning the touch screen effectively dead with nobody really minding, so that essential part sadly is missing for me. Also I'm away from home right now without the laptop, I am writing these tickets on my phone)
It's definitely worth looking into - the Wayland landscape is currently in a state where if the author does not have the compositor literally in front of their face at least once, there's about a 50% chance it'll work. So, the lab here has basically every desktop on it at the same time: A mutter environment (GNOME), kwin environment (Plasma) and at least one wlroots environment (sway). That at least covers desktop, but even with the same compositing frameworks, the mobile UIs and input devices are different enough to make it fall in that "maybe it works, maybe it doesn't" category.
The hardware is the main blocker for my lab at least; I can run simulators and nested compositors, but it won't have the real input that we seem to hit and miss on so much (and I have no touch devices at this point, the Deck will be the closest one and that's not coming until the end of the year). The others might have something, but up to this point we basically never see reviews for changes on mobile until looong after the desktop people have verified it for their tests.
Afaik Phosh uses Phoc which uses wlroots. So if your sway environment has touch then #5924 at least might reproduce there if sway also happens to support xwayland. (Or not, who knows, I wouldn't know how much XWayland support is in wlroots itself and what's on top.)
I'm optimistically adding this to the 2.24.0 milestone, hoping that someone will get hardware that we can test this on.
I just discovered the return key also doesn't work which is a way bigger problem. I used the xwayland/x11 sdl video driver up to now instead as a workaround but #5924 broke that very thoroughly. So now I can choose either no way to enter multi-line text (this bug) or all scrollbars and scroll views are unusable ( #5924 ). Needless to say neither are really workable 😭 so I'm a bit stuck. Glad for any workaround suggestions
but I would imagine it falls in line with text_input_v3
Not when it comes to things like cursor and return keys. Those are send to the compositor via virtual-keyboard and then sent to clients as regular wl-seat events.
However, since Squeekboard works with arbitrary layouts, it uses automatically generated keymaps that don't match any actual physical keyboard layouts. Some applications have troubles with that, and judging from the description this may be the case with SDL here.
There's no need for any special hardware for testing, Phosh consists of regular GNU/Linux applications that you can run on any desktop. Just launch phoc nested, run Squeekboard in it (SQUEEKBOARD_DEBUG=force-show env var may be handy) and then launch SDL app.
Also, this doesn't actually sound related to phosh/phoc at all - I'm pretty sure it will be reproducible on sway just as well.
That all makes sense to me - that said, someone else will need to look into this:
As of today I've signed onto a job that will take me very far away from Wayland development beyond review and running things through the lab when explicitly requested; I will not be writing any Wayland patches for the rest of 2022. To try and keep things organized I've added a Wayland label to everything that had Wayland on it and/or was assigned to me; all of those items are now unassigned.
(Separately: Still working on audio and PipeWire-by-default, as well as general Linux-y things, only this is on hold.)
I just ordered a PinePhone; I have no idea when it'll arrive, but it's definitely not going to land in this milestone.
I'm pretty sure it will be reproducible on sway just as well.
For the record, I've verified that now and, indeed, it's also happening under sway on desktop (and likely any other compositor + OSK combo that happens to use custom keymaps as well). Unfortunately, I personally don't know much about how SDL handles keymaps, so I'm hoping that someone with that knowledge jumps into it before I go into this rabbit hole myself...
Judging from what I've seen in that code though, I bet it's SDL making some incorrect assumption about keymaps being related to physical keyboard layouts when that doesn't have to be the case.
Here's our keyboard listener: https://github.com/libsdl-org/SDL/blob/main/src/video/wayland/SDL_waylandevents.c#L1127
Both the keymap and modifiers callbacks will be where all the related activity is.
The most interesting part seems to be https://github.com/libsdl-org/SDL/blob/main/src/video/wayland/SDL_waylandevents.c#L1009, which appears... hardcoded?
That array refers to the scancode value AFAIK. From what I remember, the key is a scancode that we're expected to translate, so we make the translation table in keymap/modifiers and then use it when passing to SDL_SendKeyboardKey. Note that this is key presses rather than text, which is handled with keyboard_input_get_text within that function. SDL_IME usually handles text but keyboard_input_get_text does have an xkb fallback that supposedly translates as well, but this is where things started getting done with trial-and-error to get it working on both GNOME and Plasma.
I believe keyboard_input_get_text works fine, it's SDL_SendKeyboardKey where things go wrong, and I don't see how can it even be supposed to work. Unless I'm missing something, it doesn't seem to consult the keymap at all and just uses a hardcoded array, which obviously fails since Squeekboard uses autogenerated keymaps with arbitrary scancodes.
For what it's worth, if my memory isn't tricking me I think the X11/XWayland code path of SDL2 had the exact same problem with squeekboard's dynamic keyboards, and I think someone similarly discovered SDL2 used some wrongly hardcoded assumptions about the keymap. So maybe there is some insight in SDL2's X11/XWayland code path and how it was fixed there? (I might be wrong though, I have almost no clue how lowlevel X11 or Wayland code works like.)
I finally got a librem5 VM configured with SDL building (note that the current version of SDL doesn't build the Wayland backend without modification due to librem shipping wayland-client 1.16 and SDL requiring 1.18), and squeekboard uses both arbitrary scancodes and keycode names, so the previous approach won't work.
I have a new idea to address this, keeping a table of control key (esc, return, arrows, etc...) mappings between xkb keysyms and SDL scancodes, which is used to dynamically map these keys, and treating everything else as a text generating key only, so they will generate text events, but not key up/down events. You won't be able to use things like WASD controls with squeekboard, but that's not really what it's meant for anyways.
Key up/down are IMHO quite important, they might be used for navigating menus that may not be fully ported to touch, keyboard shortcuts people may want to use even on a phone (I e.g. use CTRL+C sometimes), and some simpler games also use them for text input even though they maybe shouldn't. Also, I vaguely remember for X11 there was the same problem with no up/down being generated by squeekboard and other such keyboards and it was also added in as faked events somehow for compatibility I think, so not doing so for Wayland would be inconsistent. It might again be worth checking the X11 code path for how that was done
Edit: oh, I forgot the most important example,the return key for entering multiple lines may also often only be handled with key up/down I imagine.