plover
plover copied to clipboard
openstenoproject
Not working with Gnome on Wayland in Linux
Summary
Plover (latest github version) is not working on Gnome on Wayland.
Reproducing
- Use Gnome in wayland mode (default mode)
- Navigate to plover git directory and launch it by
.\launch.sh - Enable the output. However, it is not enabled.
The output of ./launch.sh might be useful.
running launch
running build_py
running build_ui
running egg_info
writing plover.egg-info/PKG-INFO
writing dependency_links to plover.egg-info/dependency_links.txt
writing entry points to plover.egg-info/entry_points.txt
writing requirements to plover.egg-info/requires.txt
writing top-level names to plover.egg-info/top_level.txt
reading manifest template 'MANIFEST.in'
warning: no files found matching 'tox.ini'
writing manifest file 'plover.egg-info/SOURCES.txt'
generating plover/gui_qt/resources_rc.py
running compile_catalog
compiling catalog plover/gui_qt/messages/fr/LC_MESSAGES/plover.po to plover/gui_qt/messages/fr/LC_MESSAGES/plover.mo
copying plover/gui_qt/resources_rc.py -> build/lib/plover/gui_qt
running build_ext
Warning: Ignoring XDG_SESSION_TYPE=wayland on Gnome. Use QT_QPA_PLATFORM=wayland to run on Wayland anyway.
xkbcommon: ERROR: Key "<CAPS>" added to modifier map for multiple modifiers; Using Control, ignoring Lock
Plover Version
Latest from github (Plover 4.0.0.dev8; commit: 2943a976f1d3ec)
System
OS: Debian Linux (kernel version: 4.19.0-2-amd64) WM: Gnome (3.30.2)
Wayland is not supported. Not sure if it provides everything needed for Plover to even support it.
Has anyone ever looked into adapting plover to act as an input method engine? Idk if it's feasible at all but that's how most of the asian language input systems are implemented.
https://blogs.gnome.org/happyaron/2011/01/15/linux-input-method-brief-summary/
Hey everyone,
I've recently been persuaded to start using sway, which is also a wayland-based window manager, so I've started to look into this issue a bit more. I thought it might be a good idea to share what I've learnt so far, so that other people can pitch in without having to start all the way from the beginning.
Quick recap: what is Wayland?
Wayland is a protocol that allows programs to talk to a compositor, which is the thing that makes all of the windows appear on your screen. Each application tells the compositor what should be drawn inside of its window (basically it just sends an image, several times per second), and the compositor then takes all of these images, places them in the right positions (i.e. where you put the corresponding window), and gives you a screen to look at. It also takes your input (keyboard, mouse, etc.), and forwards it to the currently active window.
This is also kind of where the problems for plover start, since the base protocol is very restrictive about these kinds of things. It neither allows programs to intercept keyboard input (only the active window receives these events), nor does it allow them to emulate it.
Luckily, wayland also has an extension mechanism that can allow programs to do these kinds of things, provided that the compositor supports the relevant extensions. I am saying this in advance, since both of the solutions below require such extensions. Neither is stabilized yet, and at least one of them is still undergoing active development. Unfortunately, I don't know how well gnome supports these yet, but I will be able to talk about their support in sway.
Input methods
Thanks to @0chroma for bringing this one up, this is actually a very interesting approach. Again, a short introduction:
Input method protocols are what allow a special program (often called an input method engine, or IME) to control text entry, usually by intercepting keyboard events and generating the appropriate output. To do this, it gets special capabilities that allows it to directly manipulate text, without having to emulate a keyboard. This can avoid certain problems, for example, with autocomplete in a browser's address bar. It also means that the active application, into which the text is to be entered, needs to actively support its end of the input method protocol – if it only listens to keyboard input, it won't know about any of this. However, most apps do have support for this as its kind of a requirement for Chinese, or Japanese, or Korean, etc. users.
As far as I can tell, there are currently two ways for an IME to work under wayland. The first is a system called "ibus", which is largely independent of wayland and (I think) integrated directly into gnome. I don't quite understand how it works, but it somehow allows a user to add IMEs and switch between them, and then allows the active IME to do the things described above. (I think, I really don't know a lot about ibus, please correct me if I'm wrong.)
The second approach is an extension built directly into wayland (called input_method_unstable_v2). It allows a program to register itself as an IME directly with the compositor, and then also do the things described above. The nice thing about it is that it is well documented and very easy to get started with. The not-so-nice-thing is that parts of it are still under development, sway hasn't yet released a version that supports it (since the changes were merged only a few months ago), and I don't know whether gnome supports it at all but I would honestly be a bit surprised. I also have some UX questions, like how people will be able to switch between different IMEs over this interface. (Currently, the IME has to try to register when it starts, which will simply fail if another IME is already active.)
In principle, I like the input method approach a lot. It maps very cleanly onto what plover does (take some form of user input, then translate it into normal text and output that somehow), and it avoids the slightly-annoying-but-I-guess-noone-really-cares problem with things like autocomplete. In principle, it would also allow plover to detect when the cursor moves to a different input field, so that it could reset the internal translation buffer. I guess you could even do fancy stuff like detecting if the cursor has been moved to a different location and adapting to that, but I don't expect anyone to need this.
Currently, the input method protocols are also the only way for a background program to intercept keyboard input. (Though this is still being worked on for the wayland extension.) This would mean that the keyboard machine type would definitely need one of these protocols to work.
The obvious disadvantage is that input methods can't do keyboard emulation. So things like shortcuts, cursor keys, etc. are not things that an input method can activate. The only things it can do are insert and delete text.
Keyboard emulation
Yes, there's a protocol extension for this! It's called virtual_keyboard_unstable_v1, but as far as I can tell, it's a bit more stable than the input method extension. I would guess maybe gnome has support for it too, since it's meant to be used for on-screen keyboards? I haven't looked into this too much, since the keymap hackery to get it to output full unicode felt a bit too arcane to me, but there's a tool that can output arbitrary unicode text over this interface, so this might be a good starting point for porting to plover.
I think that the advantage of this is that it maps very well to how plover currently operates, i.e. by keyboard emulation. Of course, it loses the advantages of having a proper IME, but plover currently doesn't have those anyways. And since things like arrow keys or shortcuts do need keyboard emulation, using this interface might be unavoidable. On the other hand, only the input method extension actually allows grabbing keyboard input, which is necessary for the keyboard machine.
Conclusion
Overall, it seems like it might be optimal to combine the two approaches: An input method for (optionally) capturing keyboard input and outputting normal text, and a virtual keyboard for actual keyboard emulation. In fact, this approach is also taken by one of the experimental input methods written for wayland right now (where it's necessary to forward keyboard events that the IME doesn't handle).
For now, I think I will continue to experiment with input_method_unstable_v2, since that is the simplest protocol to work with, and I just want to be able to use plover again. However, this will only help people who use compositors that support this new protocol, which I think excludes gnome. (Hopefully they'll adopt the new protocol at some point?)
Anyway, I hope that I could give a brief overview of all the things left to do to give plover full wayland support (a lot), and the compatibility problems it may face. As I said, a lot of the wayland stuff is still being worked on, and it may still be some time until it is stabilized and widely adopted. If you have any questions about any of this, please feel free to ask. (I'm sorry that I didn't take as much time to write this as I probably should have, and these things are kind of complicated and might deserve more of an explanation.)
It should also be possible to provide a evdev backend, that could be usable from X, Wayland and even the TTY, I think.
Hi, sorry for the late response, and thanks for the tip! On my system, it seems that I can't access the evdev devices as a normal user, is this expected? I've read a bit and apparently I can add myself to a special group that would allow me to access these devices without being root, which would circumvent the issue. I'm not sure if this is desirable though, wouldn't it undermine Waylands security model?
Yes, that is expected (for security reasons).
It doesn't undermine Wayland's security model, as there are many other ways to completely log everything happening when you run stuff as root (or give out permissions for everything to everyone); just Google for "Wayland keylogger" to find many great disussions about this.
Oof, I didn't know that there were keyloggers for wayland. Thanks for the info! So the plan with evdev would be to ask people "to use the evdev backend, please add yourself to the input group"? Maybe I'll start looking into evdev too, it seems like it might be simpler (and more compatible) than a normal wayland backend.
Oh, I forgot to ask: Are you interested on working on this, too? Or did you just want to make a suggestion?
That would be the idea if we decide to go with udev.
Generally I'd be interested, mostly because I want to have Plover on Wayland though, not because I'm highly motivated.
Generally I'd be interested, mostly because I want to have Plover on Wayland though, not because I'm highly motivated.
Okay, yeah, same for me :D
That would be the idea if we decide to go with udev.
Wait, did you mean evdev? Or is udev also important somehow?
Okay, no problem! I don't know a lot about this, that's why I wanted to be sure. Thanks :)
Can you hook into libinput or does going lower level with evdev make more sense? Working with libinput would cover Wayland and Xorg with a consistent interface as well maybe.
https://wayland.freedesktop.org/libinput/doc/latest/what-is-libinput.html#libinput-and-wayland
https://wayland.freedesktop.org/libinput/doc/latest/what-is-libinput.html#libinput-and-x-org
Hmm, so I skimmed the documentation very briefly and I didn't see anything about hooking into libinput. (evdev just provides files you can read/write from, if I remember this correctly.) Would there be any advantages to looking into libinput more deeply?
After reading through a little more, I think you are write that if you are attempting to make a "virtual" keyboard interface you'll likely need to talk to evdev or a /dev/eventN device directly as libinput is more about enumerating the capabilities of devices or correcting for "quirks" of touchpads etc.
Oh, thanks for asking, I probably would have forgotten about this on my own. So, there's some code I wrote a few weeks ago, which will basically make this work if you're using sway and an external stenotype (i.e. not the keyboard machine because implementing keyboard grabbing is kind of involved apparently).
The code is still unpublished because there are a couple of problems with it.
First, it needs a library to interact with wayland. The library that seems most mature here is pywayland, but that only provides v1 of the input_method protocol, but we want v2 (I don't even know if sway supports v1). It also requires libwayland to work, which is probably fine but is also the kind of thing that feels like trouble. There are also two other libraries (see sde1000/python-wayland and sapoturge/python-wayland) which can both work directly off of the wayland protocol files, which means you can just hand them the protocol file for input_method_v2 and everything should work. I hadn't found the first one yet when I started this, and I wasn't super happy with the second one, so... I decided to spend a day and write my own. This works for me, right now, but is obviously not shippable. The sde1000/python-wayland library might improve the situation a bit, but it isn't published anywhere so you would still have to ship it along with plover.
The second problem is that, in general, the entire implementation is currently more in the style of "I would like to be able to use plover again, right now", and it does some slightly confusing things that are okay for me but that I probably wouldn't be comfortable with other people using.
I haven't done any work on the evdev side of things, apart from looking into it briefly. For wayland, even if the problem with the interface library is fixed, someone would still have to sit down and implement keyboard grabbing, which last time I checked was still being worked on in sway. (See here for a library that actually uses it.) Doing this seems kind of annoying, and I don't think I will have the energy for it in the next few weeks. And even then, I don't think it would work for anyone without an appropriately patched sway. (i.e. basically no one, I guess.)
In the next few days, I can look over my current code and maybe add a bit of text on the pitfalls that it currently has and then push it to my fork, just in case someone is in the same position as me and just wants something that works, under sway, if you have an external stenotype. But I want to be clear that I don't think the current code is going to be particularly helpful in actually solving this issue in general, there's just too many problems right now. I'd guess that evdev really is going to be the way forward here, and maybe I'll start looking into that a bit more when I find the time, or someone else does it.
Just wanted to mention that UHID seems like it's probably the correct solution for creating a proper virtual keyboard. I don't see any python wrappers for it, however.
Thanks for the suggestion! I just took a brief look at the documentation, and it seems to be fairly low-level. (You would have to emulate the HID events sent by a keyboard, right?) Do you think that there's an advantage to using this instead of evdev?
Ah, I was planning on getting started with steno over the holidays, but I run on wayland (gnome)... :-(
I guess this means I may have some time to help on this issue? We'll see!
@antoniusf
Unfortunately, I don't know how well gnome supports these yet, but I will be able to talk about their support in sway.
I think neither of these will work in gnome because they're both wlroots extensions (for now). I'll do a bit of reading up on ibus and the v2 input method protocol, and see if there's a reasonable chance of abstracting over them both.
Sorry to hear that this has put a roadblock to your steno learning? :/
I look forward to hearing what you find out about ibus! (Don't stress though, it's totally okay if nothing comes back.)
@antoniusf any chance you can share your rough code? would be useful to understand what kinds of changes had to happen. I'm unfamiliar with basically every aspect of this issue, so any connect would help :-)
Hmm, I'm not sure I want to publish the full code (due to the hacky wayland interface library I mentioned above, I should probably switch to something better), but I can show you the code that does the important stuff? If having the full thing (something you could run yourself under sway) is important to you, I can try to make it publishable in the next few days.
Context (as far as I understand it):
The file that's important for this is plover/oslayer/keyboardcontrol.py. It checks which OS you're using and then imports an OS specific file that implements two classes, one for capturing keyboard input and one for emulating keyboard output. It then subclasses them and makes them available to the rest of plover.
In principle, I would implement the wayland interface in the same way, by making a separate file for the wayland interface, just like there is one for the X interface right now, and then use some method to dynamically decide which to use.
The reason why I'm explaining all this is that the current version doesn't do it. All I did was modify keyboardcontrol.py to only support wayland, because I mainly wanted to get wayland support working and not think about how and under which conditions to dispatch to the wayland version instead of the one that uses X. But in a proper, mergeable implementation, that (I think) is what you'd want.
I don't think I can really explain wayland in this space, but if you're actually trying to understand the code, you may find the specification for the relevant part of the protocol useful, along with this section on basic principles of the wayland protocol.
I'm not sure how useful it is for you to read all this, since there already is a working implementation for this interface, and it's also the thing that won't help you if you're on gnome. ibus (ie what gnome uses) is totally separate from wayland, and I'm not sure that knowing how all of this works in wayland will help you with ibus. The wayland spec I linked above might help you understand how input methods work in principle (it did for me), but I'm not sure how accessible it is if you're not already familiar with how wayland works.
All this being said, here is what I currently have in my keyboardcontrol.py, to make keyboard emulation work (in parts) in wayland: https://gist.github.com/antoniusf/e78b2ffc6ae0e4bee7b8c63148655162
I hope that what I've written made sense to you, and that it wasn't too much. Otherwise, feel free to ask more questions.
@antoniusf meant to say thanks for the pointers. I don't have time to poke around at the moment, but I really appreciate the in-depth explanation!
@antoniusf have you made any progress on this? I'd like to give plover a try, but this is a blocking issue for me, so I want to see what I can do to help out.
From what I understand you have some code that is working in wlroots based compositors (but not other wayland implementations such as gnome like the original issue is about) with a few major limitations, some of which will need to be fixed in the compositors.
Having build, and I assume used, this implantation for a while now how well does it work? Is it something that could get merged with some work, or would I be better off working on an evdev interface? Is evdev potentially a better solution because it will support more environments, such as gnome, without needing changes to those projects?
Hi! I'm sorry to say that I haven't worked on this since then. On the state of the wayland implementation: I'm not sure if you've read this yet, but if not, I think that this comment gives a pretty good summary on that. I think the one thing that that comment doesn't mention is that keyboard shortcuts also don't work currently, which would also require quite a bit of extra work. (Input methods can only input text, not press random keys on the keyboard.) Here's a TLDR of the limitations on the current wayland implementation:
– only works with external stenotypes. Anything that is, or emulates, a computer keyboard won't work. – no keyboard shortcuts – some minor ux issues where text sometimes gets lost if you switch away from the window you're writing to. This should be relatively easy to fix though. – custom wayland interface library. This is not mergeable in its current state and would have to be replaced with another, properly maintained library.
Given how much work proper wayland support would be, and that it doesn't seem to me, right now, that gnome will implement the necessary wayland extension (since they have their own system), evdev is currently my preferred approach and what I would work on (if I was working on this).
uinput, I suppose? I'm not very familiar with that library.
I don't think it supports Unicode however.
It would be easier (if you know Python) to write an output plugin (although Plover doesn't support output plugin in the current version.)
I made something similar: https://github.com/user202729/plover-uinput-output (requires root privilege, and completely unsupported!)
