New keyboard API for macOS

[ArturKovacs]

trafficstars

• [x] Tested on all platforms changed
• [x] Compilation warnings were addressed
• [x] cargo fmt has been run on this branch
• [x] cargo doc builds successfully
• [ ] Added an entry to CHANGELOG.md if knowledge of this change could be valuable to users
• [ ] Updated documentation to reflect any user-facing changes, including notes of platform-specific behavior
• [x] Created or updated an example program if it would help users understand this functionality
• [x] Updated feature matrix, if new features were added or implemented

This is the macOS implementation of #753 and it also brings the Android backed up to date with the new API.

This should be merged after #1788.

Side-note: the IME implementation is based on #1669 but as I don't know any language which naturally uses IME, it's difficult for me to know if what I ended up implementing is correct.

[ArturKovacs]

If I'm not mistaken, this implements all the functionality that was implemented with the previous API but I'm not a 100% sure. For this reason, this definitely needs a review from someone who can test this on real hardware. @francesca64 you are my trusty go-to for this kind of stuff 😊 But if anyone has to opportunity please test this and report back, it would help a ton!

[kjmph]

Pretty excited to see this. I tried building a release target, and saw this error:

warning: unknown lint: `broken_intra_doc_links`
   --> src/lib.rs:133:9
    |
133 | #![deny(broken_intra_doc_links)]
    |         ^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[warn(unknown_lints)]` on by default

warning: 1 warning emitted

As it is a warning, I moved on. Unfortunately, one can't build Alacritty with these changes. Do you happen to have a branch of Alacritty built against this?

[ArturKovacs]

Hi thanks for dropping by.

I tried building a release target, and saw this error

I believe you either have an older version of Rust or you are compiling on nighty because I could not reproduce this on stable.

Do you happen to have a branch of Alacritty built against this?

I don't, but @chrisduerr might be able to help you with that.

[kjmph]

I tried building a release target, and saw this error

I believe you either have an older version of Rust or you are compiling on nighty because I could not reproduce this on stable.

Sorry about that! You are correct. Updating solved this.

[ArturKovacs]

Rebased this on top of the latest state of #1788 and after the merge of the PR, merged rust-windowing:new-keyboard back into this.

[ArturKovacs]

@madsmtm could you give this a review? If you could approve this, then I'd feel reasonably confident merging it.

[mockersf]

I tried running winit_keyboard_test with this PR on macOS 11.2.2, it works fine 👍 No issue in AZERTY with the laptop keyboard or an external keyboard

One small nit I have is that while modifier keys are differentiated between left and right, if I press them at the same time the second is not.

Example with shift:

| Number | Kind   | Synth | State    | KeyCode              | Key                                        | Location | Text         | Modifiers   | Key (no modifiers)        | Text (all modifiers) |
| ------ | ------ | ----- | -------- | -------------------- | ------------------------------------------ | -------- | ------------ | ----------- | ------------------------- | -------------------- |
| 0      | Window | false | Pressed  | ShiftLeft            | Shift                                      | Left     |              |             | Shift                     |                      |
| 1      | ModC   |       |          |                      |                                            |          |              | SH          |                           |                      |
| 2      | ModC   |       |          |                      |                                            |          |              | SH          |                           |                      |
| 3      | ModC   |       |          |                      |                                            |          |              | SH          |                           |                      |
| 4      | Window | false | Released | ShiftLeft            | Shift                                      | Left     |              |             | Shift                     |                      |
| 5      | ModC   |       |          |                      |                                            |          |              |             |                           |                      |

[ArturKovacs]

Thanks a lot @mockersf for testing! I pushed a fix for modifiers key events not being reported in the case you described.

[mockersf]

Thanks! Modifiers detection works as I expected 👍

Just noticed another thing with modifiers key: they don't trigger the repeat event

• kept tab pressed for a few seconds, the repeat event is there:

| Number | Kind   | Synth | State    | KeyCode              | Key                                        | Location | Text         | Modifiers   | Key (no modifiers)        | Text (all modifiers) |
| ------ | ------ | ----- | -------- | -------------------- | ------------------------------------------ | -------- | ------------ | ----------- | ------------------------- | -------------------- |
| 0      | Window | false | Pressed  | Tab                  | Tab                                        | Standard | "\t"         |             | Tab                       | "\t"                 |
| 2      | Window | false | Rpt   20 | Tab                  | Tab                                        | Standard | "\t"         |             | Tab                       | "\t"                 |
| 2      | Window | false | Released | Tab                  | Tab                                        | Standard |              |             | Tab                       | "\t"                 |

• kept control pressed for a few seconds, no repeat event:

| Number | Kind   | Synth | State    | KeyCode              | Key                                        | Location | Text         | Modifiers   | Key (no modifiers)        | Text (all modifiers) |
| ------ | ------ | ----- | -------- | -------------------- | ------------------------------------------ | -------- | ------------ | ----------- | ------------------------- | -------------------- |
| 0      | Window | false | Pressed  | ControlLeft          | Super                                      | Left     |              |             | Super                     |                      |
| 1      | ModC   |       |          |                      |                                            |          |              | CO          |                           |                      |
| 2      | Window | false | Released | ControlLeft          | Super                                      | Left     |              |             | Super                     |                      |
| 3      | ModC   |       |          |                      |                                            |          |              |             |                           |                      |

Not sure if that's expected, and it seems logical as when I press another key with modifier, this other key will be repeated. But what should happen if I press just the modifier?

While reading the PR, I noticed a comment about ctrl + space. Is it expected when pressing ctrl + space that the event for space won't be triggered? even though I don't have multiple keyboard layout enabled on my system

[ArturKovacs]

Just noticed another thing with modifiers key: they don't trigger the repeat event

This is the native behavior on macOS. As you mentioned, this does make sense as modifier keys, by themselves, don't produce text and aren't used for navigation. (Not in regular applications at least.)

I actually assumed that this matches the behavior on Windows, but fun fact: on Windows, the Control and Alt modifiers don't produce repeat events, but the Shift and the Windows key (Super) do. There's probably an interesting piece of history behind that :D

[ArturKovacs]

Is it expected when pressing ctrl + space that the event for space won't be triggered? even though I don't have multiple keyboard layout enabled on my system

I think so, but we can test this. I have a bunch of layouts installed so I'd be happy if you could help me with this. If it's okay for you, open a new tab in your preferred browser, then press F12 and execute the following command in the console, then try pressing Ctrl+Space and see which keypresses get printed to the console.

document.addEventListener("keydown", e => console.log(e.code))

[mockersf]

then try pressing Ctrl+Space and see which keypresses get printed to the console.

only ControlLeft 👍

[ArturKovacs]

That's great, thanks!

[Kethku]

I think there is an issue here where ModifiersState reports alt as held down when using a keyboard layout where the right alt key is actually AltGr. This leads to incorrect keybindings when using the modifiers state to determine if alt is held in combination with another character.

[Kethku]

Ok I've been discussing with some folks and my comment is incorrect. The problem is that for certain keyboard layouts on macos holding option should modify the key sent via the text event. However it appears that the modification isn't reproduced in the text field of the key event

Example: For a german keyboard layout, holding option and pressing 8 should result in the character '{'. However in my testing the key I get back is just 8 with a modifier alt reported via the modifiers changed event.

I believe this represents a bug, but I could be wrong. Any insight would be greatly appreciated.

[ArturKovacs]

@Kethku thank you for reporting this, I completely missed this bug!

It should be fixed by the commit I just pushed. Test for this behaviour and potential regressions would be greatly appreciated.

[Kethku]

Edit: adding @ArturKovacs because this request is mostly targeted at you who designed the change

I will test this soon. Over all the new design is a huge improvement. One problem I'm fighting though is that as far as I can tell it's not possible to determine whether a given modifier was used in the construction of a character. My usecase is in a vim client so if the user binds the key binding ctrl-!, it's not correct to send ctrl-shift-! Because the shift was "used up". This becomes more complicated with bindings where control or alt modify the character.

I don't know if it's possible, but an api which allows you to compare the values with or without individual (not just all modifiers) modifiers would make this perfect. But that's just my two cents. Amazing improvement regardless

[ArturKovacs]

One problem I'm fighting though is that as far as I can tell it's not possible to determine whether a given modifier was used in the construction of a character.

The intended usage is that you listen to WindowEvent::ModifiersChanged events and keep track of the pressed modifiers yourself. Then you can use this in any event you want (e.g. mouse input, device events)

The reason for not putting the modifier info into the KeyEvent is to avoid having multiple sources of truth. And to avoid appending this info to a bunch of events when the application itself could instead keep track of the modifiers. Here's one of the places where this argument was formulated: https://github.com/rust-windowing/winit/issues/1124#issuecomment-558849766

[ArturKovacs]

I don't know if it's possible, but an api which allows you to compare the values with or without individual (not just all modifiers) modifiers would make this perfect.

If I understand you correctly, this is already possible:

match event {
    Event::WindowEvent {
        event: WindowEvent::ModifiersChanged(new_modifiers),
        ..
    } => {
        if new_modifiers.contains(winit::keyboard::ModifiersState::CONTROL) {
            // Do the thing
        }
    }
    _ => ()
}

[Kethku]

That's not quite the problem. Knowing whether a modifier is held is easy like you've shown. Knowing whether a particular modifier contributed to the selected character is not easy though.

Due to a quirk in neovim's keybinding code, shift-{character} is different from {character} so shift-! would be handled differently than ! even though the shift key is necessary to produce the ! character. This problem extends to other modifiers such as ctrl. I'm told that some keyboard layouts require ctrl to input given characters.

So the thing that would help is a way to tell if a given modifier contributed to the selection of the resulting character and so should be omitted from the keybinding reporting or if the modifier could have been omitted to produce the same character. I don't know if such an api is possible on all platforms so it may be a moot point but I figured I'd ask and see

[ArturKovacs]

Knowing whether a particular modifier contributed to the selected character is not easy though.

That is true, in fact I doubt it's possible on macOS or Windows. However, assuming that all active modifiers contributed to the generated character could be a useable approximation. Also note that on desktop platforms you have access to the pressed key as if there was no modifiers active, using key_without_modifiers.

If I understand correctly, you have keybindings that you want to trigger in response to the corresponding input. And you want to allow this "corresponding input" to be either

1. the exact input specified by the binding
2. and any other input where additionnal modifiers are held down which do not contribute to the produced character.

I don't understand why 2 is a requirement. For example I would not expect that my ctrl+A keybinding gets triggered when I press cmd+ctrl+A. In fact I would hope it doesn't get triggered in that case.

In any case, if you want to allow 2, then what you could do, is to check if the current input is exactly the same as the specified binding, then remove cmd from the current input and check if it matches the binding that way. This assumes that cmd is the only key which doesn't influece the pressed key but that could be a decent approximation for your use case.

[Kethku]

Interesting can you elaborate on what you mean by remove the modifier? Are you suggesting I construct a new key event somehow?

[Kethku]

I think you mean submit the keybinding to neovim with and without the modifier?

I think the conclusion I'm coming to is that a change needs to happen in neovim to handle with and without bindings the same.

I will think some more on whether that's possible. From neovim's perspective, they can't know whether a key used modifiers to produce a given character, so they have to assume that shift-! shouldn't be handled the same as ! because a given keyboard layout may input the ! without it.

I guess what I sorta need is an api to look up predicively what a given set of keys would produce. Again though that might not be possible...

[ArturKovacs]

Alright it took me a while, but I think now I understand what you want to achieve. Correct me if I'm wrong but what you want seems to be that Shift+1=! should behave the same as !=!. Again, it seems that you could achieve something like this by assuming that Shift always infulences the character (unless it's a non-charater input). In practice this would mean that you define all keybindings without Shift in neovim, and then send all input to neovim without telling neovim that Shift is held down. There may be some collision between keybindings on certain keyboard layouts, but I actually can't think of a specific example of such a collision.

I guess what I sorta need is an api to look up predicively what a given set of keys would produce.

Something similar is probably possible on desktop platforms, but I would like to merge the current API into master before we make further additions to it, as it takes a very long time to come to an agreement about and finish the implementation for such additions.

[TobTobXX]

Sooo.... is the code on this branch still a candidate for master? If yes, can you do a rebase?

(The folks from the neovide project would like to use the new keyboard API.)

Thank you!

[kchibisov]

closing as we have mono branch.