Input handling seems inconsistent across terminals

[kmarius]

I'm still trying to get the new input system to work. On foot and kitty input events are generated when presing modifier keys, e.g. shift. I am not sure how to map subsequent keypresses to the actual key. For example shift+2 generates two events (plus release events) containing ids for shift and 2. On xterm, st and in tmux a single event is generated for ". This can be observed with notcurses-input which will always print uncapitalised letters in kitty.

[michaelsbradleyjr]

The following might not be helpful to you, it's just what I've learned from exploring the input system, and I thought sharing might be helpful to others who are trying to figure it out and end up reading this issue:

General

Only in terminal emulators that implement kitty's keyboard protocol will you get multiple events per key, e.g. when pressing a successive values for evtype of ncinput would be NCTYPE_PRESS then NCTYPE_RELEASE, but in other terminals you would only see NCTYPE_UNKNOWN.

Note that in v3.0.5, the alt, shift, and ctrl fields of ncinput have been marked as deprecated.

Also note that in v3.0.5 (per a7a47f8e0c53561f31a5c2a79c6d748e879f0ca6) the family of functions for working with the modifiers field of ncinput has been expanded.

Tying that all together...

In a terminal that supports kitty's keyboard protocol, you can detect ctrl+alt+a: there will be an NCTYPE_PRESS event for A (note it's capitalized) and modifiers will be 6 (ncinput_ctrl_p and ncinput_alt_p would return true).

In one that doesn't (I'm trying in iTem2 and Terminal.app on macOS), you can detect ctrl+a but not ctrl+alt+a: when pressing either combination, there will be an NCTYPE_UNKNOWN event for A and modifiers will be 4 (ncinput_ctrl_p would return true).

So in your programs, if you want to have a richer set of keybindings for some terminals but fully support all terminals, then you'll need an alternate/fallback set of keybindings and/or another way to trigger actions (maybe from a menu).

Possible Inconsistency

If we bring the shift key into the mix, perhaps there is an inconsistency, though I wonder if it's much different than the general case of handling detection in kitty-compatible terminals differently from other terminals.

When shift+2 is pressed in a kitty-compat terminal, there will be an NCTYPE_PRESS event for 2 and modifiers will be 1. When shift+a is pressed, there will be an NCTYPE_PRESS event for A and modifiers will be 1.

But in other terminals, when shift+2 is pressed, there will be an NCTYPE_UNKNOWN event for @ and modifiers will be 0. When shift+a is pressed, there will be an NCTYPE_UNKNOWN event for A and modifiers will be 0.

Maybe it would be more consistent if in the kitty-compat terminal, when shift+a is pressed, there would be an NCTYPE_PRESS event for a (note lowercase) and modifiers would be 1?

For what it's worth, to explore the input system, I've primarily used a program that I wrote, similar in purpose to notcurses-input, which makes use of the Nim wrapper I'm working on:

https://github.com/michaelsbradleyjr/nim-notcurses/blob/version_3_revamp/examples/poc/cli2.nim

The output is a bit verbose but I've found it useful for following exactly what happens with various key and key-combo presses.

[kmarius]

Thank you for the comprehesive write up. I figured it had to be some new protocol as from my knowledge those things were not possible in "old" terminals. I find the shift+2 case more interesting. Is the application supposed to know how to map these events to keys? Clearly on your keyboard it should resolve to @ but to " on mine.

The simplest solution from my point of view would be an option for notcurses to fall back to legacy mode even on kitty-compatible terminals. It would also be great to have an option somewhere in the middle, meaning that all events are generated as if in a non-kitty terminal so that they can be dealt with uniformly (no matter the terminal), but with the possibility to make use of e.g. ctrl-alt-a if the terminal allows it.

[michaelsbradleyjr]

I find the shift+2 case more interesting. Is the application supposed to know how to map these events to keys? Clearly on your keyboard it should resolve to @ but to " on mine.

That's a really good question.

In the output of my cli2.nim program I get this sequence of reported inputs when pressing and releasing shift+2:

press any key, q to quit

event : Press
point : 1115171
key   : LShift
utf8  : None[string]
input : (id: 1115171, y: 0, x: 0, utf8: ['\x00', '\x00', '\x00', '\x00', '\x00'], alt: false, shift: true, ctrl: false, evtype: NCTYPE_PRESS, modifiers: 1, ypx: 0, xpx: 0)
 
press any key, q to quit
 
event : Press
point : 50
key   : None[Keys]
utf8  : 2
input : (id: 50, y: 0, x: 0, utf8: ['2', '\x00', '\x00', '\x00', '\x00'], alt: false, shift: true, ctrl: false, evtype: NCTYPE_PRESS, modifiers: 1, ypx: 0, xpx: 0)
 
press any key, q to quit
 
event : Release
point : 50
key   : None[Keys]
utf8  : 2
input : (id: 50, y: 0, x: 0, utf8: ['2', '\x00', '\x00', '\x00', '\x00'], alt: false, shift: true, ctrl: false, evtype: NCTYPE_RELEASE, modifiers: 1, ypx: 0, xpx: 0)
 
press any key, q to quit
 
event : Release
point : 1115171
key   : LShift
utf8  : None[string]
input : (id: 1115171, y: 0, x: 0, utf8: ['\x00', '\x00', '\x00', '\x00', '\x00'], alt: false, shift: false, ctrl: false, evtype: NCTYPE_RELEASE, modifiers: 0, ypx: 0, xpx: 0)
 
press any key, q to quit

It's not clear to me how one would go from this input in the sequence:

event : Press
point : 50
key   : None[Keys]
utf8  : 2
input : (id: 50, y: 0, x: 0, utf8: ['2', '\x00', '\x00', '\x00', '\x00'], alt: false, shift: true, ctrl: false, evtype: NCTYPE_PRESS, modifiers: 1, ypx: 0, xpx: 0)

To the character that corresponds to a particular keyboard layout, e.g. @ (US) or " (UK).

There may already be a way to do it in notcurses, I don't know the whole api, yet. I've been working my through it by porting src/poc/*.c to Nim, and wrapping needed functions in the process.

Or, it may be that in a context where you're interested in detecting a key-combo, you really don't care about the character mapping, i.e. it's enough to know that it was shift+2 and that should trigger some action in your program, as opposed to printing a specific character in the terminal.

[dankamongmen]

sorry, i've been wrapped up in #2573. i can finally look at this now =].

[kmarius]

Any news on this, or a workaround? Is it possible to simply disable the kitty protocol? I would love to upgrade from 2.4.1.

[drewt]

Three solutions come to mind:

1. Provide a way for the application to opt out of the kitty keyboard protocol.
2. Enable alternate key reporting and report the shifted key to the user instead of the unicode-key-code.
3. Same as (2), except add members to struct ncinput to store the unicode-key-code and base layout key so that they are also available to the user (this would be an ABI change).

(2) would require handling these additional escape sequences in build_cflow_automaton:

"[\\N:\\Nu"               // [<key>:<shifted>u
"[\\N::\\Nu"              // [<key>::<base>u
"[\\N:\\N:\\Nu"           // [<key>::<shifted>:<base>u
"[\\N:\\N;\\Nu"           // [<key>:<shifted>;<modifiers>u
"[\\N::\\N;\\Nu"          // [<key>::<base>;<modifiers>u
"[\\N:\\N:\\N;\\Nu"       // [<key>:<shifted>:<base>;<modifiers>u
"[\\N:\\N;\\N:\\Nu"       // [<key>:<shifted>;<modifiers>:<event-type>u
"[\\N::\\N;\\N:\\Nu"      // [<key>::<base>;<modifiers>:<event-type>u
"[\\N:\\N:\\N;\\N:\\Nu"   // [<key>:<shifted>:<base>;<modifiers>:<event-type>u

It's unclear to me whether kitty would report legacy functional keys with alternate key codes, but if so then those escapes would have to be handled as well.

[kovidgoyal]

You can see exactly what kitty will report (when all enhancements have been enabled) by running

kitty -o clear_all_shortcuts=yes kitty +kitten show_key -m kitty

then press the keys you are interested in. And yes in general you will get the CSI ~ and CSI [ABCDEFHPQRS] forms with all the enhancements including shifted and alternate keys. The ~ and capital letter forms are unfortunate but needed for legacy compatibility.

You can see code to decode these keys robustly in the kitty source in both python and Go (from which it should be easy to adapt to C). Look for the functions decode_key_event() (python) and KeyEventFromCSI (Go)