Switching windows causes modifiers to reach an inconsistent state

[mtlynch]

Description

If a user pushes a modifier key in the TinyPilot web UI, holds it down, switches windows, and releases the modifier key, TinyPilot misses the key release, so the backend continues treating the modifier key as pressed.

What's the behavior that you expect?

TinyPilot treats switching windows the same as releasing all pressed keys.

What's happening instead?

TinyPilot thinks that the modifier keys are still pressed.

What are the steps to reproduce this behavior?

1. Open TinyPilot in Chrome
2. Hit Alt+Tab key to switch windows
3. Release all keys
4. Return to TinyPilot browser window

Workaround

Press the "stuck" modifier key upon returning to TinyPilot to reset the key state.

Screenshots

N/A

Logs

N/A

[mtlynch]

My intended fix is to use the window.onblur event to iterate through the currently pressed keys and send key release events for anything that's currently pressed to reset the key state.

[ComputerDoktor]

perfect.

[db39]

Another user has requested for this bug fix to be implemented because it impacts the keyboard's state when they alt + tab to different windows.

[db39]

Another user has responded in this thread requesting this bug fix (specifically for ALT+TAB on Windows).

[AltF5]

Additionally this also impacts the Windows key

[jdeanwallace]

I'm struggling to recreate this issue on macOS <-> Ubuntu with a TinyPilot device running Pro 2.5.0, 2.5.2. However, I definitely do remember experiencing this issue in the past.

Any possibility this might have been fixed in the meantime? However, I do see the last report was 3 days ago.

@db39 - Do you perhaps know what version number of TinyPilot software the users were running?

We might need to assign this task to someone with a Windows machine to recreate & fix.

[mtlynch]

@jdeanwallace - You should be able to reproduce this on a Linux target. You can use evtest to print out raw keyboard inputs.

To reproduce

1. Install evtest on the target with sudo apt install evtest
2. Open the terminal on the target through TinyPilot
3. Run sudo evtest /dev/input/event0
4. Try typing a key
5. If you see evtest print output in response, continue to step 6. If not, go to step 3 and try the next eventN (on mine, the TinyPilot keyboard input was event3)
6. Hold down the Control key
7. Hit Alt+Tab to switch to another window

Expected

Target computer sees that keys are no longer pressed when the user switches focus to a non-TinyPilot window

Actual

Target computer continues seeing the Alt+Tab keys pressed down and never shows them as being released until the user returns focus to the TinyPilot window.

[mtlynch]

Here's a video demo of what evtest looks like:

https://user-images.githubusercontent.com/7783288/217077484-235cdc8d-c5fb-4fe0-8dff-d3714bc5521a.mp4

[jdeanwallace]

Thanks @mtlynch, that did the trick! I am able to reproduce the issue now.

[jdeanwallace]

Small findings update:

The browser doesn't detect when the alt-tab/cmd-tab menu is actually showing. It only detects when you change applications and therefore fires the blur/visibilitychange event.

So if we do use the blur or visibilitychange events to trigger a key release (as previously suggested), users will still experience the bug if they use alt-tab/cmd-tab to "change" to the same application:

https://user-images.githubusercontent.com/6730025/217255508-80e02f7b-fdfa-4951-8264-8ef77257461d.mov

[mtlynch]

Oh, interesting!

I think even if we can reduce the bug to only happen when you Alt+Tab back into the original window, that's still a decent fix. It's probably like 98% of the time, they'll Alt+Tab to a different window.