List of keyboard-related aspects that need to work on iOS (in Collabora Office app, browser, or Nextcloud app)

[tml1024]

There has been a large number of bug reports (in the issue trackers and bugzillas previous used for this software) about the keyboard in the iOS app. This is an attempt to get a concise and maintainable list of all the various ways the different kinds of keyboards can be used.

1. There are four kinds of keyboards:

• On-screen keyboards. (Not sure if third-party on-screen keyboards is a thing on iOS, or if only Apple's own built-in on-screen keyboards (for different input languages) are relevant.)
• Electrically through the "Smart Connector" and thus also physically attached hardware keyboards for iPad. Apple's own Smart Keyboard Folio and Magic Keyboard for iPad, and third-party ones
• USB keyboards attached either directly to the USB-C connector on current iPads, or through a dongle ("Camera Connector Kit", I think it is called) to the Lightning connector on older iPads or iPhones. No idea whether anybody actually uses such an external keyboard with an iPhone, and whether it actually works or not
• Bluetooth keyboards

2. The app runs on

• iPad
• iPhone

The on-screen keyboards are (obviously) of different size on those, and also their functionality is slightly different. On iPad, there are extra "buttons" for things like pasting, not on iPhone.

3. There are multiple different kinds of locations in which (the cross-platform parts of) the app expects keyboard input. These might (or might not) be handled differently in the JavaScript code and have different quirks.

• In the text contents of text documents
• In cells in spreadsheets
• In the formula bar in spreadsheets
• On slides of presentations
• In controls in tunnelled dialogs
• In comments of text documents
• Possibly more

4. There are different types of keyboard input that in some cases need to be handled differently

• Letters and other printable characters directly present on the keyboard
• Generic platform shortcuts like ⌘-C for Copy and ⌘-V for paste
• LibreOffice-specific shortcuts (on hardware keyboards) like ⌘-B for Bold. I have no idea whether we even want to support such, and whether we do it in web-based Collabora Online.
• On-screen keyboard buttons (visible at least on iPad) for Copy and Paste
• Arrow keys
• Characters in scripts that use what on desktop platforms are called IMEs (CJK etc)
• Emojis, especially non-BMP ones
• More?

Ideally, after each change related to keyboard handling, something like 4*2*6*6 = 288 combinations would need to be tested to verify that no regressions have been introduced. In reality, the number of possible combinations is smaller, sure, but in any case, several dozens of combinations.

One more thing. Unless you specifically restrict any change to have an impact on the iOS app only, or are very sure that you didn't break anything else (haha), you would ideally also need to change how the behaviour changes (or doesn't change):

• In Safari on iOS against a "make run" Collabora Online server that includes the change
• In the Nextcloud app on iOS, against a Nextcloud server that uses a Collabora Online that includes the change
• In various desktop browsers on various platforms against "make run"
• In various browsers on Android
• In the Android app
• In the Nextcloud app on Android

Currently I am struggling with issue #399 , where fixing that in one way then breaks functionality in other locations in the above four-dimensional space.

[tml1024]

@nikslor , if you have suggestions on how to improve the matrix above, please comment.

[nikslor]

Hi @tml1024 thanks A LOT for this summary!

I hope to give some usefull input and keep it short initially; happy to go into more details on the journey to find a good match of what we specifically test / don't specifically test.

iPads are more prone to input issues than iPhones

My use case is mostly about "iPads". I believe, that we have no tunneled dialogs on the phone, so we only have HTML input fields there (where input issues are mostly not a problem) and also close to no hardware keyboards are involved in the phone scenario. Therefore I propose to focus on the iPad use case when we talk about "input issues".

The on screen keyboard has less priority than the hardware keyboards

In the "iPad" use case I believe most users will use a hardware keyboard when they actually work on a document. I therefore would call the on screen keyboard a second priority. Users who just have the on screen keyboard will probably use the device mostly to "consume" documents, not "create" documents.

"Smart Connector" keyboards and Bluetooth keyboards are more important than USB keyboards

USB on the iPads is quite new; only few people use the USB keyboards at the moment

Emojis are more important than keyboard shortcuts (like e.g. cmd+b)

Users want to use Emojis, but many users don't use/know keyboard shortcuts.

The cursor keys are important

User use the cursor key to position the cursor in an input field or to navigate cells. It's an important tool and it would be good if they work well.

The "helper bar" at the bottom is annoying

The "helper bar" that pops up from the bottom (offering undo/redo, copy/paste, word prediction) is annoying and I believe it should just go away.

We should get rid of the "tunneled dialogs" (also the sidebar)

Most of the focus issues and input problems happen in the tunneled dialogs and the sidebar. I tend to believe, that we should use the same approach as in the notebookbar to get rid of the current sidebar (which I would call a "tunneled dialog" as well) and use HTML/JS there too. This eliminates input problems in a "top level" dialog which the users often use. The dialogs further down in the stack (e.g. paragraph style) will remain; but I believe, they are not used that often and can be tackled later on.

@tml1024 happy to be challenged on this; let me know, where you don't agree and where I could / should provide more information.

[tml1024]

But on the other hand, for another important group of users (can't go into details) there is never any mention of iPad, iPhone is all they care for, and the on-screen keyboard.