Implement Swipe to Select While Holding Shift

[JarrColl]

[WIP]

Trying to add the ability to hold shift and swipe on the spacebar to select. (and understand the codebase as I go).

Deciding if SelectionStart should always be to the left of SelectionEnd, or if SelectionStart should be the anchor.

closes #653

[Helium314]

I just want to add 2 things that are somewhat related to this PR:

• Doing anything with shift has potential to introduce bugs, due to the very special way the shift key is working in AOSP-based keyboards (it's closer to a layout switch key than to a traditional shift key). This is not meant to block you, but I want to emphasize that there could be quite a bunch of unexpected side effects, depending on how you change shift behavior.
• Changes to KeyboardActionListenerImpl are likely to require resolution of merge conflicts once @devycarol finds the time to fix the broken test in #1408

[JarrColl]

Hi, thanks for the notes, so far in my testing I haven't found anything significant related to the layout switching but some things I found so far:

• SelectionStart will need to be less than SelectionEnd as some apps expect this such as google docs.
• There are times when releaseKey is not called thus not clearing the meta_shift_on state. Such as holding to lock shift (but release key is called when double tapping to lock shift).
• some other keyboard apps have implemented this by checking if the shift layout has been manually set rather than checking if the shift key is being held. Which means swipe to select can be active when holding but also after tapping the shift key to set the shift layout manually. This could be a safer implementation than checking if shift is held if that behaviour is desirable.

[devycarol]

I've implemented a similar gesture in my D-pad homebrew. It's a challenging thing. I'll point out some development considerations:

• The code for the backspace swipe gesture can be reused (possibly renamed)
  • You can logically invert that method to get a "forward delete" selection slider.
• This may be easier if #983 is done first?

Some UX considerations:

• Holding shift triggers caps lock. It might make more sense if this behavior were associated with the caps lock mode—this would allow the gesture to be performed one-handedly.
• That might be unexpected and frustrating, but I feel having it as a config option would be rather dense.

In developing the D-pad, I've put this gesture in the navigation keyboard's Select Word button, positioned at the center of the arrow keys:

But that's not a final decision. I've also considered associating the gesture with the "selection mode" I'm developing—I really don't know, it's a pretty hard UX problem.

[Helium314]

SelectionStart will need to be less than SelectionEnd as some apps expect this such as google docs.

Fuck Google so much, seriously. In the documentation they write "Editor authors, be ready to accept a start that is greater than end.". Really great way of introducing perceived "bugs" in other (keyboard) apps. Things would be so much easier if they (and others, specifically looking at Mozilla) would bother to handle this call as documented.

his may be easier if https://github.com/Helium314/HeliBoard/pull/983 is done first?

Thanks for reminding me of this. Maybe I was too enthusiastic and pushed a bit too much work regarding more ideas on you... Anyway, it might be interesting to have the interface ready to be extended for flicks, as I plan to add the linked format with k3lp once it's done. (but that's off topic here...)

Anway, I agree it would be easier with #983 done.

[JarrColl]

SelectionStart will need to be less than SelectionEnd as some apps expect this such as google docs.

Fuck Google so much, seriously. In the documentation they write "Editor authors, be ready to accept a start that is greater than end.". Really great way of introducing perceived "bugs" in other (keyboard) apps. Things would be so much easier if they (and others, specifically looking at Mozilla) would bother to handle this call as documented.

I started testing the implementation that keeps start before end, but the fallback hardware input movement only moves selection end so it breaks over emoji. Will have to look into it more😬

The code for the backspace swipe gesture can be reused (possibly renamed)

Unrelated, but I've noticed that the backspace swipe gesture doesn't handle emoji so you can slice them in half.

Holding shift triggers caps lock. It might make more sense if this behavior were associated with the caps lock mode—this would allow the gesture to be performed one-handedly.

It doesn't trigger capslock when holding space at the same time. But I've been thinking about this, it could be quite useful for single hand use like you said and would make the logic of when to trigger selection easier too.

[devycarol]

Unrelated, but I've noticed that the backspace swipe gesture doesn't handle emoji so you can slice them in half.

Flags? Or emojis more generally?

[Helium314]

I think generally (probably every emoji that consists of more than 1 codepoint). I vaguely remember adding a fixes to some places (e.g. cursor movement), but probably not to delete swipe.

[devycarol]

Do emojis ever have more than 2 codepoints?

[Helium314]

Do emojis ever have more than 2 codepoints?

Yes I think some even have more than 10 (or maybe that was characters, but definitely more than 2 codepoints).

As for the selection start before end, I think it should be generically set in RichInputConnection.setSelection instead of before calling setSelection. (I just did that, as apparently it's necessary for some broken apps).

Related: I tried setting the shift meta state when shift was enabled manually. This allows selecting using the arrow keys and similar (some only after extra work). Though maybe it should really be limited to work only with arrow and related keys, in addition to being optional.

[JarrColl]

@Helium314 Do you know if anyone has tried and found problems with implementing the scrolling using BreakIterator to jump over unicode grapheme clusters and avoid breaking visible characters?

I've implemented it today and it appears to work quite well from my limited testing and doesn't add any performance hit over the existing codepoint counting + hardware fallback.

Changing to using BreakIterator would make swipe to select very simple to implement if there aren't any other problems.