Add android ime support

[lucasmerlin]

This adds Support for opening the Soft Keyboard on android and passing through TextInputState events to the application. TextInputState contains the currently edited text as string, the current selection / cursor range and a optional compose region. This enables applications to support text input on android including autocomplete and autocorrect. While the TextInputState would currently be exclusive to android, I think it should be possible to implement the same api on iOS so there is a unified api for soft keyboards.

I implemented this as a new event because I think the current Ime event wouldn't work well for mobile text input, but it might also be possible to make it work with some tweaks. There would at least need to be some way to set the current text field content, so android has some context on what to predict for autocomplete.

I implemented this api for egui here: https://github.com/lucasmerlin/egui/commit/d47677a3d49a016a176b12a15a1f6beceff94530

See it in action

https://user-images.githubusercontent.com/8009393/251972889-5ac5299f-16d2-429e-899c-6d1e8d31987d.mp4

A disadvantage of this api is, that it'd probably be difficult if not impossible to reliably edit rich text with this. The alternative would be to implement some kind of shared abstraction over InputConnection and UITextInput but these apis are a lot more complicated and for android it'd be difficult to implement it with the game-activity approach.

I'm bad at naming things, so there is probably a better name for the event than TextInputState.

Some other things missing to allow a great keyboard experience on mobile devices:

• [ ] Inset events / some way to query the current insets, so one can make sure the text field is not covered by the keyboard
• [ ] Support setting keyboard flags, for e.g. number / password / email input, enabling / disabling autocorrect, toggling capitalization for sentences

Part of https://github.com/rust-windowing/winit/issues/1823.

---

• [ ] Tested on all platforms changed
• [ ] Added an entry to CHANGELOG.md if knowledge of this change could be valuable to users
• [ ] Updated documentation to reflect any user-facing changes, including notes of platform-specific behavior
• [ ] Created or updated an example program if it would help users understand this functionality
• [ ] Updated feature matrix, if new features were added or implemented

[lucasmerlin]

Thanks for the review @kchibisov! I see how having an additional api is not ideal. I'll try to get it working with the existing Ime event.

[kchibisov]

I see how having an additional api is not ideal. I'll try to get it working with the existing Ime event.

I'm just saying that nothing stops us from extending the existing event, especially given that what you've added looks similar to it.

[lucasmerlin]

I've updated my code and added a Ime::Replace event. It looks pretty similar to the Preedit event but works differently: When this event is called the whole text (the one that was previously set with set_surrounding_text) should be replaced with the updated text.

I can't use the Preedit and Commit events, since the android game text input has no such thing as a commit. Whenever the user enters text, we get the complete updated text and a compose and selection region. One could maybe estimate whether a commit has happened based on the existence of a compose region and by diffing the updated text but I don't think that'd work reliably.

[kchibisov]

I've updated my code and added a Ime::Replace event. It looks pretty similar to the Preedit event but works differently: When this event is called the whole text (the one that was previously set with set_surrounding_text) should be replaced with the updated text.

On Wayland we have a similar concept, it's called https://wayland.app/protocols/text-input-unstable-v3#zwp_text_input_v3:event:delete_surrounding_text .

Besides, users can't rely on the replace event alone, we need Commit event when the input is done into the editor. Maybe you could adapt the technique from the Wayland protocol, which looks similar to what you're doing? Read the https://wayland.app/protocols/text-input-unstable-v3#zwp_text_input_v3:event:done it should provide an algorithm to process events, which we can adopt on Android.

The key thing here is that Commit is mandatory for the work cross platform, Preedit is not that useful, but you can mark the selection with it.

Basically you should send Commit each time the text is written into the widget, if each input implies text being written into the keyboard widget, then you should send commit for each update. The users on the other side, should reply with set_surrounding_text in response to Commit, so the Replace/delete surrounding text could work. Some clients for example, can't work with surrounding text, because they forward the input into some side channel (think terminal emulator).

However, maybe Android should Commit the text when the keyboard is being hidden or something like that?

[lucasmerlin]

I cloud in theory do something like this, basically diff the current and previous TextInputState to guess whether there should be a preedit or commit event:

let events = if let Some(compose_region) = &ime_state.compose_region {
    vec![event::Ime::Preedit(ime_state.text[compose_region.start..compose_region.end].to_string(), Some((compose_region.start, compose_region.end)))]
} else {
    if let Some(previous_text_input_state) = &self.previous_text_input_state {
        if let Some(prev_compose_region) = &previous_text_input_state.compose_region {
            vec![
                event::Ime::Preedit("".to_string(), None),
                event::Ime::Commit(previous_text_input_state.text[prev_compose_region.start..prev_compose_region.end].to_string()),
            ]
        } else {
            vec![]
        }
    } else {
        vec![]
    }
};

But I don't think this would ever work reliably, no matter how many edgecases are handled.

That's why I added the Replace event, which would replace the Commit event for android. If the application doesn't handle the Replace event, it won't have keyboard input support on android. Maybe Replace is a bad name, I guess it could be called CommitReplace or something like that. Since this event would likely be android specific, maybe it could also be a android exclusive platform extension, if there is such a thing for events?

However, maybe Android should Commit the text when the keyboard is being hidden or something like that?

That would be an idea. While typing the whole text could be sent as a Preedit. This would mean the whole text is highlighted while typing though, instead of the current compose region. Also when the keyboard is hidden the text field has already lost focus, so it would probably not get the final commit event and the text would be dismissed?

Some clients for example, can't work with surrounding text, because they forward the input into some side channel (think terminal emulator).

I see how that would be a problem. I think if you don't set any keyboard flags(disable autocomplete/autocorrect), you should reliably be able to get the last typed character from the end of the string in the Ime::Replace event. By comparing the length you could see if the user pressed backspace. But that would also be a horrible hack. I have no better idea though.

[kchibisov]

The issue with the Replace alone is that it's not clear what it tries to do, because this event doesn't map into any other platform directly. The idea behind Replace is likely fine, but what if don't set surrounding text? Could we just get a regular input and such?

Preedit is optional, only Commit is mandatory, because that's what is being inserted in the end. Surely you have some logic in egui to determine when the text is inserted? Do you just count the last state of the Replace as it? You should have some heuristic for that, I guess.

[kchibisov]

Hm, maybe we could do that differently, could you compute the difference to replace based on the surrounding text provided by user, send replace event telling it to Delete the text from the start/end of the cursor, and then you send a Commit event resulting in effectively what the Replace does?

The only issue I think is the selection or something like that, but it probably should be transparent to the user?

It just looks like the Replace event you're proposing is basically what Wayland spec suggests (2-4 points)?

1. Replace existing preedit string with the cursor. 2. Delete requested surrounding text. 3. Insert commit string with the cursor at its end. 4. Calculate surrounding text to send. 5. Insert new preedit text in cursor position. 6. Place cursor inside preedit text.

[lucasmerlin]

Hm, maybe we could do that differently, could you compute the difference to replace based on the surrounding text provided by user, send replace event telling it to Delete the text from the start/end of the cursor, and then you send a Commit event resulting in effectively what the Replace does?

Yeah, something like that could work. I could use the Delete Event to delete the current text and use commit to replace it with the newly received text. Then we would need a new event to update the cursor and compose position though, or else the cursor would always be shown at the end of the text. I could give this a go. In the end, this solution would still require a non standard event (update cursor position, haven't found anything like that in the wayland events) to be added though, so I'm not sure if this is better than the replace solution.

I tried to implement a diffing algorithm, but I haven't gotten it to work. Since the android keyboard could in theory randomly replace text independent of the cursor position, I feel like a solution that uses diffing would always have some edge cases and bugs.

@rib do you maybe have an idea how this could be solved? I think you originally also tried to use the existing ime events?

[kchibisov]

We can commit the cursor along side the commit, it's not a big deal.

You also don't have to diff actually, you could just delete everything that was before and commit new, so it's exactly like replace, you just send 2 events.

[lucasmerlin]

@kchibisov I removed the Replace event and added a DeleteSurroundingText event, this seems to work well. Is this what you had in mind?

I think it might actually also be possible to use the Preedit event instead of Surrounding Text, if I set the text to "" and the range to 0 and usize::MAX, the application should be deleting the current text, if I understand correctly? But this seems really hacky to me.

[rib]

Sorry for the delay following up here.

I do generally agree with the direction of trying to work with the Winit Ime events if we can make it work (based on adding a DeleteSurroundingText event).

It's unfortunate that the GameActivity interface we get doesn't give us more fine grained events but the way Android's InputConnection works isn't totally dissimilar to other IME protocols so I feel like we should try to avoid leaking the current limitations of GameActivity if at all possible.

It would be nice too if it's not necessary for a toolkit like Egui to have to treat Android as a special case when it comes to input method handling.

is also for soft keyboards which also generate KeyEvents

I just tried this, and none of the soft keyboards I have on my phone triggered any key events, only the TextEvent is triggered.

Curious that you see no KeyEvents at all.

I certainly see KeyEvents for at least basic virtual keyboard input (essentially just ascii characters).

We currently rely on these events in our engine while we haven't hooked up IME support yet but it seems like it may really depend on the implementation of different virtual keyboards as to what keys might generate KeyEvents.

I had actually thought more keys would be supported so we could rely on KeyEvents as a stop gap for a little longer but yeah it looks like it's not really something that can be relied on - especially if some virtual keyboards in fact don't emit any KeyEvents :/

[xorgy]

Hey @lucasmerlin just wanted to let you know that I am resurrecting this work. I'm going to split off a version without the API changes first. I'll try to make sure the commits retain your attribution. :+ )

If you're still interested in this, I'll rope you in to the review.

[lucasmerlin]

Awesome, good luck @xorgy! I'm happy to help / review, and I'm still interested in landing this. Let me know if you need access to my repo so you can push to the branch of this PR, or if there is some other way to do this. Otherwise just open a new one, and I can close this one with a notice.

[jb55]

I tried to follow the discussion, what's the next steps here? I would be happy to help since I am currently using it for my android egui app.

[lucasmerlin]

@jb55 Maybe a you could try to get basic text entry working with winit's existing api. There wouldn't be autocomplete support but it would be a first step and maybe enough for some apps.

Also the people from linebender are very interested in android keyboard support, theres a long discussion here: https://xi.zulipchat.com/#narrow/stream/351333-glazier/topic/Comparisons.20against.20Winit

I think @xorgy also looked into this but I'm not sure if he made any progress

[kchibisov]

The API could be extended, it's just certain aspects should be shared. Like the actual insert to widget part should be shared, etc, so so users will write some kind of cross platform code.

[ardocrat]

FYI with current implementation on Android you can not get input from non-english languages as keycodes, at Java side you can get them at Activity::dispatchKeyEvent callback, characters field of KeyEvent. Ideally will be to provide already converted text (also with pressed shift) at input event, like Android made for non-english language.

Also I have problem with back button when keyboard appears after call of AndroidApp::show_soft_input from Rust code and press on at any character at keyboard. Some input listener keeps to receive events and blocking others.

[xorgy]

I have this now rebased (I can't update @lucasmerlin 's branch myself) https://github.com/rust-windowing/winit/compare/master...xorgy:winit:android_ime_support

But Android 11 (API 30) and later make it more painful to get the soft keyboard to show up, as others have discovered.

[xorgy]

So I have managed to get the soft keyboard to show up, but the only way I was able to do it was by calling into the WindowInsetsController with JNI. Possible good news for #1823 is that I think we are getting keyboard events from the IME; the mixed news is that it doesn't show any complex IMEs currently (won't switch to them). Proper IMEs will likely require an InputConnection subclass or other such thing.

https://github.com/rust-windowing/winit/assets/1272018/2d7831e5-5461-4a9f-9cb3-6710df341c79

[lucasmerlin]

Closing this in favor of #3787 by @xorgy, which has a better approach

[xorgy]

And just to clarify, #3787 only opens the soft keyboard (which can send key events). Real IME on Android is looking like it will require specific support from the Activity, or at very least, the introduction of a View subclass (either when building the Activity, or possibly injected with runtime dex loading).

[MarijnS95]

or at very least, the introduction of a View subclass (either when building the Activity, or possibly injected with runtime dex loading).

Fyi this is actively beeing prototyped and worked on. Will be in xbuild first but can easily be ported to cargo-apk too.

(It can also be hacked into a Rust app with the InMemory dex class loader, but ehh :sweat_smile:)

[kchibisov]

Nothing stops winit form using the xbuild btw, it's just we had issues with it once we tried to switch the CI.

[MarijnS95]

I don't think winit must absolutely migrate to xbuild right now, just saying that I might be working on some solutions to make embedding Java with otherwise-native apps a bit nicer and easier :)