flutter
EditText box in TLHC handles resize and offset changes poorly
Is there an existing issue for this?
- [X] I have searched the existing issues
- [X] I have read the guide to filing a bug
Steps to reproduce
Filing this issue coz comments are disabled on the original one: https://github.com/flutter/flutter/issues/112712 We thought we found a way to use TLHC as a substitute for HC when wrapping a native textfield. The solution suggested by @johnmccutchan (see here https://github.com/flutter/flutter/issues/112712#issuecomment-1807704679) indeed resolves the issue related to Magnifier lense, but a new issue originated as a consequence of that fix.
Basically, due to that override, the surface gets rendered with some delay and the platform view bumps when its position is changed (in our case the inputted text bumps when a new line is added).
Please do not discontinue HC as long as a fix for this is found. We've no alternative to it and we've also invested serious money/time on this.
Expected results
We would love to use TLHC but texture-rendering must be as smooth as with HC in the above described scenario.
Actual results
Platform View bumps when its position is changed on the scene.
Code sample
Code available here: https://github.com/delfme/flutter-native-text-input-repo
The plugin uses TLHC and issue is reproducible https://github.com/collinjackson/flutter-native-text-input/blob/1ef42d30462d97a3fafc856e356af83e106fe6b6/lib/flutter_native_text_input.dart#L334
Fork the plugin and change TLHC to HC to see that no issue occurs with HC.
Screenshots or Video
Smooth with HC https://github.com/flutter/flutter/assets/53510751/90728c72-cd6f-425a-a418-d2708fa5a02e
Bumping with TLHC: note: this is a low res recording, issue is way more visible on the device. https://github.com/flutter/flutter/assets/53510751/a43183d9-6656-42d1-a845-955502838ef8
Logs
Logs
[Paste your logs here]
Flutter Doctor output
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel master, 3.17.0-17.0.pre.25, on macOS 12.6.5 21G531 darwin-arm64, locale en-IT)
[!] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
✗ cmdline-tools component is missing
Run path/to/sdkmanager --install "cmdline-tools;latest"
See https://developer.android.com/studio/command-line for more details.
✗ Android license status unknown.
Run flutter doctor --android-licenses to accept the SDK licenses.
See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.
[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.2)
[✓] Connected device (3 available)
[✓] Network resources
Could you be more specific about what the issue is? I've watched both videos multiple times and they seem the same to my eyes.
Check out this video
https://github.com/flutter/flutter/assets/53510751/ecfe8591-b61d-4192-a571-dc56db649790
Focus on “Fffg” and “G”, you will see a bump. It doesnt occur on every line. Also, as weird as it might sound, HC seems to render faster than TLHC. How that can be? Basically “Message” placeholder is shown with a bigger delay if TLHC is used.
I took a screeenshot of when this happened. It doesn't look like a rendering error exactly, everything is rendering correctly but the text input state has been reset or lost. Maybe this is a text input client state error:
FYI @justinmc
I took a screeenshot of when this happened. It doesn't look like a rendering error exactly, everything is rendering correctly but the text input state has been reset or lost. Maybe this is a text input client state error:
FYI @justinmc
That might also explain the nonsense of TLHC rendering slower than HC. We noticed that with TLHC the textfield placeholder is displayed after some time and assumed that it was due to a bigger loading time. But in light of above discovery, maybe the text is lost.
It is unclear that this isn't a bug in the plugin that you're using for the native text input field.
It is unclear that this isn't a bug in the plugin that you're using for the native text input field.
No issue with HC and text is not cleared while typing.
Also, are we really sure that the screenshot with empty textfield means that the text inside textfield is lost? Cant it also be that the entire platform view is not displayed for a fraction of sec?
We could assign a solid background color to textfield (now it is transparent) and see if it is the text being cleared or the entire textfield disappearing
The platform view embeds an EditText widget which draws the border around the text, the text, and the cursor.
When in TLHC mode we render the EditText widget into a Canvas just as normal Android does, on the same Android UI thread. The only difference is that we save the rendered canvas in a texture and draw the texture on the screen.
In the frame where the text disappears you can see the border, no text, and the text cursor positioned as if there is no text. This suggests that the EditText widget is drawing itself correctly but the text contents is cleared.
I tried to run your sample app but it doesn't compile:
Launching lib/main.dart on sdk gphone64 arm64 in debug mode...
Building with Flutter multidex support enabled.
e: /Users/johnmccutchan/t/flutter-native-text-input-repo/android/app/src/main/kotlin/dev/henryleunghk/flutter_native_text_input_example/MainActivity.kt: (6, 33): Unresolved reference: RenderMode
e: /Users/johnmccutchan/t/flutter-native-text-input-repo/android/app/src/main/kotlin/dev/henryleunghk/flutter_native_text_input_example/MainActivity.kt: (7, 16): Unresolved reference: RenderMode
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':app:compileDebugKotlin'.
> A failure occurred while executing org.jetbrains.kotlin.compilerRunner.GradleCompilerRunnerWithWorkers$GradleKotlinCompilerWorkAction
> Compilation error. See log for more details
* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 2s
Running Gradle task 'assembleDebug'... 2,654ms
Error: Gradle task assembleDebug failed with exit code 1
I fixed the compilation failure but the text is not visible (it's white on a white background?) and the sample app doesn't look anything like the videos you've attached.
After fixing the text color the text input box seems completely buggy. The cursor always jumps back to position 0 after each key is inputted.
TLHC
https://github.com/flutter/flutter/assets/53510751/4044ece2-db80-4498-a6aa-4f7b6e1eeaf7
TLHC Slow motion, I do not see text disappearing in this test but more like a delay in rendering it. Cant find correct words to describe issue
https://github.com/flutter/flutter/assets/53510751/a4529595-35df-4345-936c-0379d95dd686
HC
https://github.com/flutter/flutter/assets/53510751/ccc4f072-e733-4b36-8608-33582fdc358e
HC slow motion, no issue
https://github.com/flutter/flutter/assets/53510751/f1492994-d4c0-4f85-9930-8779f5dde50b
Platform View impl. is added directly inside the sample code. https://github.com/delfme/flutter-native-text-input-repo
How to test it:
Background color is set to red from here. This helps understand whether android EditText's text is lost or the entire platform-view is lost. https://github.com/delfme/flutter-native-text-input-repo/blob/294c630a8db70f5308016d4f1055cd7adc91ff7d/flutter-native-text-input/android/src/main/kotlin/dev/henryleunghk/flutter_native_text_input/NativeTextInput.kt#L63-L65
Switch between HC and TLHC from here: https://github.com/delfme/flutter-native-text-input-repo/blob/294c630a8db70f5308016d4f1055cd7adc91ff7d/flutter-native-text-input/lib/flutter_native_text_input.dart#L575
Type in text and press Return key to write multiple lines (as shown in above videos). The described issue is reproducible with TLHC. Issue doesn't occur with HC.
Note: I added a red background with some opacity to see whether it is 1) entire platform view disappearing, or 2) text getting lost or 3) an issue with platform-view rendering while platform-view position changes on the scene. I might be wrong, but at first glance, option 3 seems most likely.
Thanks for the updated demo. I think the issue you're observing is triggered by the platform view changing size.
Yes, basically we change both position and size. One or both is not managed properly by TLHC.
There is also that other issue about loading time. If you change a bit the sample so that the textfield is shown on a second page (after tapping a button), you will see that TLHC is slower than HC. Just focus on the placeholder loading time. There is a 1sec delay before it is displayed.
Please file a separate issue and provide a reproduction for the delayed display
Reproducible using the code sample provided in https://github.com/flutter/flutter/issues/139230#issuecomment-1833392115
flutter doctor -v
[!] Flutter (Channel stable, 3.16.2, on macOS 14.1.1 23B81 darwin-arm64, locale en-GB)
• Flutter version 3.16.2 on channel stable at /Users/nexus/dev/sdks/flutter
! Warning: `flutter` on your path resolves to /Users/nexus/dev/sdks/flutters/bin/flutter, which is not inside your current Flutter SDK checkout at /Users/nexus/dev/sdks/flutter. Consider adding /Users/nexus/dev/sdks/flutter/bin to the front of your path.
! Warning: `dart` on your path resolves to /Users/nexus/dev/sdks/flutters/bin/dart, which is not inside your current Flutter SDK checkout at /Users/nexus/dev/sdks/flutter. Consider adding /Users/nexus/dev/sdks/flutter/bin to the front of your path.
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 9e1c857886 (14 hours ago), 2023-11-30 11:51:18 -0600
• Engine revision cf7a9d0800
• Dart version 3.2.2
• DevTools version 2.28.3
• If those were intentional, you can disregard the above warnings; however it is recommended to use "git" directly to perform update checks and upgrades.
[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0-rc1)
• Android SDK at /Users/nexus/Library/Android/sdk
• Platform android-34, build-tools 34.0.0-rc1
• Java binary at: /Users/nexus/Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)
• All Android licenses accepted.
[✓] Xcode - develop for iOS and macOS (Xcode 15.0)
• Xcode at /Applications/Xcode-15.0.0-Release.Candidate.app/Contents/Developer
• Build 15A240d
• CocoaPods version 1.13.0
[✓] Chrome - develop for the web
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
[✓] Android Studio (version 2022.3)
• Android Studio at /Users/nexus/Applications/Android Studio.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)
[✓] IntelliJ IDEA Ultimate Edition (version 2023.2.5)
• IntelliJ at /Users/nexus/Applications/IntelliJ IDEA Ultimate.app
• Flutter plugin version 76.3.4
• Dart plugin version 232.10072.19
[✓] VS Code (version 1.84.2)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Flutter extension version 3.76.0
[✓] Connected device (5 available)
• Pixel 7 (mobile) • 28291FDH2001SA • android-arm64 • Android 14 (API 34)
• Nexus (mobile) • 00008020-001875E83A38002E • ios • iOS 17.1.1 21B91
• Dean’s iPad (mobile) • 00008103-000825C811E3401E • ios • iOS 17.1.1 21B91
• macOS (desktop) • macos • darwin-arm64 • macOS 14.1.1 23B81 darwin-arm64
• Chrome (web) • chrome • web-javascript • Google Chrome 119.0.6045.199
[✓] Network resources
• All expected network resources are available.
! Doctor found issues in 1 category.
[✓] Flutter (Channel master, 3.17.0-20.0.pre.10, on macOS 14.1.1 23B81 darwin-arm64, locale en-GB)
• Flutter version 3.17.0-20.0.pre.10 on channel master at /Users/nexus/dev/sdks/flutters
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision be268ebde2 (4 hours ago), 2023-11-30 22:07:26 -0500
• Engine revision c26e6ced11
• Dart version 3.3.0 (build 3.3.0-174.2.beta)
• DevTools version 2.30.0
[✓] Android toolchain - develop for Android devices (Android SDK version 34.0.0-rc1)
• Android SDK at /Users/nexus/Library/Android/sdk
• Platform android-34, build-tools 34.0.0-rc1
• Java binary at: /Users/nexus/Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
• Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)
• All Android licenses accepted.
[✓] Xcode - develop for iOS and macOS (Xcode 15.0)
• Xcode at /Applications/Xcode-15.0.0-Release.Candidate.app/Contents/Developer
• Build 15A240d
• CocoaPods version 1.13.0
[✓] Chrome - develop for the web
• Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
[✓] Android Studio (version 2022.3)
• Android Studio at /Users/nexus/Applications/Android Studio.app/Contents
• Flutter plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/9212-flutter
• Dart plugin can be installed from:
🔨 https://plugins.jetbrains.com/plugin/6351-dart
• Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)
[✓] IntelliJ IDEA Ultimate Edition (version 2023.2.5)
• IntelliJ at /Users/nexus/Applications/IntelliJ IDEA Ultimate.app
• Flutter plugin version 76.3.4
• Dart plugin version 232.10072.19
[✓] VS Code (version 1.84.2)
• VS Code at /Applications/Visual Studio Code.app/Contents
• Flutter extension version 3.76.0
[✓] Connected device (5 available)
• Pixel 7 (mobile) • 28291FDH2001SA • android-arm64 • Android 14 (API 34)
• Nexus (mobile) • 00008020-001875E83A38002E • ios • iOS 17.1.1 21B91
• Dean’s iPad (mobile) • 00008103-000825C811E3401E • ios • iOS 17.1.1 21B91
• macOS (desktop) • macos • darwin-arm64 • macOS 14.1.1 23B81 darwin-arm64
• Chrome (web) • chrome • web-javascript • Google Chrome 119.0.6045.199
[✓] Network resources
• All expected network resources are available.
• No issues found!
Added issue here https://github.com/flutter/flutter/issues/139354
This vid gives you an idea about what I mean for TLHC slow rendering. Focus on first run.
https://github.com/flutter/flutter/assets/53510751/ea7ab85f-f114-415b-a2a9-b0ef4cb13015
I spent some time looking at things this week and I have to say I'm not really sure how this would work. The gist is about what you would think: there is no synchronization of rendering when resizing the texture layer, so your choices are to either partially drop frames (current) or to rendering the old contents either stretched or positioned incorrectly (without alignment it won't be possible to determine where to place the smaller rectangle in all circumstances).
Lets go through the basic sequence of events that lead to the scenario above:
- [UI Thread] A message is sent to the platform thread with new layout information.
- [Platform Thread] A message is receieved that the platform view should be resized. a. The current image reader is discarded. A null image is added to the texture entry immediately. b. A new image reader is created. A listener is attached that will update the texture entry with a new image. c. The layout properties of the underlying view are updated. This should eventually trigger a new frame.
- [Raster thread] A message is recieved that there is a new texture entry, a forced frame is scheduled if 1) didn't already schedule a frame. a. The null entry is received. No work is done, but since the image reader has been disposed/closed, the image will be empty.
- [Raster thread] A message is receieved that there is a new texture entry. A forced frame is scheduled and the updated contents are rendered.
We're dealing with all three threads here, so 2 and 3/4 are more or less happening at the same time. For this specific example, generally the problem seems to be that 1) will trigger a frame and that frame might be rasterized after/during the old texture entry is torn down but before the new texture entry is populated.
Actually, in general, the only reason this seems to work at all is that in most cases we don't care if the Dart Ui and the Android UI are perfectly in sync. But resizing breaks this - in the same way that window resizing breaks things on desktop.
To fix this, when resizing you'd need a way to ensure that the frame scheduled by the actions in 1) aren't triggered until 3) is completed.
Another way to fix it would be to change the API such that it was not possible for the Flutter UI to treat the platform view as having been resizing until after the new frame is produced, then update the image reader disposal logic to hold onto the last image until the new image is rendered.
@jonahwilliams might your description above explain also why textfield and native keyboard animations are out of sync, producing a stuttering/jerking effect when keyboard moves up? This is a known issue that nobody was able to fix so far and impairs the UI layout where texfield is placed at the bottom of the screen (ie. chatscreen).
We are workarounding this by:
- resizeToAvoidBottomInset : false
- wrap textfield inside a Container and assign a bottom margin that will be used to move up textfield when keyboard is opened
- listen when keyboard is open and grab keyboard height via native
- update bottom margin and move up the texfield instantly, with margin value == keyboard height
This avoid the conflicts between dart and native (coz textfield is moved up in a snap), but visual result is “meh” - a bump instead of the smooth animation which occurs with native apps.
Thanks @jonahwilliams for the analysis. I will start to experiment with changes.
With https://github.com/flutter/engine/pull/49201 applied we no longer drop frames. There is still image stretching though (that is a separate issue we need to resolve).
@johnmccutchan that’s great, we would like to dismiss HC for textfield. Is the image stretch caused by the size-change? Also, do you see a loading-time improvement with TLHC, coz I dont.