[Wayland] Onscreen keyboard obscures TextField

[psstoyanov]

Unlike on Android or iOS, the TextField is obscured by the onscreen keyboard.

This behavior has been observed with both Phosh and Plasma Mobile.

Example Flutter project

import 'package:flutter/material.dart';


void main() {
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      theme: ThemeData.dark(),
      debugShowCheckedModeBanner: false,
      home: Scaffold(
        body: MyWidget(),
      ),
    );
  }
}

class MyWidget extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return Column(
      mainAxisAlignment: MainAxisAlignment.center,
      children: <Widget>[
        Expanded(
          child: Container(),
        ),
        Text('Example with TextField at the bottom'),
        Expanded(
          child: Container(),
        ),
        TextField(
          decoration: InputDecoration(
            border: OutlineInputBorder(),
            hintText: "Enter text here",
          ),
        ),
      ],
    );
  }
}

Something interesting:

After I've setup squeekboard on Wayfire I've noticed that once the keyboard is opened the programs were resized. Afterwards, I've checked on Plasma Mobile and that was the case as well (Bhushan confirmed the kwin behavior on the matrix channel). I'm still to double-check if Phosh does the same but it could be the same behaviour (the ticket for virtual keyboard support on Wayfire does mention Phosh).

It looks like this ticket is closely related to #6 and a resize would indeed resolve most instances where an input field would be obscured (like the input field in a messaging program).

[HidenoriMatsubayashi]

Thank you for reporting a bug!

[HidenoriMatsubayashi]

I found out that we need to set inset parameters such as physical_view_inset_top in the FlutterEngineSendWindowMetricsEvent API of embedder.h. See: https://github.com/sony/flutter-embedded-linux/blob/master/src/flutter/shell/platform/embedder/embedder.h#L576

However, the biggest problem is that I have no ideas how to detect the position of the on-screen keyboard.

[psstoyanov]

@HidenoriMatsubayashi , I don't believe this is possible at the moment.

I can see this ticket on PlasmaMobile which has some insights and possible hack to improve the behaviour for Qt apps: https://invent.kde.org/teams/plasma-mobile/issues/-/issues/23

Tracking https://gitlab.freedesktop.org/wayland/wayland-protocols/-/issues/39 seems the best approach and hopefully text-input-unstable-v4 will implement such a method.

If I am reading correctly the PlasmaMobile ticket, to avoid stutters the animations are minimised with Qt apps (resizing is sent multiple times while the keyboard is opened?). However there are still questionable behaviours when the application is in fullscreen mode and the proposed method doesn't work with apps thar aren't using Qt.

As far as I understand it, currently the compositors are resizing the client apps (so in this case the embedded Flutter app) to allow space for the keyboard. This would fall towards implementing resizing on ticket #6

When it comes down to Weston and it's keyboard - I'm not sure if they utilize similar behaviour like wlroots or Kwin forcing a resize of the client app.

[HidenoriMatsubayashi]

I don't believe this is possible at the moment.

Yes, I think so, too. Currently, there is no way to fix this issue using Wayland protocols. Thus, we need to watch it.

[psstoyanov]

Hi, It's been awhile since I've checked here.

I can see there have been quite a few commits that are affecting this ticket and possibly it should be closed.

I can see both Phosh && Plasma Mobile resizing the window once the software keyboard is opened (possibly https://github.com/sony/flutter-embedded-linux/pull/184 && https://github.com/sony/flutter-embedded-linux/pull/225 enabled this functionality?)

Today or tomorrow, I will be making a couple of screencaptures with both environments. Just need to update my test Flutter example to lose focus when tapped outside of the TextField and observe if the resulting window is restoring it's original size once the keyboard is closed.

[HidenoriMatsubayashi]

possibly https://github.com/sony/flutter-embedded-linux/pull/184 && https://github.com/sony/flutter-embedded-linux/pull/225 enabled this functionality?

FYI, window decoration can be enabled only when we use -d option.

[psstoyanov]

Yes, I was looking at the flags when checking again https://github.com/sony/flutter-elinux/issues/74 (as -k is required for the onscreen keyboard) but I don't want to add more information outside of the scope of this ticket for the onscreen keyboard :slightly_smiling_face:

[psstoyanov]

Alright! First video incoming!

Sample code used for testing

Example main.dart

import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return GestureDetector(
      onTap: () => (){FocusManager.instance.primaryFocus?.unfocus(); print("test");},
      child: MaterialApp(
        theme: ThemeData.dark(),
        debugShowCheckedModeBanner: false,
        home: const Scaffold(
          body: MyWidget(),
        ),
      ),
    );
  }
}

class MyWidget extends StatelessWidget {
  const MyWidget({Key? key}) : super(key: key);


  void increment() {
    
  }
  @override
  Widget build(BuildContext context) {
    return Column(
      mainAxisAlignment: MainAxisAlignment.center,
      children: <Widget>[
        Expanded(
          child: Container(),
        ),
        ElevatedButton(
          onPressed: (){increment; FocusManager.instance.primaryFocus?.unfocus();},
          child: const Text('Increment'),
        ),
        const Text('Example with TextField at the bottom'),
        const TextField(
          decoration: InputDecoration(
            border: OutlineInputBorder(),
            hintText: "Enter text here",
          ),
        ),
        Expanded(
          child: Container(),
        ),
        const TextField(
          decoration: InputDecoration(
            border: OutlineInputBorder(),
            hintText: "Enter text here",
          ),
        ),
      ],
    );
  }
}

Test setup with Plasma Mobile

Manjaro ARM PinePhone Plasma Mobile (some difficulties atm to get a screencapture but soon :slightly_smiling_face: )

https://github.com/manjaro-pinephone/plasma-mobile/releases/

-

flutter-elinux doctor

flutter-elinux doctor
Flutter 3.0.5 • channel unknown • unknown source
Framework • revision f1875d570e (5 weeks ago) • 2022-07-13 11:24:16 -0700
Engine • revision e85ea0e79c
Tools • Dart 2.17.6 • DevTools 2.12.2
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel unknown, 3.0.5, on Manjaro ARM 5.19.1-1-MANJARO-ARM, locale
   en_US.UTF-8)
[✓] eLinux toolchain - develop for embedded Linux devices
[✗] Chrome - develop for the web (Cannot find Chrome executable at
   google-chrome)
   ! Cannot find Chrome. Try setting CHROME_EXECUTABLE to a Chrome executable.
[✗] Linux toolchain - develop for Linux desktop
   ✗ ninja is required for Linux development.
     It is likely available from your distribution (e.g.: apt install
     ninja-build), or can be downloaded from
     https://github.com/ninja-build/ninja/releases
[!] Flutter IDE Support (No supported IDEs installed)
[✓] Connected device (3 available)
[✓] HTTP Host Availability

! Doctor found issues in 3 categories.

• To execute with osk enabled:
  • cd <project_main_dir>/build/elinux/arm64/debug/runner
  • ./<project_name> -b ../bundle -k

Plasma Mobile video capture

The video itself (will make a new with better lighting tomorrow):

https://user-images.githubusercontent.com/3979134/185764853-07121118-3fbe-40d3-a76e-b7310bb224da.mp4

Plasma Mobile results

• Rotation works as expected :heavy_check_mark:
• The window is resized as expected in accordance with the wayland compositor behaviour when the keyboard is opened/ closed :heavy_check_mark:

The only odd thing is that it doesn't seem to loose focus when I tap outside of the TextField but it could be something from the sample code.

Note: Just checked with a mouse attached and the focus is indeed lost when clicking outside of the TextField. I'm not sure what's the deal there or why there is a difference between tapping vs mouse pointer click.

Phosh video capture

Same thing with Phosh on Mobian:

https://user-images.githubusercontent.com/3979134/185765821-c4b35436-e166-4834-b8b6-c936829b0451.mp4

Phosh results:

You can't see my touches outside of TextField but the behaviour is the same:

• Rotation is working as expected :heavy_check_mark:
• The window is resized as expected when the keyboard is opened or closed :heavy_check_mark:

I will have another note about why I had to run it with ./<project_name> -b ../bundle -k but that's for https://github.com/sony/flutter-elinux/issues/74

Other observations

The odd behaviours can be checked in other tickets:

• Not losing focus when using touchscreen tap (the gesture is detected which is weird) vs mouse click outside of TextField
• Crash when a mouse is connected after initial launch (I think I saw another ticket with this issue already)

Mouse attached after the Flutter Embedded program has started

``` ➜ runner ./test -b ../bundle -k flutter: The Dart VM service is listening on http://127.0.0.1:40947/5fBZ2m0vaLY=/ [FATAL:flutter/lib/ui/window/pointer_data_packet_converter.cc(78)] Check failed: states_.find(pointer_data.device) == states_.end(). ```

• flutter-elinux difference in execution method between x86_64 and aarch64 (possible difference caused by my setup but warrants some investigation)

Conclusion

@HidenoriMatsubayashi , if you are happy with the results, I believe the ticket can be closed :slightly_smiling_face: The current behaviour is working great and in line with the confines of the limitations of the current wayland input-text protocols :+1:

[HidenoriMatsubayashi]

Thanks a lot. Yes, I agree with you. Could you please create new issues for the below?

Just checked with a mouse attached and the focus is indeed lost when clicking outside of the TextField. I'm not sure what's the deal there or why there is a difference between tapping vs mouse pointer click.

Not losing focus when using touchscreen tap (the gesture is detected which is weird) vs mouse click outside of TextField

Actually, the implementation of between touch and mouse is different. See https://github.com/sony/flutter-embedded-linux/blob/master/src/flutter/shell/platform/linux_embedded/window/elinux_window_wayland.cc#L411

Crash when a mouse is connected after initial launch (I think I saw another ticket with this issue already)

[psstoyanov]

Thank you, @HidenoriMatsubayashi .

Will gather the info on the other ones later today.

Closing this ticket.

[HidenoriMatsubayashi]

Thanks a lot.