Maestro not working with Flutter Web

[johannesstricker]

Is there an existing issue for this?

• [x] I have searched the existing issues and didn't find mine.

Steps to reproduce

Reproduction:

1. Start any Flutter web project with a Button wrapped in Semantics(label: "Click me")
2. Use Maestro with

url: http://localhost:5001
---
- launchApp
- tapOn: "Click me"

Actual results

Maestro does not find the button.

Expected results

Maestro should find the button. The docs say that Flutter web is supported: https://docs.maestro.dev/platform-support/flutter#known-limitations

About app

It's a closed source Flutter app, build only for web.

About environment

Flutter 3.24 and Maestro 1.41 on a MacBook Pro M1 Max with Chrome Browser.

Logs

Logs

<!-- Replace this line with your logs. *DO NOT* remove the backticks! -->

Maestro version

1.41.0

How did you install Maestro?

Homebrew

Anything else?

No response

[linear[bot]]

MAE-233 Maestro not working with Flutter Web

[Fishbowler]

I'm no flutter dev - is there a Flutter Hello World you can point me at, or a simple site in the real world, to give this a whirl locally?

[johannesstricker]

Well..here's a DartPad: https://dartpad.dev/?id=9e6ef617dfdc6094e95290f3d4d8b43e When you click "Run", the app will start. It's not exactly the same as running an app standalone in the browser, but it should be good enough to test.

Flutter renders everything on a canvas, but when you use the Semantics widget, Flutter will generate accessible elements on top of the canvas, that should be clickable.

[ramlaxmangroup]

yes i found the same issue.

[krille-chan]

Hey, I can reproduce it. Here is a minimal Flutter web project to reproduce it. It works fine on Android for me so this is really a shame and unfortunately a deal breaker for my project to use Maestro. I have tried different ways to set a semantics label but had no luck. There is no way I can find to detect anything on the canvas. I tried debug mode and release mode and also wasm compile mode. Any workaround for this? The docs https://docs.maestro.dev/platform-support/web-desktop-browser#known-limitations claim that flutter web would work but as far as I can see that is not the case right now :-(

url: http://localhost:<add-the-port>
---
- launchApp
- tapOn: 
    id: "Increment"

import 'package:flutter/material.dart';

void main() {
  runApp(const MyApp());
}

class MyApp extends StatelessWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'Flutter Demo',
      theme: ThemeData(
        colorScheme: ColorScheme.fromSeed(seedColor: Colors.deepPurple),
      ),
      home: const MyHomePage(title: 'Flutter Demo Home Page'),
    );
  }
}

class MyHomePage extends StatefulWidget {
  const MyHomePage({super.key, required this.title});

  final String title;

  @override
  State<MyHomePage> createState() => _MyHomePageState();
}

class _MyHomePageState extends State<MyHomePage> {
  int _counter = 0;

  void _incrementCounter() {
    setState(() {
      _counter++;
    });
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        backgroundColor: Theme.of(context).colorScheme.inversePrimary,
        title: Text(widget.title),
      ),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: <Widget>[
            const Text('You have pushed the button this many times:'),
            Text(
              '$_counter',
              style: Theme.of(context).textTheme.headlineMedium,
            ),
            Semantics(
              label: 'Increment',
              identifier: 'Increment',
              child: ElevatedButton(
                onPressed: _incrementCounter,
                child: Text('Increment'),
              ),
            ),
          ],
        ),
      ),
    );
  }
}

[ecolab-neil]

@Fishbowler @krille-chan @ramlaxmangroup @steviec It will "work" (at least mostly) if you enable semantics See: https://docs.flutter.dev/ui/accessibility/web-accessibility#turn-on-accessibility-mode-in-code @johannesstricker Wrapping in Semantics like your dartpad example would not be needed afaict and would not add value (semantics already there if you enable, and aren't if you don't)

However, i am concerned about situations where we cannot make code changes to the flutter code. how can we use Maestro easily to tap the invisible button?

This should be possible with javascript, but i personally did not figure it out yet. It is supposed to be so that a screen reader can turn it on.

[krille-chan]

@Fishbowler @krille-chan @ramlaxmangroup @steviec It will "work" (at least mostly) if you enable semantics See: https://docs.flutter.dev/ui/accessibility/web-accessibility#turn-on-accessibility-mode-in-code @johannesstricker Wrapping in Semantics like your dartpad example would not be needed afaict and would not add value (semantics already there if you enable, and aren't if you don't)

However, i am concerned about situations where we cannot make code changes to the flutter code. how can we use Maestro easily to tap the invisible button?

This should be possible with javascript, but i personally did not figure it out yet. It is supposed to be so that a screen reader can turn it on.

I tried your workaround but it does indeed work for me! 🥳

So changing the main method to:

void main() {
  runApp(const MyApp());
  if (kIsWeb) {
    SemanticsBinding.instance.ensureSemantics();
  }
}

And the maestro script to:

url: http://localhost:8081
---
- launchApp
- tapOn: "Increment"

Then run it with: flutter run -d chrome --web-port=8081 can click the button

[krille-chan]

Unfortunately we are blocked again. Three things are holding us back now in my experiments with Flutter web:

1. Tapping on something does not work reliably. In my tests the tap on the "Password" textfield failed for unknown reason every 2nd time or so
2. Impossible to tap on IconButtons, neither tooltip, Semantic-label nor Semantic-identifier works, while this works on Android
3. Impossible to scroll, neither scrollUntilVisible nor Swipe works

Also if we have multiple scrollareas or can only scroll a specific part of the app, then I don't see a way to pick it. Tried it with Swipe because there you can select something but however nothing worked anyway.

So in the current state I would not say that we can use Maestro for Flutter web 😭 which is a shame as it is so nice to use. I can provide minimal examples to reproduce if needed.

[luizvnegrini]

Unfortunately we are blocked again. Three things are holding us back now in my experiments with Flutter web:

1. Tapping on something does not work reliably. In my tests the tap on the "Password" textfield failed for unknown reason every 2nd time or so
2. Impossible to tap on IconButtons, neither tooltip, Semantic-label nor Semantic-identifier works, while this works on Android
3. Impossible to scroll, neither scrollUntilVisible nor Swipe works

Also if we have multiple scrollareas or can only scroll a specific part of the app, then I don't see a way to pick it. Tried it with Swipe because there you can select something but however nothing worked anyway.

So in the current state I would not say that we can use Maestro for Flutter web 😭 which is a shame as it is so nice to use. I can provide minimal examples to reproduce if needed.

Same here

[ff-vivek]

@krille-chan @luizvnegrini, Please provide your feedback on: https://github.com/mobile-dev-inc/Maestro/pull/2828

[ff-vivek]

@krille-chan @luizvnegrini this too: https://github.com/mobile-dev-inc/Maestro/pull/2824