Connect to VM timeout when doing integration tests

[tianhaoz95]

Steps to Reproduce

1. Run flutter drive --target=test_driver/app.dart.
2. see the error on test startup:

VMServiceFlutterDriver: It is taking an unusually long time to connect to the VM...
VMServiceFlutterDriver: It is taking an unusually long time to connect to the VM...
VMServiceFlutterDriver: It is taking an unusually long time to connect to the VM...
VMServiceFlutterDriver: It is taking an unusually long time to connect to the VM...
...

Expected results: the integration tests should start running in the emulator

Actual results: it keeps saying cannot connect to VM and finally timeout.

see this CI run: https://github.com/tianhaoz95/photochat/pull/17/checks?check_run_id=449361947#step:6:936

logs_157.zip

[jagsme]

I am also facing a similar issue, but it is when you try to pause the build and run in parallel,

as mentioned in here -https://github.com/flutter/flutter/pull/41583

I am able to run the app on the devices and pause them as well, but unable to attach the driver to the URI

this command flutter drive -d all --use-existing-app="$(cat foo)" --driver=test_driver/app_test.dart -v

[TahaTesser]

Could be related #34196

[jagsme]

the #34196 is not what I am facing, I am able to execute tests with flutter drive, the thing I am facing here is when we connect flutter driver to the URI as mentioned in here #41583

[jagsme]

@TahaTesser the issue is fixed when we upgrade flutter version to Flutter 1.12.13+hotfix.8.

[tianhaoz95]

UPDATE: It doesn't happen to all emulators. For example, pixel and pixel_xl are okay, but pixel_c has this issue.

pixel_c repeatedly fail with this error.

[tianhaoz95]

UPDATE: upgrading to 1.12.13+hotfix.8 didn't fix the issue. Please see https://github.com/tianhaoz95/photochat/runs/461236331?check_suite_focus=true for more logging details.

[HerrNiklasRaab]

@tianhaoz95 Upgrading to 1.12.13+hotfix.8 fixed it for me

[tianhaoz95]

@HerrNiklasRaab can you add the channel you were on? I tried stable. Maybe dev should be better?

[HerrNiklasRaab]

I was on stable to:

[✓] Flutter (Channel stable, v1.12.13+hotfix.8, on Mac OS X 10.15.3 19D76, locale de)
 
[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3)
[✓] Xcode - develop for iOS and macOS (Xcode 11.3.1)
[✓] Android Studio (version 3.5)
[✓] VS Code (version 1.42.1)
[✓] Connected device (2 available)

• No issues found!

[tianhaoz95]

Interesting, this test failed 16 hours ago with v1.12.13+hotfix.8 defined here:

https://github.com/tianhaoz95/photochat/blob/f95c5f0d7880e7ca90e4684fc618d299a90c83d2/.github/workflows/thor.yml#L29-L32

[tianhaoz95]

oh actually nvm. it's a different issue

[tianhaoz95]

I think that fixed it for me too! thanks!

[tianhaoz95]

mark as resolved ;)

[tianhaoz95]

actually after a few days I found that it only made the failure less frequent. It still happens.

https://github.com/tianhaoz95/photochat/runs/478586086?check_suite_focus=true#step:6:2017

[tianhaoz95]

ping. any new findings on this? from the run I attached above, we can find that it happens only on Android. Is there a problem with Flutter + adb?

[tianhaoz95]

Is there any progress? It's still flaky on all three channels.

[laynuk]

The same issue with real iOS device with 13.5.1 iPhone Xr Catalina 10.15.5 XCode 11.5

[✓] Flutter (Channel unknown, v1.17.0, on Mac OS X 10.15.5 19F101, locale en-SK) • Flutter version 1.17.0 at /Users/macpro2/flutter • Framework revision e6b34c2b5c (5 weeks ago), 2020-05-02 11:39:18 -0700 • Engine revision 540786dd51 • Dart version 2.8.1

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.3) • Android SDK at /Users/macpro2/Library/Android/sdk • Platform android-29, build-tools 29.0.3 • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593) • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.5) • Xcode at /Applications/Xcode.app/Contents/Developer • Xcode 11.5, Build version 11E608c • CocoaPods version 1.9.3

[✓] Android Studio (version 4.0) • Android Studio at /Applications/Android Studio.app/Contents • Flutter plugin version 46.0.2 • Dart plugin version 193.7361 • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)

[✓] Connected device (1 available) • myiphone • 00008020-001E7C8Cxxxxxxxx • ios • iOS 13.5.1

• No issues found!

but for the some reason at start of integration test appears the following record: VMServiceFlutterDriver: Connecting to Flutter application at http:// VMServiceFlutterDriver: It is taking an unusually long time to connect to the VM... VMServiceFlutterDriver: It is taking an unusually long time to connect to the VM... VMServiceFlutterDriver: It is taking an unusually long time to connect to the VM... ...

[arrmixer]

I'm getting the same issue as well any resolution so far? flutter/bin/cache/dart-sdk/bin/dart --enable-asserts --enable-vm-service:56030 ~/data_backend/test_driver/app_test.dart Observatory listening on http://127.0.0.1:56030/Nrk5twr2xdg=/

00:00 +0: Todo App (setUpAll)

VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:8888/ VMServiceFlutterDriver: It is taking an unusually long time to connect to the VM...

flutter doctor -v

[✓] Flutter (Channel stable, 1.20.3, on Mac OS X 10.15.6 19G2021, locale en-US)
    • Flutter version 1.20.3 at /Users/angelrodriguez/Library/flutter
    • Framework revision 216dee60c0 (7 days ago), 2020-09-01 12:24:47 -0700
    • Engine revision d1bc06f032
    • Dart version 2.9.2

[✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2)
    • Android SDK at /Users/angelrodriguez/Library/Android/sdk
    • Platform android-29, build-tools 29.0.2
    • Java binary at: /Applications/Android Studio.app/Contents/jre/jdk/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 11.7)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Xcode 11.7, Build version 11E801a
    • CocoaPods version 1.9.3

[✓] Android Studio (version 4.0)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin version 49.0.2
    • Dart plugin version 193.7547
    • Java version OpenJDK Runtime Environment (build 1.8.0_242-release-1644-b3-6222593)

[✓] IntelliJ IDEA Community Edition (version 2020.1.4)
    • IntelliJ at /Applications/IntelliJ IDEA CE.app
    • Flutter plugin version 49.0.3
    • Dart plugin version 201.9002

[✓] Connected device (1 available)
    • iPhone 11 (mobile) • BFF1A066-978A-4915-90C5-0D53E542C871 • ios • com.apple.CoreSimulator.SimRuntime.iOS-13-7 (simulator)

• No issues found!

[themizzi]

We began experiencing this after upgrading to 1.20.4. We did not see it previously on 1.17.5, but considering that this appears intermittently I'm not sure if that is valuable or not. This is really causing us some headaches in our build pipeline. We are specifically experiencing this on our android tests, not ios.

[jbojcic1]

I have the same issue occasionally. After I re-run it usually works ok. I am using Android 10.0 x86_x64 simulator without Google Play.

Flutter doctor output:

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 1.20.3, on Mac OS X 10.15.4 19E287, locale en-US)
 
[✓] Android toolchain - develop for Android devices (Android SDK version 28.0.3)
[✓] Xcode - develop for iOS and macOS (Xcode 11.7)
[✓] Android Studio (version 4.0)
[!] IntelliJ IDEA Ultimate Edition (version 2020.2.2)
    ✗ Flutter plugin not installed; this adds Flutter specific functionality.
    ✗ Dart plugin not installed; this adds Dart specific functionality.
[✓] VS Code (version 1.49.3)
[✓] Connected device (1 available)

[fzyzcjy]

Same issue here. It happens even in GitHub Actions, the CI environment.

The output:

https://github.com/fzyzcjy/flutter_rust_bridge/runs/3812009211

The workflow file:

https://github.com/fzyzcjy/flutter_rust_bridge/actions/runs/1310850737/workflow

[bartekpacia]

We're also facing this problem. So very annoying. This makes integration testing have such a high flakiness ratio that it's a no-go.

See also:

• our workflow file

[Kiruel]

Got the same problem on flutter 3.3.8 on emulator iPhone 14 pro max.

[praxder]

Same issue here. Keeping us from being able to use integration tests since they can't be run consistently in CI/CD.

[bartekpacia]

This was mostly fixed in #113969 by the awesome @jakubfijalkowski. Should be available in the next stable Flutter release.

[littleGnAl]

I faced a similar issue on github runner when running the ios integration test by using the flutter test integration_test, it keeps failing, here is the failed run: https://github.com/AgoraIO-Extensions/iris_method_channel_flutter/actions/runs/6321185956/job/17164814096

But if I specify the test file and the device flutter test integration_test/my_test_file.dart -d "iPhone 14 Pro Max" --verbose it finally succeeded, although it still has a chance to fail, but it more easier to succeed than without setting the test file and device. https://github.com/AgoraIO-Extensions/iris_method_channel_flutter/actions/runs/6329732359/job/17203218825

[kjxbyz]

same issue

ying@Kimmys-Mac-mini frontend-expert-app % make test_e2e
echo "╠ Running E2E Testing..."
╠ Running E2E Testing...
flutter test integration_test/demo_test.dart --flavor googleDev -d "emulator-5554"
00:00 +0: loading /Users/ying/work/frontend-expert-app/integration_test/demo_test.dart                                                                                                   R00:24 +0: loading /Users/ying/work/frontend-expert-app/integration_test/demo_test.dart                                                                                              23.6s
✓  Built build/app/outputs/flutter-apk/app-googledev-debug.apk.
00:25 +0: loading /Users/ying/work/frontend-expert-app/integration_test/demo_test.dart                                                                                                   I00:27 +0: loading /Users/ying/work/frontend-expert-app/integration_test/demo_test.dart                                                                                            1,526ms
00:33 +0 -1: loading /Users/ying/work/frontend-expert-app/integration_test/demo_test.dart [E]                                                                                            
  Failed to load "/Users/ying/work/frontend-expert-app/integration_test/demo_test.dart": Connecting to the VM Service timed out.
  package:flutter_tools/src/test/integration_test_device.dart 82:24  IntegrationTestTestDevice.start.<fn>
  dart:async/zone.dart 1391:47                                       _rootRun
  dart:async/zone.dart 1301:19                                       _CustomZone.run
  dart:async/future_impl.dart 938:34                                 Future.timeout.<fn>
  dart:async/zone.dart 1391:47                                       _rootRun
  dart:async/zone.dart 1301:19                                       _CustomZone.run
  dart:async/zone.dart 1209:7                                        _CustomZone.runGuarded
  dart:async/zone.dart 1249:23                                       _CustomZone.bindCallbackGuarded.<fn>
  dart:async/zone.dart 1399:13                                       _rootRun
  dart:async/zone.dart 1301:19                                       _CustomZone.run
  dart:async/zone.dart 1233:23                                       _CustomZone.bindCallback.<fn>
  dart:async-patch/timer_patch.dart 18:15                            Timer._createTimer.<fn>
  dart:isolate-patch/timer_impl.dart 398:19                          _Timer._runTimers
  dart:isolate-patch/timer_impl.dart 429:5                           _Timer._handleMessage
  dart:isolate-patch/isolate_patch.dart 189:12                       _RawReceivePort._handleMessage
  

To run this test again: /Users/ying/flutter/bin/cache/dart-sdk/bin/dart test /Users/ying/work/frontend-expert-app/integration_test/demo_test.dart -p vm --plain-name 'loading /Users/ying/work/frontend-expert-app/integration_test/demo_test.dart'
00:34 +0 -1: Some tests failed.                                                                                                                                                          
make: *** [test_e2e] Error 1

[✓] Flutter (Channel stable, 3.13.8, on macOS 14.0 23A344 darwin-x64, locale zh-Hans-CN)
    • Flutter version 3.13.8 on channel stable at /Users/ying/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 6c4930c4ac (14 小时前), 2023-10-18 10:57:55 -0500
    • Engine revision 767d8c75e8
    • Dart version 3.1.4
    • DevTools version 2.25.0

[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
    • Android SDK at /Users/ying/Android/sdk
    • Platform android-33, build-tools 33.0.2
    • ANDROID_HOME = /Users/ying/Android/sdk
    • Java binary at: /Applications/Android Studio.app/Contents/jbr/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 15.0.1)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • Build 15A507
    • CocoaPods version 1.13.0

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2022.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 17.0.6+0-17.0.6b829.9-10027231)

[✓] IntelliJ IDEA Ultimate Edition (version 2023.2.3)
    • IntelliJ at /Applications/IntelliJ IDEA.app
    • Flutter plugin version 75.1.4
    • Dart plugin version 232.10072.19

[!] Proxy Configuration
    • HTTP_PROXY is set
    ! NO_PROXY is not set

[✓] Connected device (3 available)
    • sdk gphone64 x86 64 (mobile) • emulator-5554 • android-x64    • Android 13 (API 33) (emulator)
    • macOS (desktop)              • macos         • darwin-x64     • macOS 14.0 23A344 darwin-x64
    • Chrome (web)                 • chrome        • web-javascript • Google Chrome 118.0.5993.88

[✓] Network resources
    • All expected network resources are available.

! Doctor found issues in 1 category.

[matanlurey]

Could someone try this again with the latest stable version(s) of Flutter and let us know if this still is a problem?

[top-master]

Root-cause of the issue

Basically, most times the battery optimization kills the integration-test's APK.

My company is 100% sure, since unless we enable "unrestricted" in battery settings, the tests fail 100% of the times we tried.

Now once the tests pass the APK gets auto-uninstalled, and the settings restore to default. Hence see "Fix" section.

Fix

We fixed with following workaround:

• You need to enable "stay awake" mode in Android-device's settings.

• Then, each time before launching the tests, turn-on your Android-device's display.

Note that said "stay awake" is too hard, also does not work in some devices, hence I use an App with the sole purpose of keeping screen on.

Extra Note: There were many such Apps, and I tested many, however, only few keep the screen on.

[top-master]

@matanlurey Basically, this is a common Android issue as mentioned.

Hence, I wonder how can a new Flutter version make any difference.

We may need a "new Android version" and/or a "new adb version".