samples icon indicating copy to clipboard operation
samples copied to clipboard

Published 20 hours ago • verified publisher icon flutter

`testing_app` hangs on Windows

Open johnpryan opened this issue 3 years ago • 6 comments

After enabling Flutter desktop on Windows, running flutter test integration_test -d windows hangs until ctrl+c, then a TimeoutException is printed.

@dnfield @csells I followed the setup instructions at https://docs.flutter.dev/desktop and I can successfully run the app under test (flutter run -d windows)

The integration tests in this sample are currently failing on a macOS device: (https://github.com/flutter/samples/issues/999)

Logs 
C:\Users\johnp\code\samples\testing_app>flutter test integration_test -d windows
00:00 +0: loading C:\Users\johnp\code\samples\testing_app\integration_test\app_test.dart                                         B21:49 +0 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\app_test.dart [E]
  TimeoutException after 0:12:00.000000: Test timed out after 12 minutes.
  package:test_api/src/backend/invoker.dart 333:28  Invoker._handleError.<fn>
  dart:async/zone.dart 1418:47                      _rootRun
  dart:async/zone.dart 1328:19                      _CustomZone.run
  package:test_api/src/backend/invoker.dart 331:10  Invoker._handleError
  package:test_api/src/backend/invoker.dart 287:9   Invoker.heartbeat.<fn>.<fn>
  dart:async/zone.dart 1426:13                      _rootRun
  dart:async/zone.dart 1328:19                      _CustomZone.run
  package:test_api/src/backend/invoker.dart 286:38  Invoker.heartbeat.<fn>
  dart:async-patch/timer_patch.dart 18:15           Timer._createTimer.<fn>
  dart:isolate-patch/timer_impl.dart 395:19         _Timer._runTimers
  dart:isolate-patch/timer_impl.dart 426:5          _Timer._handleMessage
  dart:isolate-patch/isolate_patch.dart 192:12      _RawReceivePortImpl._handleMessage

21:49 +0 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart
21:50 +0 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart                                     Terminate batch job (Y/N)? y

C:\Users\johnp\code\samples\testing_app>flutter --version
Flutter 2.10.0-0.1.pre • channel beta • https://github.com/flutter/flutter.git
Framework • revision 628f0e3f3a (9 days ago) • 2022-01-11 19:50:05 -0600
Engine • revision 63ca99584a
Tools • Dart 2.16.0 (build 2.16.0-134.1.beta) • DevTools 2.9.2

johnpryan avatar Jan 20 '22 21:01 johnpryan

@dnfield can you take a look at this? I'd love integration_test to work well on Windows.

csells avatar Jan 20 '22 21:01 csells

After running the tests again, I can see one of the tests passing but another gets a TestDeviceException:

logs 
C:\Users\johnp\code\samples\testing_app>flutter test integration_test -d windows
00:00 +0: loading C:\Users\johnp\code\samples\testing_app\integration_test\app_test.dart                                         B00:16 +0: loading C:\Users\johnp\code\samples\testing_app\integration_test\app_test.dart
00:26 +4: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart                                        B00:40 +4: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart
Error waiting for a debug connection: The log reader stopped unexpectedly, or never started.
00:40 +4 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart [E]
  TestDeviceException(Unable to start the app on the device.)
  package:flutter_tools/src/test/integration_test_device.dart 61:7  IntegrationTestTestDevice.start

00:40 +4 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\state_mgmt_test.dart                               B00:40 +4 -1: loading C:\Users\johnp\code\samples\testing_app\integration_test\perf_test.dart [E]
  Bad state: Cannot add new events after calling close
  dart:async/broadcast_stream_controller.dart 243:24  _BroadcastStreamController.add
  dart:async/zone.dart 1434:47                        _rootRunUnary
  dart:async/zone.dart 1335:19                        _CustomZone.runUnary
  dart:async/zone.dart 1244:7                         _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 341:11                  _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 271:7                   _BufferingStreamSubscription._add
  dart:async/stream_controller.dart 774:19            _SyncStreamControllerDispatch._sendData
  dart:async/stream_controller.dart 648:7             _StreamController._add
  dart:async/stream_controller.dart 596:5             _StreamController.add
  dart:io-patch/socket_patch.dart 2314:41             _Socket._onData
  dart:async/zone.dart 1442:13                        _rootRunUnary
  dart:async/zone.dart 1335:19                        _CustomZone.runUnary
  dart:async/zone.dart 1244:7                         _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 341:11                  _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 271:7                   _BufferingStreamSubscription._add
  dart:async/stream_controller.dart 774:19            _SyncStreamControllerDispatch._sendData
  dart:async/stream_controller.dart 648:7             _StreamController._add
  dart:async/stream_controller.dart 596:5             _StreamController.add
  dart:io-patch/socket_patch.dart 1839:33             new _RawSocket.<fn>
  dart:io-patch/socket_patch.dart 1322:14             _NativeSocket.issueReadEvent.issue
  dart:async/schedule_microtask.dart 40:21            _microtaskLoop
  dart:async/schedule_microtask.dart 49:5             _startMicrotaskLoop
  dart:isolate-patch/isolate_patch.dart 122:13        _runPendingImmediateCallback
  dart:isolate-patch/isolate_patch.dart 193:5         _RawReceivePortImpl._handleMessage

C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: The command "setlocal [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" -DBUILD_TYPE=Debug -P cmake_install.cmake [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: :cmEnd [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: :cmErrorLevel [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: exit /b %1 [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: :cmDone [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(156,5): error MSB3073: :VCEnd" exited with code 1. [C:\Users\johnp\code\samples\testing_app\build\windows\INSTALL.vcxproj]
Building Windows application...
00:55 +4 -2: loading C:\Users\johnp\code\samples\testing_app\integration_test\state_mgmt_test.dart [E]
  Exception: Build process failed.
  package:flutter_tools/src/base/common.dart 10:3                    throwToolExit
  package:flutter_tools/src/windows/build_windows.dart 311:5         _runBuild
  ===== asynchronous gap ===========================
  package:flutter_tools/src/windows/build_windows.dart 77:5          buildWindows
  ===== asynchronous gap ===========================
  package:flutter_tools/src/windows/windows_device.dart 62:5         WindowsDevice.buildForDevice
  ===== asynchronous gap ===========================
  package:flutter_tools/src/desktop_device.dart 123:7                DesktopDevice.startApp
  ===== asynchronous gap ===========================
  package:flutter_tools/src/test/integration_test_device.dart 53:39  IntegrationTestTestDevice.start

00:55 +4 -2: Some tests failed.

C:\Users\johnp\code\samples\testing_app>

johnpryan avatar Jan 20 '22 21:01 johnpryan

It looks like there's some sort of race condition when testing more than one file at a time... I can consistently run flutter test integration_test\app_test.dart -d windows and flutter test integration_test\perf_test.dart -d windows, but not flutter test integration_test -d windows.

johnpryan avatar Jan 20 '22 21:01 johnpryan

@cbracken might be better set up to help with this (or know who is). My access to Windows machines is fairly limited

dnfield avatar Jan 20 '22 22:01 dnfield

Thanks for the heads-up. Taking a look.

cbracken avatar Jan 20 '22 22:01 cbracken

@csells requested me to put here my findings.

Setup

flutter doctor 
$ flutter doctor
Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel beta, 2.10.0-0.2.pre, on Microsoft Windows [Version 10.0.22000.434], locale en-150)
[✓] Android toolchain - develop for Android devices (Android SDK version 32.0.0)
[✓] Chrome - develop for the web
[✓] Visual Studio - develop for Windows (Visual Studio Community 2019 16.11.4)
[✓] Android Studio (version 2020.3)
[✓] VS Code (version 1.63.2)
[✓] Connected device (3 available)
[✓] HTTP Host Availability

• No issues found!

All tests running on Windows 11 with the Git Bash on Windows Terminal.

Results

Running flutter test integration_test -d windows:

  • the app_test.dart runs and passes
  • the perf_test.dart eventually fails with the following error (see logs)
  • the state_mgmt_test.dart doesn't get to run, also fails (app is a weird state)
logs 
$ flutter test integration_test -d windows
Running "flutter pub get" in testing_app...                      2,280ms
00:00 +0: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\app_test.dart                                  B00:26 +0: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\app_test.dart
00:37 +4: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\perf_test.dart                                 B00:50 +4: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\perf_test.dart
Error waiting for a debug connection: The log reader stopped unexpectedly, or never started.
00:50 +4 -1: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\perf_test.dart [E]
  TestDeviceException(Unable to start the app on the device.)
  package:flutter_tools/src/test/integration_test_device.dart 61:7  IntegrationTestTestDevice.start

00:50 +4 -1: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart                        B00:50 +4 -1: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\perf_test.dart [E]
  Bad state: Cannot add new events after calling close
  dart:async/broadcast_stream_controller.dart 243:24  _BroadcastStreamController.add
  dart:async/zone.dart 1434:47                        _rootRunUnary
  dart:async/zone.dart 1335:19                        _CustomZone.runUnary
  dart:async/zone.dart 1244:7                         _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 341:11                  _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 271:7                   _BufferingStreamSubscription._add
  dart:async/stream_controller.dart 774:19            _SyncStreamControllerDispatch._sendData
  dart:async/stream_controller.dart 648:7             _StreamController._add
  dart:async/stream_controller.dart 596:5             _StreamController.add
  dart:io-patch/socket_patch.dart 2314:41             _Socket._onData
  dart:async/zone.dart 1442:13                        _rootRunUnary
  dart:async/zone.dart 1335:19                        _CustomZone.runUnary
  dart:async/zone.dart 1244:7                         _CustomZone.runUnaryGuarded
  dart:async/stream_impl.dart 341:11                  _BufferingStreamSubscription._sendData
  dart:async/stream_impl.dart 271:7                   _BufferingStreamSubscription._add
  dart:async/stream_controller.dart 774:19            _SyncStreamControllerDispatch._sendData
  dart:async/stream_controller.dart 648:7             _StreamController._add
  dart:async/stream_controller.dart 596:5             _StreamController.add
  dart:io-patch/socket_patch.dart 1839:33             new _RawSocket.<fn>
  dart:io-patch/socket_patch.dart 1322:14             _NativeSocket.issueReadEvent.issue
  dart:async/schedule_microtask.dart 40:21            _microtaskLoop
  dart:async/schedule_microtask.dart 49:5             _startMicrotaskLoop
  dart:isolate-patch/isolate_patch.dart 122:13        _runPendingImmediateCallback
  dart:isolate-patch/isolate_patch.dart 193:5         _RawReceivePortImpl._handleMessage


Error waiting for a debug connection: The log reader stopped unexpectedly, or never started.
01:05 +4 -2: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart [E]
  TestDeviceException(Unable to start the app on the device.)
  package:flutter_tools/src/test/integration_test_device.dart 61:7  IntegrationTestTestDevice.start

01:05 +4 -2: Some tests failed.

Running flutter test integration_test/app_test.dart -d windows:

  • test runs and passes
  • command never finishes (I had to press q to finish it)
logs 
$ flutter run integration_test/app_test.dart -d windows
Launching integration_test/app_test.dart on Windows in debug mode...
Building Windows application...
flutter: 00:00 +0: Testing App Driver Tests Finding an item in the list
Syncing files to device Windows...                                  53ms

Flutter run key commands.
r Hot reload. 🔥🔥🔥
R Hot restart.
h List all available interactive commands.
d Detach (terminate "flutter run" but leave application running).
c Clear the screen
q Quit (terminate the application on the device).

💪 Running with sound null safety 💪

An Observatory debugger and profiler on Windows is available at: http://127.0.0.1:63630/YJ2FEDpVaPI=/
The Flutter DevTools debugger and profiler on Windows is available at: http://127.0.0.1:9100?uri=http://127.0.0.1:63630/YJ2FEDpVaPI=/
flutter: 00:00 +1: Testing App Driver Tests Testing IconButtons
flutter: 00:05 +2: Testing App Driver Tests Verifying whether item gets added to favorites
flutter: 00:06 +3: Testing App Driver Tests Testing remove button
flutter: 00:10 +4: Testing App Driver Tests (tearDownAll)
flutter: Warning: integration_test plugin was not detected.
flutter:
flutter: If you're running the tests with `flutter drive`, please make sure your tests
flutter: are in the `integration_test/` directory of your package and use
flutter: `flutter test $path_to_test` to run it instead.
flutter:
flutter: If you're running the tests with Android instrumentation or XCTest, this means
flutter: that you are not capturing test results properly! See the following link for
flutter: how to set up the integration_test plugin:
flutter:
flutter: https://flutter.dev/docs/testing/integration-tests#testing-on-firebase-test-lab
flutter:
flutter: 00:10 +5: All tests passed!

Application finished.

Running performance test:

  • Run the command flutter drive --driver=integration_test/perf_driver.dart --target=integration_test/perf_test.dart -d windows
  • Runs and finishes successfully.
logs 
$ flutter drive --driver=integration_test/perf_driver.dart --target=integration_test/perf_test.dart -d windows
Running "flutter pub get" in testing_app...                        990ms
Building Windows application...
VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:63762/X8EGYWluyoE=/
VMServiceFlutterDriver: Isolate found with number: 706933644293459
VMServiceFlutterDriver: Isolate is paused at start.
VMServiceFlutterDriver: Attempting to resume isolate
flutter: 00:00 +0: Testing App Performance Tests Scrolling test
VMServiceFlutterDriver: Connected to Flutter application.
flutter: 00:04 +1: Testing App Performance Tests Favorites operations test
flutter: 00:15 +2: Testing App Performance Tests (tearDownAll)
flutter: Warning: integration_test plugin was not detected.
flutter:
flutter: If you're running the tests with `flutter drive`, please make sure your tests
flutter: are in the `integration_test/` directory of your package and use
flutter: `flutter test $path_to_test` to run it instead.
flutter:
flutter: If you're running the tests with Android instrumentation or XCTest, this means
flutter: that you are not capturing test results properly! See the following link for
flutter: how to set up the integration_test plugin:
flutter:
flutter: https://flutter.dev/docs/testing/integration-tests#testing-on-firebase-test-lab
flutter:
flutter: 00:15 +3: All tests passed!
All tests passed.

Running state_mgmt_test.dart:

  • Same result as with the app_test.dart:
  • Runs successfuly
  • the testing app remains open, needs to be closed with q
logs 
$ flutter run integration_test/state_mgmt_test.dart -d windows
Launching integration_test/state_mgmt_test.dart on Windows in debug mode...
Building Windows application...
flutter: 00:00 +0: Testing App State Management Tests Verifying add method
Syncing files to device Windows...                                  55ms

Flutter run key commands.
r Hot reload. 🔥🔥🔥
R Hot restart.
h List all available interactive commands.
d Detach (terminate "flutter run" but leave application running).
c Clear the screen
q Quit (terminate the application on the device).

💪 Running with sound null safety 💪

An Observatory debugger and profiler on Windows is available at: http://127.0.0.1:63842/agkZOurH5FE=/
The Flutter DevTools debugger and profiler on Windows is available at: http://127.0.0.1:9100?uri=http://127.0.0.1:63842/agkZOurH5FE=/
flutter: 00:00 +1: Testing App State Management Tests Verifying remove method
flutter: 00:00 +2: Testing App State Management Tests (tearDownAll)
flutter: Warning: integration_test plugin was not detected.
flutter:
flutter: If you're running the tests with `flutter drive`, please make sure your tests
flutter: are in the `integration_test/` directory of your package and use
flutter: `flutter test $path_to_test` to run it instead.
flutter:
flutter: If you're running the tests with Android instrumentation or XCTest, this means
flutter: that you are not capturing test results properly! See the following link for
flutter: how to set up the integration_test plugin:
flutter:
flutter: https://flutter.dev/docs/testing/integration-tests#testing-on-firebase-test-lab
flutter:
flutter: 00:00 +3: All tests passed!

Application finished.

Removing the files drive.dart perf_driver.dart and perf_test.dart from the integration_test folder, then running flutter test integration_test -d windows:

  • The test app_test.dart runs and passes.
  • The test state_mgmt_test.dart starts and fails.
  • A blank app window appears on the screen and remains open.
logs 
$ flutter run integration_test -d windows
Target file "integration_test" not found.

Miquel@MIGUEL-WINDOWS MINGW64 ~/Documents/dev/samples/testing_app (master)
$ flutter test integration_test -d windows
00:00 +0: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\app_test.dart                                  B00:13 +0: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\app_test.dart
00:24 +4: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart                           B00:37 +4: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart
Error waiting for a debug connection: The log reader stopped unexpectedly, or never started.
00:37 +4 -1: loading C:\Users\Miquel\Documents\dev\samples\testing_app\integration_test\state_mgmt_test.dart [E]
  TestDeviceException(Unable to start the app on the device.)
  package:flutter_tools/src/test/integration_test_device.dart 61:7  IntegrationTestTestDevice.start

00:37 +4 -1: Some tests failed.

Summary

  • It seems that when more than one test file is in the integration_test folder, the command flutter test integration_test will fail on the second file.
  • Running a single test leaves the command running and needs to be closed with q (or control+c).
  • The fact that the performance tests are in the same folder seems unrelated to the root cause of this issue.

miquelbeltran avatar Jan 25 '22 13:01 miquelbeltran