Error running perf_test in the testing_app sample

[InMatrix]

trafficstars

I had trouble getting the perf_test to work on a physical Android device. The model is Galaxy J7 (2016). I was able to run the app on the device in profile mode without an issue. Did I miss anything?

CLI output:

$ flutter drive --driver=integration_test/perf_driver.dart --target=integration_test/perf_test.dart --profile --trace-startup
Running "flutter pub get" in testing_app...                        781ms
Warning: Mapping new ns http://schemas.android.com/repository/android/common/02 to old ns http://schemas.android.com/repository/android/common/01
Warning: Mapping new ns http://schemas.android.com/repository/android/generic/02 to old ns http://schemas.android.com/repository/android/generic/01
Warning: Mapping new ns http://schemas.android.com/sdk/android/repo/addon2/02 to old ns http://schemas.android.com/sdk/android/repo/addon2/01
Warning: Mapping new ns http://schemas.android.com/sdk/android/repo/repository2/02 to old ns http://schemas.android.com/sdk/android/repo/repository2/01
Warning: Mapping new ns http://schemas.android.com/sdk/android/repo/sys-img2/02 to old ns http://schemas.android.com/sdk/android/repo/sys-img2/01
Running Gradle task 'assembleProfile'...                         2,474ms
✓  Built build/app/outputs/flutter-apk/app-profile.apk (11.4MB).
Installing build/app/outputs/flutter-apk/app.apk...                 4.8s
VMServiceFlutterDriver: Connecting to Flutter application at http://127.0.0.1:57484/MuLxdx40N4k=/
VMServiceFlutterDriver: Isolate found with number: 4331278427044967
VMServiceFlutterDriver: Isolate is paused at start.
VMServiceFlutterDriver: Attempting to resume isolate
I/flutter ( 6002): 00:00 +0: Testing App Performance Tests Scrolling test
D/libGLESv2( 6002): STS_GLApi : DTS, ODTC are not allowed for Package : dev.flutter.testing_app
VMServiceFlutterDriver: Connected to Flutter application.
I/flutter ( 6002): SocketException: OS Error: Connection refused, errno = 111, address = localhost, port
= 57506
I/flutter ( 6002): #0      _NativeSocket.startConnect (dart:io-patch/socket_patch.dart:681)
I/flutter ( 6002): #1      _RawSocket.startConnect (dart:io-patch/socket_patch.dart:1704)
I/flutter ( 6002): #2      RawSocket.startConnect (dart:io-patch/socket_patch.dart:27)
I/flutter ( 6002): #3      Socket._startConnect (dart:io-patch/socket_patch.dart:1915)
I/flutter ( 6002): #4      Socket.startConnect (dart:io/socket.dart:738)
I/flutter ( 6002): #5      _ConnectionTarget.connect (dart:_http/http_impl.dart:2440)
I/flutter ( 6002): #6      _HttpClient._getConnection.connect (dart:_http/http_impl.dart:2806)
I/flutter ( 6002): #7      _HttpClient._getConnection (dart:_http/http_impl.dart:2811)
I/flutter ( 6002): #8      _HttpClient._openUrl (dart:_http/http_impl.dart:2695)
I/flutter ( 6002): #9      _HttpClient.openUrl (dart:_http/http_impl.dart:2569)
I/flutter ( 6002): #10     _WebSocketImpl.connect (dart:_http/websocket_impl.dart:1027)
I/flutter ( 6002): #11     WebSocket.connect (dart:_http/websocket.dart:380)
I/flutter ( 6002): #12     vmServiceConnectUri (package:vm_service/vm_service_io.dart:32)
I/flutter ( 6002): #13     IntegrationTestWidgetsFlutterBinding.enableTimeline
(package:integration_test/integration_test.dart:247)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): #14     IntegrationTestWidgetsFlutterBinding.traceTimeline
(package:integration_test/integration_test.dart:272)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): #15     IntegrationTestWidgetsFlutterBinding._runAndGetGCInfo
(package:integration_test/integration_test.dart:347)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): #16     IntegrationTestWidgetsFlutterBinding.watchPerformance
(package:integration_test/integration_test.dart:402)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): #17     main.<anonymous closure>.<anonymous closure>
(file:///Users/taodong/Code/flutter/samples/testing_app/integration_test/perf_test.dart:28)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): #18     testWidgets.<anonymous closure>.<anonymous closure>
(package:flutter_test/src/widget_tester.dart:170)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): #19     TestWidgetsFlutterBinding._runTestBody
(package:flutter_test/src/binding.dart:803)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): 00:02 +0: Testing App Performance Tests Scrolling test [E]
I/flutter ( 6002):   Test failed. See exception logs above.
I/flutter ( 6002):   The test description was: Scrolling test
I/flutter ( 6002):   
I/flutter ( 6002): Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
I/flutter ( 6002): For example, 'dart test --chain-stack-traces'.
I/flutter ( 6002): 00:02 +0 -1: Testing App Performance Tests Favorites operations test
I/zygote  ( 6002): Do partial code cache collection, code=29KB, data=30KB
I/zygote  ( 6002): After code cache collection, code=29KB, data=30KB
I/zygote  ( 6002): Increasing code cache capacity to 128KB
I/flutter ( 6002): SocketException: OS Error: Connection refused, errno = 111, address = localhost, port
= 57507
I/flutter ( 6002): #0      _NativeSocket.startConnect (dart:io-patch/socket_patch.dart:681)
I/flutter ( 6002): #1      _RawSocket.startConnect (dart:io-patch/socket_patch.dart:1704)
I/flutter ( 6002): #2      RawSocket.startConnect (dart:io-patch/socket_patch.dart:27)
I/flutter ( 6002): #3      Socket._startConnect (dart:io-patch/socket_patch.dart:1915)
I/flutter ( 6002): #4      Socket.startConnect (dart:io/socket.dart:738)
I/flutter ( 6002): #5      _ConnectionTarget.connect (dart:_http/http_impl.dart:2440)
I/flutter ( 6002): #6      _HttpClient._getConnection.connect (dart:_http/http_impl.dart:2806)
I/flutter ( 6002): #7      _HttpClient._getConnection (dart:_http/http_impl.dart:2811)
I/flutter ( 6002): #8      _HttpClient._openUrl (dart:_http/http_impl.dart:2695)
I/flutter ( 6002): #9      _HttpClient.openUrl (dart:_http/http_impl.dart:2569)
I/flutter ( 6002): #10     _WebSocketImpl.connect (dart:_http/websocket_impl.dart:1027)
I/flutter ( 6002): #11     WebSocket.connect (dart:_http/websocket.dart:380)
I/flutter ( 6002): #12     vmServiceConnectUri (package:vm_service/vm_service_io.dart:32)
I/flutter ( 6002): #13     IntegrationTestWidgetsFlutterBinding.enableTimeline
(package:integration_test/integration_test.dart:247)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): #14     IntegrationTestWidgetsFlutterBinding.traceTimeline
(package:integration_test/integration_test.dart:272)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): #15     IntegrationTestWidgetsFlutterBinding._runAndGetGCInfo
(package:integration_test/integration_test.dart:347)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): #16     IntegrationTestWidgetsFlutterBinding.watchPerformance
(package:integration_test/integration_test.dart:402)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): #17     main.<anonymous closure>.<anonymous closure>
(file:///Users/taodong/Code/flutter/samples/testing_app/integration_test/perf_test.dart:56)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): #18     testWidgets.<anonymous closure>.<anonymous closure>
(package:flutter_test/src/widget_tester.dart:170)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): #19     TestWidgetsFlutterBinding._runTestBody
(package:flutter_test/src/binding.dart:803)
I/flutter ( 6002): <asynchronous suspension>
I/flutter ( 6002): 00:04 +0 -1: Testing App Performance Tests Favorites operations test [E]
I/flutter ( 6002):   Test failed. See exception logs above.
I/flutter ( 6002):   The test description was: Favorites operations test
I/flutter ( 6002):   
I/flutter ( 6002): Consider enabling the flag chain-stack-traces to receive more detailed exceptions.
I/flutter ( 6002): For example, 'dart test --chain-stack-traces'.
I/flutter ( 6002): 00:04 +0 -2: Testing App Performance Tests (tearDownAll)
I/flutter ( 6002): 00:04 +1 -2: Some tests failed.
Failure Details:
Failure in method: Scrolling test

end of failure 1


Failure in method: Favorites operations test

end of failure 2

Flutter doctor output:

[✓] Flutter (Channel dev, 2.6.0-11.0.pre, on macOS 11.6 20G165 darwin-x64,
    locale en)
    • Flutter version 2.6.0-11.0.pre at /Users/taodong/Code/flutter/flutter
    • Upstream repository https://github.com/flutter/flutter.git
    • Framework revision 4b330ddbed (5 weeks ago), 2021-09-16 17:29:58 -0700
    • Engine revision 5b81c6d615
    • Dart version 2.15.0 (build 2.15.0-116.0.dev)

[✓] Android toolchain - develop for Android devices (Android SDK version 30.0.2)
    • Android SDK at /Users/taodong/Library/Android/sdk
    • Platform android-30, build-tools 30.0.2
    • Java binary at: /Applications/Android
      Studio.app/Contents/jre/Contents/Home/bin/java
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)
    • All Android licenses accepted.

[✓] Xcode - develop for iOS and macOS (Xcode 12.3)
    • Xcode at /Applications/Xcode.app/Contents/Developer
    • CocoaPods version 1.10.1

[✓] Chrome - develop for the web
    • Chrome at /Applications/Google Chrome.app/Contents/MacOS/Google Chrome

[✓] Android Studio (version 2020.3)
    • Android Studio at /Applications/Android Studio.app/Contents
    • Flutter plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/9212-flutter
    • Dart plugin can be installed from:
      🔨 https://plugins.jetbrains.com/plugin/6351-dart
    • Java version OpenJDK Runtime Environment (build 11.0.10+0-b96-7281165)

[✓] VS Code (version 1.61.1)
    • VS Code at /Applications/Visual Studio Code.app/Contents
    • Flutter extension version 3.27.0

[✓] Connected device (3 available)
    • SM J710MN (mobile) • 5203ad88615794a7 • android-arm    • Android 8.1.0 (API 27)
    • macOS (desktop)    • macos            • darwin-x64     • macOS 11.6 20G165 darwin-x64
    • Chrome (web)       • chrome           • web-javascript • Google Chrome 95.0.4638.54

• No issues found!

[abd99]

It was running fine a few weeks ago. Now it's failing on mine too. I'm on Flutter (Channel stable, 2.5.3, on macOS 11.6 20G165 darwin-arm, locale en-US).

It is still running using the flutter test command. Eg. flutter test integration_test or flutter test integration_test/perf_test.dart I'll update the README soon.

@dnfield, do you know if we are removing support for the flutter drive command? If yes, how are we supposed to use custom drivers? And what about running the tests in profile mode?

[dnfield]

This should work. Seems like the host side isn't connecting to the device side properly

[InMatrix]

I agree that the error message suggests a connectivity issue:

D/libGLESv2( 6002): STS_GLApi : DTS, ODTC are not allowed for Package : dev.flutter.testing_app
VMServiceFlutterDriver: Connected to Flutter application.
I/flutter ( 6002): SocketException: OS Error: Connection refused, errno = 111, address = localhost, port
= 57506

However, I was able to flutter run the sample on the same device just fine. Does the flutter drive command connect to the device in a different way? I just tried it with another device (Nexus P), but the results were the same.

[redbrogdon]

CC @domesticmouse

[domesticmouse]

@dnfield should we raise this as an issue on flutter/flutter?

[dnfield]

Ah! The problem here is that DDS is enabled.

We should probably call that out if this error occurs.

Can you try the command again with --no-dds?

[InMatrix]

Can you try the command again with --no-dds?

This worked for me! Thanks, @dnfield

Should DDS be disabled by default for a profile build?

[dnfield]

No, but it should probably be disabled by default for flutter drive invocations.

This is aweird case where DDS wants to take over the observatory URI that the test is aware of.

[domesticmouse]

I'm closing out this issue as it appears resolved. Please comment to re-open if further action is required.