packages
packages copied to clipboard
flutter
[Espresso] Create integration test and remove exemption
Update documentation for how exclusions for integration test works. Add integration test to espresso and remove exclusion. Update integration test to support multidex Update Android Gradle Plugin to more recent version and update gradle wrapper to match. Because of the update to AGP modify build.gradle to set namespace following the instructions found here https://developer.android.com/studio/build/configure-app-module#set-namespace. Remove existing test that I believe is skipped in the existing tooling.
Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
- [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities.
- [x] I read and followed the [relevant style guides] and ran the auto-formatter. (Unlike the flutter/flutter repo, the flutter/packages repo does use
dart format.) - [x] I signed the [CLA].
- [x] The title of the PR starts with the name of the package surrounded by square brackets, e.g.
[shared_preferences] - [x] I listed at least one issue that this PR fixes in the description above.
- [x] I updated
pubspec.yamlwith an appropriate new version according to the [pub versioning philosophy], or this PR is [exempt from version changes]. - [x] I updated
CHANGELOG.mdto add a description of the change, [following repository CHANGELOG style]. - [ ] I updated/added relevant documentation (doc comments with
///). - [x] I added new tests to check the change I am making, or this PR is [test-exempt].
- [x] All existing and new tests are passing.
There was already a file with the existing name so I added the word Integration to the file name.
Hmm tests are failing with
/tmp/cirrus-ci-build/packages/espresso/example/android/app/src/main/java/io/flutter/app/FlutterMultiDexApplication.java:17: error: package androidx.multidex does not exist
import androidx.multidex.MultiDex;
^
/tmp/cirrus-ci-build/packages/espresso/example/android/app/src/main/java/io/flutter/app/FlutterMultiDexApplication.java:25: error: cannot find symbol
MultiDex.install(this);
^
symbol: variable MultiDex
location: class FlutterMultiDexApplication
2 errors
FAILURE: Build failed with an exception.
But this pr does not doe anything wtih multidex. I need to investigate but any suggestions are welcome.
New test is passing locally with (from root)dart run ./script/tool/bin/flutter_plugin_tools.dart drive-examples --android --packages espresso
from packages/expresso/example/android ./gradlew app:assembleAndroidTest -Pverbose=true
But is still failing in ci.
Any ideas are welcome.
As an aside if you get "INSTALL_FAILED_VERIFICATION_FAILURE" then you can bypass it with adb shell settings put global verifier_verify_adb_installs 0
I can reproduce the new failure by running ./gradlew app:mergeDexDebug from packages/espresso/example/android
https://firebase.corp.google.com/u/0/project/flutter-cirrus/testlab/histories/bh.61e29d9ae01b62d4/matrices/7984669307152441086/executions/bs.cbe8c5c67d29ab9c/videos
Test is timing out and I cant figure out why.
Current plan is to debug by getting gcloud installed and trying to run gcloud firebase test android run --type instrumentation --app build/app/outputs/apk/debug/app-debug.apk --test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk --timeout 7m --results-bucket=gs://flutter_cirrus_testlab --results-dir=plugins_android_test/espresso/5839641094389760/d8d95d2c-8a7f-443d-886d-9b9aae3a2670/example/0/ --device model=redfin,version=30 --device model=starqlteue,version=26" or some variation of that command.
Install instructions https://cloud.google.com/sdk/docs/install-sdk#mac
local setup
// Build apk
packages/espresso/example
flutter build apk
// Build Test
packages/espresso/example/android
./gradlew app:test
My version of the command.
gcloud firebase test android run --type instrumentation --app build/app/outputs/flutter-apk/app-release.apk --test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk --timeout 7m --device model=starqlteue,version=26
Unable to use gcloud to trigger builds normal google accounts are not authorized to run/modify/create projects or use existing ones. From flutter-infra chatroom the suggestion is to use led. https://g3doc.corp.google.com/company/teams/flutter/infrastructure/playbook.md?cl=head#testing-repo-changes-with-led https://chromium.googlesource.com/chromium/src/+/HEAD/docs/infra/using_led.md Installed as part of depot_tools.
Running tests with flutter drive --driver test_driver/integration_test.dart --target integration_test/espresso_launch_test.dart and flutter test integration_test/espresso_launch_test.dart pass locally.
I think I can reproduce the hanging test by running /gradlew app:connectedAndroidTest -Ptarget=pwd/../integration_test/espresso_launch_test.dart
from the example directory
I have been able to show that if I modify expresso_launch_test to have expect(false, isTrue); we still get stuck waiting for the test to timeout leading me to believe that our dart code is not getting run when we use the gradle command from https://github.com/flutter/packages/pull/3367#issuecomment-1468785378 but is being run when we execute flutter test integration_test/espresso_launch_test.dart
It seems like I am getting the same behavior as these people but non of those solutions seem to work for me https://stackoverflow.com/questions/65412897/new-integration-test-package-just-shows-test-starting-android
Also similar to this issue https://github.com/flutter/flutter/issues/86173
@goderbauer Do you have any suggestions for what else I can try? (pinged because it seems you are a semi frequent contributor)
Specifically I am trying to understand why locally from packages/expresso/exampleflutter test integration_test/espresso_launch_test.dart passes but from packages/expressso/example/android ./gradlew app:connectedAndroidTest -Ptarget=pwd/../integration_test/espresso_launch_test.dart hangs on the "Test Starting" screen.
It may be useful to compare with a PR like https://github.com/flutter/plugins/pull/4200 that set up initial integration tests for some of our other plugins. It's getting to be a while ago so it's possible that some things have changed, but it's likely still pretty close.
@bparrishMines may also have advice here since he set up a lot of the missing harnesses when we did our push on filling in testing holes.
It may be useful to compare with a PR like flutter/plugins#4200 that set up initial integration tests for some of our other plugins. It's getting to be a while ago so it's possible that some things have changed, but it's likely still pretty close.
@bparrishMines may also have advice here since he set up a lot of the missing harnesses when we did our push on filling in testing holes.
One big thing I notice is that espresso does not actually have any dart code outside of the example directory. This means that unlike in the linked pr we cant verify the existence of a plugin.
It is possible that the test is hanging because an integration test that launches from dart is actually not the right way to integration test this package?
This means that unlike in the linked pr we cant verify the existence of a plugin.
That's fine, it can be entirely a no-op.
It is possible that the test is hanging because an integration test that launches from dart is actually not the right way to integration test this package?
It's definitely not the right way to actually test the package, but that's fine; the goal here is just to ensure in CI that the example launches.
This doc is a little old, but may be helpful: https://docs.google.com/document/d/1OKqAJ5rG_HPafO_E83-INtnwAKhoBuklyuJG_vauXHs/edit?usp=sharing
it appears I am being stuck on this line. https://github.com/flutter/flutter/blob/007ed2a2d8fb02ac483bd02517752938d0bf51ec/packages/integration_test/android/src/main/java/dev/flutter/plugins/integration_test/FlutterTestRunner.java#L71
Just finished talking to @jiahaog. Filtering adb for flutter logs we find this
03-16 11:14:22.576 32213 32276 I flutter : The Dart VM service is listening on http://127.0.0.1:41313/NfgYT7sggrY=/
03-16 11:14:22.806 32213 32261 I flutter : 00:00 +0: placeholder test
03-16 11:14:23.068 32213 32261 I flutter : 00:00 +1: (tearDownAll)
03-16 11:14:23.081 32213 32261 I flutter : Warning: integration_test plugin was not detected.
03-16 11:14:23.081 32213 32261 I flutter :
03-16 11:14:23.081 32213 32261 I flutter : If you're running the tests with `flutter drive`, please make sure your tests
03-16 11:14:23.081 32213 32261 I flutter : are in the `integration_test/` directory of your package and use
03-16 11:14:23.081 32213 32261 I flutter : `flutter test $path_to_test` to run it instead.
03-16 11:14:23.081 32213 32261 I flutter :
03-16 11:14:23.081 32213 32261 I flutter : If you're running the tests with Android instrumentation or XCTest, this means
03-16 11:14:23.081 32213 32261 I flutter : that you are not capturing test results properly! See the following link for
03-16 11:14:23.081 32213 32261 I flutter : how to set up the integration_test plugin:
03-16 11:14:23.081 32213 32261 I flutter :
03-16 11:14:23.081 32213 32261 I flutter : https://flutter.dev/docs/testing/integration-tests#testing-on-firebase-test-lab
03-16 11:14:23.081 32213 32261 I flutter :
03-16 11:14:23.082 32213 32261 I flutter : 00:00 +2: All tests passed!
That plus adding a log line to https://github.com/flutter/flutter/blob/007ed2a2d8fb02ac483bd02517752938d0bf51ec/packages/integration_test/android/src/main/java/dev/flutter/plugins/integration_test/IntegrationTestPlugin.java#L44 and not seeing that line get logged makes us believe the issue is one of plugin registration.
For postarity the warning code we see is generated here https://github.com/flutter/flutter/blob/09cc1c4cdabb69f7163632afba7e0298881fbc0b/packages/integration_test/lib/integration_test.dart#L66
It is likley that when I find my issue I should update the documentation here https://docs.flutter.dev/testing/integration-tests#testing-on-firebase-test-lab
Doc covering the different android testing nuances. https://docs.google.com/document/d/1OKqAJ5rG_HPafO_E83-INtnwAKhoBuklyuJG_vauXHs/edit#
gcloud firebase test android run --type instrumentation --app build/app/outputs/flutter-apk/app-release.apk --test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk --timeout 7m --device model=starqlteue,version=26 --project flutter-cirrus --results-bucket=gs://flutter_cirrus_testlab
will try to start the test after requesting access via go/flutter-firebase-on-demand-request and getting a member of flutter-infra to approve.
This command fails locally but i think it is because i am uploading the wrong artifacts.
gcloud firebase test android run --type instrumentation --app build/app/outputs/flutter-apk/app-debug.apk --test build/app/outputs/apk/androidTest/debug/app-debug-androidTest.apk --timeout 7m --device model=starqlteue,version=26 --project flutter-cirrus --results-bucket=gs://flutter_cirrus_testlab
Command started a test successfully.
https://firebase.corp.google.com/u/0/project/flutter-cirrus/testlab/histories/bh.61e29d9ae01b62d4/matrices/5019499127722619788
Current theory is that if I follow https://github.com/flutter/flutter/wiki/Plugin-Tests#enabling-android-ui-tests in addition to the super call being made that this should pass.
Existing tests fail in ci. https://firebase.corp.google.com/u/0/project/flutter-cirrus/testlab/histories/bh.61e29d9ae01b62d4/matrices/8681745030105464066/executions/bs.a49a6e43bb7bbcf0/test-cases
https://firebase.corp.google.com/u/0/project/flutter-cirrus/testlab/histories/bh.61e29d9ae01b62d4/matrices/8496841808769326759/executions/bs.71460c71b724876d/logs
Existing tests are failing and my new test is passing
03-16 13:30:22.986: I/TestRunner(22457): started: placeholder test(com.example.espresso_example.MainActivityIntegrationTest)
03-16 13:30:22.989: I/flutter(22457): 00:00 +2: All tests passed!
03-16 13:30:22.990: I/MonitoringInstr(22457): Finishing activity: com.example.espresso_example.MainActivity@f0d6731
03-16 13:30:23.000: I/AdrenoGLES-0(1637): QUALCOMM build : 4783c89, I46ff5fc46f
03-16 13:30:23.000: I/AdrenoGLES-0(1637): Build Date : 11/30/20
03-16 13:30:23.000: I/AdrenoGLES-0(1637): OpenGL ES Shader Compiler Version: EV031.31.04.01
03-16 13:30:23.000: I/AdrenoGLES-0(1637): Local Branch : QPR2
03-16 13:30:23.000: I/AdrenoGLES-0(1637): Remote Branch :
03-16 13:30:23.000: I/AdrenoGLES-0(1637): Remote Branch :
03-16 13:30:23.000: I/AdrenoGLES-0(1637): Reconstruct Branch :
03-16 13:30:23.000: I/AdrenoGLES-0(1637): Build Config : S P 10.0.4 AArch64
03-16 13:30:23.000: I/AdrenoGLES-0(1637): Driver Path : /vendor/lib64/egl/libGLESv2_adreno.so
03-16 13:30:23.001: I/MonitoringInstr(22457): Unstopped activity count: 1
03-16 13:30:23.003: D/LifecycleMonitor(22457): Lifecycle status change: com.example.espresso_example.MainActivity@f0d6731 in: PAUSED
To locally reproduce local failures run dart run ./script/tool/bin/flutter_plugin_tools.dart native-test --android --packages espresso from the root directory packages.
Checking out main and running dart run ./script/tool/bin/flutter_plugin_tools.dart native-test --android --packages espresso
Give the following error
============================================================
|| Running for packages/espresso
============================================================
Running tests for Android...
----------------------------------------
No Android unit tests found for packages/espresso/example
No Android integration tests found for packages/espresso/example
No unit tests ran. Plugins are required to have unit tests.
The following packages had errors:
packages/espresso:
No unit tests ran (use --exclude if this is intentional).
See above for full details.
I currently believe that the tests that are failing were skipped before possibly because of the exclude rule.