Bug that tries to execute one previous scenario

[Yokoi-K]

Steps to reproduce

We have four scenarios running on Firebase TestLab. However, when executing the latter scenario, we are trying to execute one of the previous scenarios.

Implementation

Currently, I am implementing scenarios with the following configuration

• e2e/all_test.dart
  • "edit credit card"
  • "remove credit card"
  • "charge on credit card"
  • "register credit card".

The contents of those four scenarios are, In each of them, we run patrolTest. Finally, we run integration_test on a single file, all_test.dart

patrolTest(...)

Problem

While running the "register credit card" scenario, I am asked to run the previous scenario, "charge on credit card". As shown in the screenshot.

Hypothesis

I was looking at the logs on Firebase TestLab and found something puzzling.

When I run the first scenario, "edit credit card", I get the following log output

03-03 03:11:36.681:. I/PatrolServer(24844):. PatrolJUnitRunner.runDartTest(e2e.all_test edit credit card): Requested execution
03-03 03:11:36.681:. I/PatrolServer(24844): PatrolAppServiceClient.runDartTest(e2e.all_test edit credit card)
03-03 03:11:36.732: I/flutter(24844):. PatrolAppService.runDartTest(e2e.all_test edit credit card) called
03-03 03:11:36.732: I/flutter(24844):. PatrolAppService: requested execution of test "e2e.all_test edit credit card"
03-03 03:11:36.733: I/flutter(24844):. Patrol (native): configure() started

Now, when we run the second scenario, "remove credit card", we get the following log output

03-03 03:12:27.498: I/PatrolServer(28938):. PatrolJUnitRunner.runDartTest(e2e.all_test remove credit card): Requested execution
03-03 03:12:27.498: I/PatrolServer(28938):. PatrolAppServiceClient.runDartTest(e2e.all_test remove credit card)
03-03 03:12:27.525: I/flutter(28938):. PatrolAppService.runDartTest(e2e.all_test remove credit card) called
03-03 03:12:27.525: I/flutter(28938):. PatrolAppService: registered test "e2e.all_test edit credit card" was not matched by requested test "e2e.all_test remove credit card"
03-03 03:12:27.581: I/flutter(28938):. PatrolBinding: tearDown(): count: 1, results: 1   {edit credit card: success}
03-03 03:12:27.582: I/flutter(28938):. PatrolBinding: finished test e2e.all_test edit credit card, but it was not requested, so its status will not be reported back to the native side
03-03 03:12:27.592: I/flutter(28938): PatrolAppService: registered "e2e.all_test remove credit card"
03-03 03:12:27.592: I/flutter(28938): PatrolAppService: requested execution of test "e2e.all_test remove credit card"
03-03 03:12:27.592: I/flutter(28938): Patrol (native): configure() started

03-03 03:12:27.525: I/flutter(28938): PatrolAppService: registered test "e2e.all_test edit credit card" was not matched by requested test "e2e. all_test remove credit card"

As you can see, the first scenario "edit credit card" is requested in the second scenario "remove credit card" for some reason.

This phenomenon is puzzling! Is there any solution? Any information you can provide would be greatly appreciated.

Actual results

The scenario execution should be possible on Firebase TestLab as well as on Firebase TestLab normally.

Logs

Logs

<!-- Replace this line with your logs. Do not remove the backticks! -->

Patrol version

patrol: 2.3.2

Patrol Doctor output

Patrol Doctor output

Patrol CLI version: 2.2.2
Program adb found in /Users/yo/Library/Android/sdk/platform-tools/adb
Env var $ANDROID_HOME set to /Users/yo/.rbenv/shims:/Users/yo/.zplug/repos/zplug/zplug/bin:/Users/yo/.zplug/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/bin:/Library/Apple/usr/bin:/Users/yo/fvm/default/bin:/Users/yo/.pub-cache/bin:/Users/yo/Library/Android/sdk/platform-tools:/Users/yo/bin:/Users/yo/Library/Android/sdk
Program xcodebuild found in /usr/bin/xcodebuild
Program ideviceinstaller found in /opt/homebrew/bin/ideviceinstaller

Flutter Doctor output

Flutter Doctor output

[✓] Flutter (Channel stable, 3.13.9, on macOS 14.3.1 23D60 darwin-arm64, locale ja-JP)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.2)
[✓] Xcode - develop for iOS and macOS (Xcode 14.2)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.1)
[✓] VS Code (version 1.85.2)
[✓] Connected device (2 available)
[✓] Network resources

[bartekpacia]

You're using a really old version of patrol – 2.3.2. I strongly recommend updating to the latest one.

Re your bug: I remember that this type of problem usually was usually called by not passing nativeAutomation: true to patrolTest() function. In patrol v3 you'd use patrolIntegrationTest() instead.

[gcostaapps]

@bartekpacia sorry, don't know why I typed this version, I think it's because I was looking another issue with this version. I'm using the 3.5.2, so it was not expected to have this behavior.

[bartekpacia]

Can you share your test code?

This looks suspicious:

registered test "e2e.all_test edit credit card" was not matched by requested test "e2e. all_test remove credit card"

[lexxxel]

I have the same issue with patrol 3.6.1:

I/flutter ( 6494): The Dart VM service is listening on http://127.0.0.1:45391/3wrqztssBS4=/
I/flutter ( 6494): Patrol (native): NativeAutomatorClient created, port: 8081
I/flutter ( 6494): Patrol (native): initialize() started
I/flutter ( 6494): Patrol (native): initialize() succeeded
I/flutter ( 6494): PatrolBinding: Register Patrol service extensions
I/flutter ( 6494): Patrol (native): NativeAutomatorClient created, port: 8081
I/flutter ( 6494): Patrol (native): NativeAutomatorClient created, port: 8081
I/flutter ( 6494): 00:00 +0: patrol_test_explorer
I/flutter ( 6494): patrol_test_explorer: obtained Dart-side test hierarchy:
I/flutter ( 6494): -- group: ''
I/flutter ( 6494):      -- group: 'Test change notification settings'
I/flutter ( 6494):      -- group: 'Test morning survey'
I/flutter ( 6494):      -- group: 'Test Login and Patient Survey'
I/flutter ( 6494):      -- group: 'Test evening survey'
I/flutter ( 6494):      -- group: 'Test insert survey on different date(morning)'
I/flutter ( 6494):      -- group: 'Test insert survey on different date (evening)'
I/flutter ( 6494):      -- group: 'test privacy notice page'
I/flutter ( 6494):      -- group: 'Test order of surveys in main menu'
I/flutter ( 6494):      -- group: 'Test create of second surveys on the same day'
I/flutter ( 6494):      -- group: 'Test app reset'
I/flutter ( 6494):      -- group: 'Test reset SubmissionTarget'
I/flutter ( 6494):      -- group: 'Test change patient survey'
I/flutter ( 6494):      -- group: 'Test send email note in main menu'
I/flutter ( 6494):      -- group: 'Test send email'
I/flutter ( 6494):      -- group: 'Test local data export'
I/flutter ( 6494):      -- group: 'Test send feedback mail'
I/flutter ( 6494):      -- group: 'Test non existing QR-Code Id'
I/flutter ( 6494):      -- group: 'Test doc has no valid license'
I/flutter ( 6494): PatrolAppService started, address: 0.0.0.0, host: 0.0.0.0, port: 8082
I/flutter ( 6494): Patrol (native): markPatrolAppServiceReady() started
I/flutter ( 6494): 00:00 +1: golden_integration_test Test change notification settings
I/flutter ( 6494): PatrolAppService: registered "golden_integration_test Test change notification settings"
I/flutter ( 6494): Patrol (native): markPatrolAppServiceReady() succeeded
I/flutter ( 6494): PatrolAppService.listDartTests() called
I/flutter ( 6494): 2024-03-11T15:19:07.385656  0:00:00.005585 GET     [200] /listDartTests
I/flutter ( 6494): PatrolAppService.runDartTest(golden_integration_test Test change notification settings) called
I/flutter ( 6494): PatrolAppService: requested execution of test "golden_integration_test Test change notification settings"
I/flutter ( 6494): PatrolAppService: registered "golden_integration_test Test change notification settings"
I/flutter ( 6494): PatrolAppService: requested execution of test "golden_integration_test Test change notification settings"
I/flutter ( 6494): Patrol (native): configure() started
I/flutter ( 6494): Patrol (native): configure() succeeded
---- the relevant log after the first test finishes: ----
I/flutter ( 6494): PatrolBinding: tearDown(): count: 1, results: {Test change notification settings: success}
I/flutter ( 6494): PatrolBinding: finished test golden_integration_test Test change notification settings. Will report its status back to the native side
I/flutter ( 6494): PatrolBinding: tearDown(): test "Test change notification settings" in group "golden_integration_test Test change notification settings", passed: true
I/flutter ( 6494): PatrolAppService.markDartTestAsCompleted(): golden_integration_test Test change notification settings
I/flutter ( 6494): 2024-03-11T15:19:07.424441  0:01:21.196631 POST    [200] /runDartTest
I/flutter ( 6494): 01:21 +2: golden_integration_test Test morning survey
I/flutter ( 6494): PatrolAppService: registered "golden_integration_test Test morning survey"
I/flutter ( 6494): PatrolAppService: registered test "golden_integration_test Test morning survey" was not matched by requested test "golden_integration_test Test change notification settings"
I/flutter ( 6494): PatrolAppService: registered "golden_integration_test Test morning survey"
I/flutter ( 6494): PatrolAppService: registered test "golden_integration_test Test morning survey" was not matched by requested test "golden_integration_test Test change notification settings"
I/flutter ( 6494): PatrolBinding: tearDown(): count: 2, results: {Test change notification settings: success, Test morning survey: success}
I/flutter ( 6494): PatrolBinding: finished test golden_integration_test Test morning survey, but it was not requested, so its status will not be reported back to the native side
I/flutter ( 6494): 01:21 +3: golden_integration_test Test Login and Patient Survey
I/flutter ( 6494): [ERROR] 2024-03-11 15:20:28.626626   0:00:00.002403  POST    /runDartTest
I/flutter ( 6494): 'package:patrol/src/native/patrol_app_service.dart': Failed assertion: line 147 pos 12: '_testExecutionCompleted.isCompleted == false': is not true.
I/flutter ( 6494): dart:core                                                                 _AssertionError._throwNew
I/flutter ( 6494): package:patrol/src/native/patrol_app_service.dart 147:12                  PatrolAppService.runDartTest
I/flutter ( 6494): package:patrol/src/native/contracts/patrol_app_service_server.dart 27:28  PatrolAppServiceServer.handle
I/flutter ( 6494): PatrolAppService: registered "golden_integration_test Test Login and Patient Survey"
I/flutter ( 6494): PatrolAppService: registered test "golden_integration_test Test Login and Patient Survey" was not matched by requested test "golden_integration_test Test change notification settings"
I/flutter ( 6494): PatrolAppService: registered "golden_integration_test Test Login and Patient Survey"
I/flutter ( 6494): PatrolAppService: registered test "golden_integration_test Test Login and Patient Survey" was not matched by requested test "golden_integration_test Test change notification settings"
I/flutter ( 6494): PatrolBinding: tearDown(): count: 3, results: {Test change notification settings: success, Test morning survey: success, Test Login and Patient Survey: success}
I/flutter ( 6494): PatrolBinding: finished test golden_integration_test Test Login and Patient Survey, but it was not requested, so its status will not be reported back to the native side
I/flutter ( 6494): 01:21 +4: golden_integration_test Test evening survey

$ patrol doctor
Patrol doctor:
Patrol CLI version: 2.7.0
Flutter command: flutter 
  Flutter 3.16.9 • channel stable
Android: 
• Program adb found in /usr/bin/adb
• Env var $ANDROID_HOME set to /home/<redacted>/AndroidSDK/

[lexxxel]

OK I dug a bit deeper. I looks like there is something strange happening to the test groups. In my original code, I had no groups around tests and the resulted log was:

I/flutter ( 6494): -- group: ''
I/flutter ( 6494):      -- group: 'Test change notification settings'
I/flutter ( 6494):      -- group: 'Test morning survey'
I/flutter ( 6494):      -- group: 'Test Login and Patient Survey'
I/flutter ( 6494):      -- group: 'Test evening survey'
I/flutter ( 6494):      -- group: 'Test insert survey on different date(morning)'
I/flutter ( 6494):      -- group: 'Test insert survey on different date (evening)'
I/flutter ( 6494):      -- group: 'test privacy notice page'
I/flutter ( 6494):      -- group: 'Test order of surveys in main menu'
I/flutter ( 6494):      -- group: 'Test create of second surveys on the same day'
I/flutter ( 6494):      -- group: 'Test app reset'
I/flutter ( 6494):      -- group: 'Test reset SubmissionTarget'
I/flutter ( 6494):      -- group: 'Test change patient survey'
I/flutter ( 6494):      -- group: 'Test send email note in main menu'
I/flutter ( 6494):      -- group: 'Test send email'
I/flutter ( 6494):      -- group: 'Test local data export'
I/flutter ( 6494):      -- group: 'Test send feedback mail'
I/flutter ( 6494):      -- group: 'Test non existing QR-Code Id'
I/flutter ( 6494):      -- group: 'Test doc has no valid license'

with a group wraped around all patrolTestcalls, named after the test file:

I/flutter (13187): patrol_test_explorer: obtained Dart-side test hierarchy:
I/flutter (13187): -- group: ''
I/flutter (13187):      -- group: 'bla_test.dart'
I/flutter (13187):           -- test: 'Test morning survey'
I/flutter (13187):      -- group: 'golden tests'
I/flutter (13187):           -- test: 'Test Login and Patient Survey'
I/flutter (13187):           -- test: 'Test change notification settings'
I/flutter (13187):           -- test: 'Test morning survey'
I/flutter (13187):           -- test: 'Test evening survey'
I/flutter (13187):           -- test: 'Test insert survey on different date(morning)'
I/flutter (13187):           -- test: 'Test insert survey on different date (evening)'
I/flutter (13187):           -- test: 'test privacy notice page'
I/flutter (13187):           -- test: 'Test order of surveys in main menu'
I/flutter (13187):           -- test: 'Test create of second surveys on the same day'
I/flutter (13187):           -- test: 'Test app reset'
I/flutter (13187):           -- test: 'Test reset SubmissionTarget'
I/flutter (13187):           -- test: 'Test change patient survey'
I/flutter (13187):           -- test: 'Test send email note in main menu'
I/flutter (13187):           -- test: 'Test send email'
I/flutter (13187):           -- test: 'Test local data export'
I/flutter (13187):           -- test: 'Test send feedback mail'
I/flutter (13187):           -- test: 'Test non existing QR-Code Id'
I/flutter (13187):           -- test: 'Test doc has no valid license'

with the test_bundle.dart containing:

  // START: GENERATED TEST GROUPS
  group('bla_test', bla_test.main);
  group('golden_integration_test', golden_integration_test.main);
  // END: GENERATED TEST GROUPS

I do not unterstand why this groups are missing and where the root group: '' comes from in both cases.

[lexxxel]

I found my issue: I used runApp to start my application.

patrolTest('Test morning survey', ($) async {
    runApp(   // <<< runApp can not be called multiple times 
      app.initMultiRepoProviders(
        child: MainApp(),
      ),
    );
...

If I rewrite it with pumpWidgetAndSettle everything works

  patrolTest('Test morning survey', ($) async {
  await $.pumpWidgetAndSettle(
    app.initMultiRepoProviders(
      child: MainApp(),
    ),
  );

@Yokoi-K do you use runApp?

[Yokoi-K]

Sorry for the late reply. Yes, I'm using runApp.

I gave up on installing patrol due to other factors, but perhaps if I don't use runApp, I can avoid this problem.

[jBorkowska]

Closing as it looks like resolved. Feel free to reopen or create a new issue

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar problem, please file a new issue. Make sure to follow the template and provide all the information necessary to reproduce the issue.