patrol Some of iOS tests are marked as failed but should be successful on physical iOS device

Steps to reproduce

Go to my branch ci/small_fixes https://github.com/leancodepl/patrol/pull/1468
Run patrol test --release --verbose -t integration_test/webview_leancode_test.dart in patrol example app on physical iOS device

Actual results

You can see:

PatrolServer: INFO: done tapping on view with text "Subscribe" in app pl.leancode.patrol.Example
 2023-07-06 15:46:27.633293+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: result: ()
/Users/fylyppo/Documents/GitHub/patrol/packages/patrol/example/ios/RunnerUITests/RunnerUITests.m:5: error: -[RunnerUITests webviewLeancodeTest] : ((response.passed) is true) failed

Tapping on "Subscribe" button is the last action in my test and it's done as you can see in logs so test should be successful. I have the same problem also in other test but I can't find any connections between them. All failing tests:

webview_leancode_test
open_quick_settings_test
service_airplane_mode_test
service_bluetooth_test
service_cellular_test
webview_hackernews_test
webview_stackoverflow_test

Last run on CI where all tests were successful was Sun, 25 Jun 2023 21:32:25 GMT

Logs

patrol test --release --verbose -t integration_test/webview_leancode_test.dart
You're using Patrol CLI 2.0, which has breaking changes.
Read the migration guide at https://patrol.leancode.co/v2.
Disable this warning by setting the PATROL_MIGRATED environment variable.

Verbose mode enabled. More logs will be printed.
Received 1 test target(s)
Received test target: /Users/fylyppo/Documents/GitHub/patrol/packages/patrol/example/integration_test/webview_leancode_test.dart
Generated entrypoint /Users/fylyppo/Documents/GitHub/patrol/packages/patrol/example/integration_test/test_bundle.dart with 1 bundled test(s)
$ flutter --no-version-check devices --machine
No device specified, using the first one (Julia’s iPhone)
Received 1 device(s) to run on
Received device: Julia’s iPhone
Received 7 --dart-define(s) (0 custom, 7 internal)
Received internal --dart-define: PATROL_WAIT=0
Received internal --dart-define: PATROL_APP_PACKAGE_NAME=pl.leancode.patrol.example
Received internal --dart-define: PATROL_APP_BUNDLE_ID=pl.leancode.patrol.Example
Received internal --dart-define: PATROL_ANDROID_APP_NAME=Patrol example
Received internal --dart-define: PATROL_IOS_APP_NAME=Patrol example
Received internal --dart-define: INTEGRATION_TEST_SHOULD_REPORT_RESULTS_TO_NATIVE=false
Received internal --dart-define: PATROL_TEST_LABEL_ENABLED=true
• Building app with entrypoint test_bundle.dart for iOS device (release)...
$ flutter build ios --no-version-check --config-only --no-codesign --release --target /Users/fylyppo/Documents/GitHub/patrol/packages/patrol/example/integration_test/test_bundle.dart --dart-define PATROL_WAIT=0 --dart-define PATROL_APP_PACKAGE_NAME=pl.leancode.patrol.example --dart-define PATROL_APP_BUNDLE_ID=pl.leancode.patrol.Example --dart-define PATROL_ANDROID_APP_NAME=Patrol example --dart-define PATROL_IOS_APP_NAME=Patrol example --dart-define INTEGRATION_TEST_SHOULD_REPORT_RESULTS_TO_NATIVE=false --dart-define PATROL_TEST_LABEL_ENABLED=true
        Warning: Building for device with codesigning disabled. You will have to manually codesign before deploying to device.
        Building pl.leancode.patrol.Example for device (ios-release)...
$ xcodebuild build-for-testing -workspace Runner.xcworkspace -scheme Runner -configuration Release -only-testing RunnerUITests -sdk iphoneos -destination generic/platform=iOS -quiet -derivedDataPath ../build/ios_integ OTHER_SWIFT_FLAGS=$(inherited) -D PATROL_ENABLED
        2023-07-06 15:45:28.167 xcodebuild[97250:1857258] DVTCoreDeviceEnabledState: DVTCoreDeviceEnabledState_Disabled set via user default (DVTEnableCoreDevice=disabled)
        note: Building targets in dependency order
        warning: Run script build phase 'xcode_backend build' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'Runner' from project 'Runner')
        warning: Run script build phase 'xcode_backend embed_and_thin' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'Runner' from project 'Runner')
        warning: Run script build phase 'xcode_backend build' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'RunnerUITests' from project 'Runner')
        warning: Run script build phase 'xcode_backend embed_and_thin' will be run during every build because it does not specify any outputs. To address this warning, either add output dependencies to the script phase, or configure it to run in every build by unchecking "Based on dependency analysis" in the script phase. (in target 'RunnerUITests' from project 'Runner')
✓ Completed building app with entrypoint test_bundle.dart for iOS device (24.4s)
• Running app with entrypoint test_bundle.dart for iOS device on device Julia’s iPhone...
$ xcodebuild -showsdks -json
Assuming SDK version 16.4 for iphonesimulator
Looking for .xctestrun matching Runner_iphoneos16.4*.xctestrun at /Users/fylyppo/Documents/GitHub/patrol/packages/patrol/example/build/ios_integ/Build/Products
Found 1 match(es), the first one will be used
Found /Users/fylyppo/Documents/GitHub/patrol/packages/patrol/example/build/ios_integ/Build/Products/Runner_iphoneos16.4-arm64.xctestrun
$ xcodebuild test-without-building -xctestrun /Users/fylyppo/Documents/GitHub/patrol/packages/patrol/example/build/ios_integ/Build/Products/Runner_iphoneos16.4-arm64.xctestrun -only-testing RunnerUITests -destination platform=iOS,name=Julia’s iPhone
        2023-07-06 15:45:44.862 xcodebuild[98847:1861379] DVTCoreDeviceEnabledState: DVTCoreDeviceEnabledState_Disabled set via user default (DVTEnableCoreDevice=disabled)
        Command line invocation:
            /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild test-without-building -xctestrun /Users/fylyppo/Documents/GitHub/patrol/packages/patrol/example/build/ios_integ/Build/Products/Runner_iphoneos16.4-arm64.xctestrun -only-testing RunnerUITests -destination "platform=iOS,name=Julia’s iPhone"

        User defaults from command line:
            IDEPackageSupportUseBuiltinSCM = YES

        2023-07-06 15:45:57.486533+0200 RunnerUITests-Runner[1779:432270] Running tests...
        2023-07-06 15:45:58.686892+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: PATROL_PORT is null, falling back to default (8081)
        2023-07-06 15:45:58.687087+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: PatrolServer constructor called
        2023-07-06 15:45:58.687157+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: PATROL_ENABLED flag is defined
        2023-07-06 15:45:58.687299+0200 RunnerUITests-Runner[1779:432270] PatrolAppServiceClient: created, port: 8082
        2023-07-06 15:45:58.700342+0200 RunnerUITests-Runner[1779:432523] [si_destination_compare] send failed: Invalid argument
        2023-07-06 15:45:58.700523+0200 RunnerUITests-Runner[1779:432523] [si_destination_compare] send failed: Undefined error: 0
            t =      nans Checking existence of `"Allow" Button`
        2023-07-06 15:45:58.705730+0200 RunnerUITests-Runner[1779:432523] PatrolServer: INFO: Server started on http://localhost:8081
            t =      nans Ignoring failure to get hierarchy for remote element in process 768 (Error getting main window kAXErrorServerNotFound)
            t =      nans Open pl.leancode.patrol.Example
            t =      nans     Launch pl.leancode.patrol.Example
            t =      nans         Setting up automation session
            t =      nans         Wait for pl.leancode.patrol.Example to idle
        2023-07-06 15:46:08.076340+0200 RunnerUITests-Runner[1779:432520] PatrolServer: INFO: App reported that it is ready
        2023-07-06 15:46:09.070537+0200 RunnerUITests-Runner[1779:432523] PatrolAppServiceClient.listDartTests()
        2023-07-06 15:46:10.074562+0200 RunnerUITests-Runner[1779:432270] Got 1 Dart tests: (
            "webview_leancode_test"
        )
        2023-07-06 15:46:10.076318+0200 RunnerUITests-Runner[1779:432270] RunnerUITests.testInvocations(): selectorName = webviewLeancodeTest, signature: <NSMethodSignature: 0x8115b23479f94db7>
        Test Suite 'All tests' started at 2023-07-06 15:46:10.082
        Test Suite 'RunnerUITests.xctest' started at 2023-07-06 15:46:10.085
        Test Suite 'RunnerUITests' started at 2023-07-06 15:46:10.087
        Test Case '-[RunnerUITests webviewLeancodeTest]' started.
            t =     0.00s Start Test at 2023-07-06 15:46:10.092
            t =     0.10s Set Up
            t =     0.10s Open pl.leancode.patrol.Example
            t =     0.18s     Launch pl.leancode.patrol.Example
            t =     0.18s         Terminate pl.leancode.patrol.Example:1790
            t =     2.04s         Setting up automation session
        2023-07-06 15:46:12.158187+0200 RunnerUITests-Runner[1779:432523] PatrolServer: INFO: App reported that it is ready
            t =     2.08s         Wait for pl.leancode.patrol.Example to idle
        2023-07-06 15:46:13.291531+0200 RunnerUITests-Runner[1779:432562] PatrolAppServiceClient.runDartTest(webview_leancode_test)
        2023-07-06 15:46:13.866942+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: tapping on view with text "Accept cookies" in app pl.leancode.patrol.Example...
        2023-07-06 15:46:13.868847+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: waiting for existence of view with text "Accept cookies"
            t =     3.78s Waiting 20.0s for "Accept cookies" Any to exist
            t =     4.81s     Checking `Expect predicate `exists == 1` for object "Accept cookies" Any`
            t =     4.81s         Checking existence of `"Accept cookies" Any`
        2023-07-06 15:46:16.962958+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: found view with text "Accept cookies", will tap on it
            t =     6.87s Find the "Accept cookies" Any
            t =     7.83s Tap "Accept cookies" Button
            t =     7.83s     Wait for pl.leancode.patrol.Example to idle
            t =     7.91s     Find the "Accept cookies" Button
            t =     8.71s     Check for interrupting elements affecting "Accept cookies" Button
            t =     9.08s     Synthesize event
            t =     9.23s     Wait for pl.leancode.patrol.Example to idle
        2023-07-06 15:46:19.435426+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: done tapping on view with text "Accept cookies" in app pl.leancode.patrol.Example
        2023-07-06 15:46:19.436570+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: result: ()
        2023-07-06 15:46:19.439940+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: entering text "[email protected]" by index 0 in app pl.leancode.patrol.Example...
            t =     9.35s Get all elements bound by index for: Elements matching predicate 'elementType == 49 OR elementType == 50'
            t =     9.74s Checking existence of `Any`
            t =    10.04s Find the Any
            t =    10.35s Tap "Type your email" TextField
            t =    10.35s     Wait for pl.leancode.patrol.Example to idle
            t =    10.42s     Find the "Type your email" TextField
            t =    10.78s     Check for interrupting elements affecting "Type your email" TextField
            t =    11.11s     Synthesize event
            t =    11.26s     Wait for pl.leancode.patrol.Example to idle
            t =    11.43s Type '[email protected]' into "Type your email" TextField
            t =    11.43s     Wait for pl.leancode.patrol.Example to idle
            t =    11.96s     Find the "Type your email" TextField
            t =    12.32s     Check for interrupting elements affecting "Type your email" TextField
            t =    12.69s     Synthesize event
            t =    13.07s     Wait for pl.leancode.patrol.Example to idle
        2023-07-06 15:46:23.240397+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: done entering text "[email protected]" by index 0 in app pl.leancode.patrol.Example
        2023-07-06 15:46:23.240539+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: result: ()
        2023-07-06 15:46:23.242822+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: tapping on view with text "Subscribe" in app pl.leancode.patrol.Example...
        2023-07-06 15:46:23.243381+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: waiting for existence of view with text "Subscribe"
            t =    13.15s Waiting 20.0s for "Subscribe" Any to exist
            t =    14.16s     Checking `Expect predicate `exists == 1` for object "Subscribe" Any`
            t =    14.16s         Checking existence of `"Subscribe" Any`
        2023-07-06 15:46:25.258025+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: found view with text "Subscribe", will tap on it
            t =    15.17s Find the "Subscribe" Any
            t =    15.96s Tap "Subscribe" Button
            t =    15.96s     Wait for pl.leancode.patrol.Example to idle
            t =    16.03s     Find the "Subscribe" Button
            t =    16.86s     Check for interrupting elements affecting "Subscribe" Button
            t =    17.29s     Synthesize event
            t =    17.44s     Wait for pl.leancode.patrol.Example to idle
        2023-07-06 15:46:27.633108+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: done tapping on view with text "Subscribe" in app pl.leancode.patrol.Example
        2023-07-06 15:46:27.633293+0200 RunnerUITests-Runner[1779:432270] PatrolServer: INFO: result: ()
        /Users/fylyppo/Documents/GitHub/patrol/packages/patrol/example/ios/RunnerUITests/RunnerUITests.m:5: error: -[RunnerUITests webviewLeancodeTest] : ((response.passed) is true) failed
            t =    18.68s Tear Down
        Test Case '-[RunnerUITests webviewLeancodeTest]' failed (18.886 seconds).
        Test Suite 'RunnerUITests' failed at 2023-07-06 15:46:28.977.
                 Executed 1 test, with 1 failure (0 unexpected) in 18.886 (18.890) seconds
        Test Suite 'RunnerUITests.xctest' failed at 2023-07-06 15:46:28.982.
                 Executed 1 test, with 1 failure (0 unexpected) in 18.886 (18.896) seconds
        Test Suite 'All tests' failed at 2023-07-06 15:46:28.985.
                 Executed 1 test, with 1 failure (0 unexpected) in 18.886 (18.903) seconds
        2023-07-06 15:46:29.102 xcodebuild[98847:1861379] [MT] IDETestOperationsObserverDebug: 41.176 elapsed -- Testing started completed.
        2023-07-06 15:46:29.102 xcodebuild[98847:1861379] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
        2023-07-06 15:46:29.102 xcodebuild[98847:1861379] [MT] IDETestOperationsObserverDebug: 41.176 sec, +41.176 sec -- end
        2023-07-06 15:46:29.224 xcodebuild[98847:1861379] [MT] IDEResultKit: Warning: While writing a result bundle to /Users/fylyppo/Library/Developer/Xcode/DerivedData/Runner-fzvimjnrizbvjsbixjaedstibtoo/Logs/Test/Test-Runner-2023.07.06_15-45-47-+0200.xcresult, not all contents have been imported, as the Staging directory still contains the following files/directories: ["1_Test"]

        Test session results, code coverage, and logs:
                /Users/fylyppo/Library/Developer/Xcode/DerivedData/Runner-fzvimjnrizbvjsbixjaedstibtoo/Logs/Test/Test-Runner-2023.07.06_15-45-47-+0200.xcresult

        Failing tests:
                -[RunnerUITests webviewLeancodeTest]

        ** TEST EXECUTE FAILED **

        Testing started
✗ Failed to execute tests of app with entrypoint test_bundle.dart for iOS device on device Julia’s iPhone (xcodebuild exited with code 65) (45.3s)
Error: xcodebuild exited with code 65
#0      throwToolExit (package:patrol_cli/src/base/exceptions.dart:7:3)
#1      IOSTestBackend.execute.<anonymous closure> (package:patrol_cli/src/ios/ios_test_backend.dart:177:9)
<asynchronous suspension>
#2      DisposeScope.run (package:dispose_scope/src/dispose_scope.dart:44:7)
<asynchronous suspension>
#3      IOSTestBackend.execute (package:patrol_cli/src/ios/ios_test_backend.dart:143:5)
<asynchronous suspension>
#4      TestCommand._execute (package:patrol_cli/src/commands/test.dart:232:7)
<asynchronous suspension>
#5      TestCommand.run (package:patrol_cli/src/commands/test.dart:168:23)
<asynchronous suspension>
#6      CommandRunner.runCommand (package:args/command_runner.dart:212:13)
<asynchronous suspension>
#7      PatrolCommandRunner.runCommand (package:patrol_cli/src/runner/patrol_command_runner.dart:310:18)
<asynchronous suspension>
#8      PatrolCommandRunner.run (package:patrol_cli/src/runner/patrol_command_runner.dart:254:18)
<asynchronous suspension>
#9      patrolCommandRunner (package:patrol_cli/src/runner/patrol_command_runner.dart:72:20)
<asynchronous suspension>
#10     main (file:///Users/fylyppo/Documents/GitHub/patrol/packages/patrol_cli/bin/main.dart:6:20)
<asynchronous suspension>

See the logs above to learn what happened. Also consider running with --verbose. If the logs still aren't useful, then it's a bug - please report it.
$ ideviceinstaller --udid dcea94d2d56e1961028a4bd71d12c6ceaa85808c --uninstall pl.leancode.patrol.Example
$ ideviceinstaller --udid dcea94d2d56e1961028a4bd71d12c6ceaa85808c --uninstall pl.leancode.patrol.Example.RunnerUITests.xctrunner

Patrol version

patrol 2.0.2

Patrol Doctor output

Patrol CLI version: 2.0.1
Program adb found in /opt/homebrew/bin/adb
Env var $ANDROID_HOME is not set
Program xcodebuild found in /usr/bin/xcodebuild
Program ideviceinstaller not found (install with `brew install ideviceinstaller`)

Flutter Doctor output

Doctor summary (to see all details, run flutter doctor -v):
[✓] Flutter (Channel stable, 3.10.5, on macOS 13.4.1 22F82 darwin-arm64, locale pl-PL)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.0)
[✓] Xcode - develop for iOS and macOS (Xcode 14.3.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2021.3)
[✓] VS Code (version 1.79.2)
[✓] Connected device (3 available)
[✓] Network resources

• No issues found!

Jul 06 '23 13:07 fylyppo

I downloaded logs and it looks like it's a problem with SemanticsHandles... again.

Exampe:

Jun 30 03:17:05 iPhone RunnerUITests-Runner(libswiftFoundation.dylib)[808] <Notice>: PatrolServer: INFO: done enabling bluetooth
Jun 30 03:17:05 iPhone RunnerUITests-Runner(libswiftFoundation.dylib)[808] <Notice>: PatrolServer: INFO: result: ()
...
Jun 30 03:17:05 iPhone Runner(Flutter)[859] <Notice>: flutter: A SemanticsHandle was active at the end of the test.
Jun 30 03:17:05 iPhone Runner(Flutter)[859] <Notice>: flutter: #0      WidgetTester._verifySemanticsHandlesWereDisposed (package:flutter_test/src/widget_tester.dart:1051)
Jun 30 03:17:05 iPhone Runner(Flutter)[859] <Notice>: flutter: #1      WidgetTester._endOfTestVerifications (package:flutter_test/src/widget_tester.dart:1042)
Jun 30 03:17:05 iPhone Runner(Flutter)[859] <Notice>: flutter: #2      TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:998)
Jun 30 03:17:05 iPhone Runner(Flutter)[859] <Notice>: flutter: <asynchronous suspension>
Jun 30 03:17:05 iPhone Runner(Flutter)[859] <Notice>: flutter: 00:17 +5: service_bluetooth_test disables and enables bluetooth twice [E]
Jun 30 03:17:05 iPhone Runner(Flutter)[859] <Notice>: flutter:   Test failed. See exception logs above.
  The test description was: disables and enables bluetooth twice
Jun 30 03:17:05 iPhone Runner(Flutter)[859] <Notice>: flutter:

Jul 10 '23 11:07 bartekpacia

ah-shit-here-we-go-again-ah-shit

Jul 10 '23 11:07 bartekpacia

The workaround has been implemented in #1491.

Jul 11 '23 09:07 bartekpacia

The workaround is good enough for now, but if possible, we should try to make a proper fix, like I did in the case of Android.