maestro
maestro copied to clipboard
mobile-dev-inc
App (React Native) seems to become unresponsive when running with Maestro
I'm finding that with certain test runs, the test will get to a certain point, then the app will immediately freeze (i.e. it won't crash, but animations will stop and it will become unresponsive). Eventually maestro will timeout & fail the test.
What's weird is this exact test suite was working reliably yesterday, and a different one was failing repeatedly.
I'm guessing it's when I get to something consuming more memory/CPU:
- Yesterday it happened when opening up a deep link from app close & navigating to a screen => I guess lots things happening at once
- Today it's freezing on a loading placeholder screen while navigating => Quite a bit of data fetched in the background, but not a lot on the screen
If I follow the exact same steps manually on the simulator (without maestro running) everything works reliably & smoothly every time.
There's no visible error, but if I rebuild Xcode after the crash I get a debug report:
-------------------------------------
Translated Report (Full Report Below)
-------------------------------------
Incident Identifier: 3A1127B9-D600-41D4-80BA-3BAA935E475F
CrashReporter Key: 5CDB127C-9FD0-635F-E716-77C74B52A34C
Hardware Model: MacBookPro18,3
Process: my_app [3211]
Path: /Users/USER/Library/Developer/CoreSimulator/Devices/71BBA01C-2A5D-42CF-BC1C-1D14D1C3C031/data/Containers/Bundle/Application/15AC1364-CC10-43D5-B01C-D39BB5E504BB/my_app.app/my_app
Identifier: com.my_app
Version: 0.0.0 (1)
Code Type: X86-64 (Native)
Role: Foreground
Parent Process: launchd_sim [97381]
Coalition: com.apple.CoreSimulator.SimDevice.71BBA01C-2A5D-42CF-BC1C-1D14D1C3C031 [30920]
Responsible Process: SimulatorTrampoline [37118]
Date/Time: 2023-02-16 19:05:34.3661 +0000
Launch Time: 2023-02-16 18:56:56.0366 +0000
OS Version: macOS 12.5.1 (21G83)
Release Type: User
Report Version: 104
Exception Type: EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Termination Reason: FRONTBOARD 2343432205
<RBSTerminateContext| domain:10 code:0x8BADF00D explanation:[application<com.my_app>:3211] failed to terminate gracefully after 5.0s
ProcessVisibility: Foreground
ProcessState: Running
WatchdogEvent: process-exit
WatchdogVisibility: Foreground
WatchdogCPUStatistics: (
"Elapsed total CPU time (seconds): 13.610 (user 10.590, system 3.020), 27% CPU",
"Elapsed application CPU time (seconds): 6.479, 13% CPU"
) reportType:CrashLog maxTerminationResistance:Interactive>
Triggered by Thread: 0
Thread 0 Crashed:: Dispatch queue: com.apple.accessibility.AXSideStorageQueue
0 <translation info unavailable> 0x106a062ac ???
1 <translation info unavailable> 0x106a06210 ???
2 libobjc.A.dylib 0x1104b8089 objc_retainAutoreleasedReturnValue + 45
3 AccessibilityUtilities 0x14fa1f78f __53-[NSObject(AXSideStorage) _accessibilityValueForKey:]_block_invoke + 47
4 libdispatch.dylib 0x1157b0a3a _dispatch_client_callout + 8
5 libdispatch.dylib 0x1157c114d _dispatch_lane_barrier_sync_invoke_and_complete + 92
6 AccessibilityUtilities 0x14fa1f4a7 -[NSObject(AXSideStorage) _accessibilityValueForKey:] + 615
7 UIAccessibility 0x152f239ac -[NSObject(UIStorage) _accessibilityGetBlockForAttribute:] + 44
8 UIAccessibility 0x152f207d7 -[NSObjectAccessibility accessibilityLabel] + 55
9 my_app 0x102c661c1 RCTRecursiveAccessibilityLabel + 369 (RCTView.m:86)
10 my_app 0x102c6601d -[RCTView accessibilityLabel] + 109 (RCTView.m:225)
11 my_app 0x102c661c1 RCTRecursiveAccessibilityLabel + 369 (RCTView.m:86)
...
76 AXRuntime 0x13b7056f8 _handleNonMainThreadCallback + 424
77 AXRuntime 0x13b705e36 _AXXMIGCopyParameterizedAttributeValue + 662
78 AXRuntime 0x13b77c0b5 _XCopyParameterizedAttributeValue + 693
79 AXRuntime 0x13b723a1d mshMIGPerform + 1021
80 CoreFoundation 0x1171c87f4 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ + 41
81 CoreFoundation 0x1171c7d3b __CFRunLoopDoSource1 + 538
82 CoreFoundation 0x1171c2538 __CFRunLoopRun + 2740
83 CoreFoundation 0x1171c16a7 CFRunLoopRunSpecific + 560
84 GraphicsServices 0x11d03a28a GSEventRunModal + 139
85 UIKitCore 0x127bf1ad3 -[UIApplication _run] + 994
86 UIKitCore 0x127bf69ef UIApplicationMain + 123
87 my_app 0x10216f748 main + 104 (main.m:15)
88 dyld_sim 0x10f02c2bf start_sim + 10
89 dyld 0x206c9a52e start + 462
Thread 1:: com.apple.rosetta.exceptionserver
0 ??? 0x7ff7ffec5944 ???
1 ??? 0x7ff7ffede1f0 ???
Thread 2:: com.apple.uikit.eventfetch-thread
0 ??? 0x1068dd940 ???
1 <translation info unavailable> 0x106999954 ???
2 libsystem_kernel.dylib 0x11bb83ce8 mach_msg + 56
3 CoreFoundation 0x1171c788e __CFRunLoopServiceMachPort + 145
4 CoreFoundation 0x1171c1fdf __CFRunLoopRun + 1371
5 CoreFoundation 0x1171c16a7 CFRunLoopRunSpecific + 560
6 Foundation 0x115dc48b4 -[NSRunLoop(NSRunLoop) runMode:beforeDate:] + 213
7 Foundation 0x115dc4b2d -[NSRunLoop(NSRunLoop) runUntilDate:] + 72
8 UIKitCore 0x127cc7286 -[UIEventFetcher threadMain] + 535
9 Foundation 0x115dee11b __NSThread__start__ + 1009
10 libsystem_pthread.dylib 0x11bc314e1 _pthread_start + 125
11 libsystem_pthread.dylib 0x11bc2cf6b thread_start + 15
Thread 3:: com.facebook.react.JavaScript
0 ??? 0x1068dd940 ???
1 <translation info unavailable> 0x106999954 ???
2 libsystem_kernel.dylib 0x11bb83ce8 mach_msg + 56
3 CoreFoundation 0x1171c788e __CFRunLoopServiceMachPort + 145
4 CoreFoundation 0x1171c1fdf __CFRunLoopRun + 1371
5 CoreFoundation 0x1171c16a7 CFRunLoopRunSpecific + 560
6 oban 0x102b81715 +[RCTCxxBridge runRunLoop] + 949 (RCTCxxBridge.mm:378)
7 Foundation 0x115dee11b __NSThread__start__ + 1009
8 libsystem_pthread.dylib 0x11bc314e1 _pthread_start + 125
9 libsystem_pthread.dylib 0x11bc2cf6b thread_start + 15
Versions: React Native: 0.68.5 Maestro CLI: 1.23.0 Xcode 14.2 Running on Mac M1 16GB Memory
I'm guessing this is something to do with my machine struggling with running both the app and the test runner? But not sure if there's any way of addressing. Anything else I can provide to help debug the issue?
Not sure if it's related, but I've also now started getting this in a similar part of the test:
Exception: Failed to connect to localhost/127.0.0.1:22087
xcuitest.XCTestDriverClient$XCTestDriverUnreachable: Failed to reach out XCUITest Server
at xcuitest.XCTestDriverClient$okHttpClient$2.invoke$lambda-2(XCTestDriverClient.kt:75)
This is happening every time I try and scroll (either scroll or scrollUntilVisible) on a certain screen. Again works perfectly when just scrolling manually on the simulator.
Not sure if this is a different symptom of the same problem, or a separate issue entirely. I can open up a different bug report if you'd prefer.
Hey @matt-dalton can you please try again on the latest Maestro version? We've made a lot of improvements lately and would like to verify whether this is still an issue. Thank you for the help!
Hey @Leland-Takamine - thanks for fixing. I'm currently on some other projects but will hopefully get back to this in a couple of weeks.
I'm on the latest release of maestro and I've seen this a few times as well. ActivityIndicators will just freeze and the app hangs. Sometimes waitForAnimationToEnd helps it get through that but not always.
I'm having the same issue on latest Maestro.
Our react native app will freeze on loading screens a second or so after the loading animation starts. Is there any update or workaround?
Joining the club. Same issue. Will be glad to provide any helpful details to help to resolve this issue.
My observations: Once the test gets failed the app in iOS simulator hits 100% of CPU load. A couple of minutes later the app gets unfreeze and CPU load becomes normal. Though test by this moment has already died.
Error from maestro log file:
[ERROR] maestro.cli.runner.TestRunner - Failed to run flow
java.lang.IllegalStateException: Unable to obtain active app id
at ios.xctest.XCTestIOSDevice.contentDescriptor(XCTestIOSDevice.kt:57)
at ios.LocalIOSDevice.contentDescriptor(LocalIOSDevice.kt:35)
at maestro.drivers.IOSDriver.contentDescriptor(IOSDriver.kt:184)
at maestro.ViewHierarchy$Companion.from-c1iYVAs(ViewHierarchy.kt:29)
at maestro.Maestro.viewHierarchy-prqvCes(Maestro.kt:416)
at maestro.orchestra.Orchestra.findElement(Orchestra.kt:770)
at maestro.orchestra.Orchestra.evaluateCondition(Orchestra.kt:463)
at maestro.orchestra.Orchestra.assertConditionCommand(Orchestra.kt:253)
at maestro.orchestra.Orchestra.executeCommand(Orchestra.kt:196)
at maestro.orchestra.Orchestra.executeCommands(Orchestra.kt:155)
at maestro.orchestra.Orchestra.runFlow(Orchestra.kt:98)
at maestro.cli.runner.MaestroCommandRunner.runCommands(MaestroCommandRunner.kt:179)
at maestro.cli.runner.TestRunner$runSingle$result$1.invoke(TestRunner.kt:45)
at maestro.cli.runner.TestRunner$runSingle$result$1.invoke(TestRunner.kt:42)
at maestro.cli.runner.TestRunner.runCatching(TestRunner.kt:138)
at maestro.cli.runner.TestRunner.runSingle(TestRunner.kt:42)
at maestro.cli.command.TestCommand$call$1.invoke(TestCommand.kt:166)
at maestro.cli.command.TestCommand$call$1.invoke(TestCommand.kt:126)
at maestro.cli.session.MaestroSessionManager.newSession(MaestroSessionManager.kt:99)
at maestro.cli.session.MaestroSessionManager.newSession$default(MaestroSessionManager.kt:57)
at maestro.cli.command.TestCommand.call(TestCommand.kt:126)
at maestro.cli.command.TestCommand.call(TestCommand.kt:44)
at picocli.CommandLine.executeUserObject(CommandLine.java:1933)
at picocli.CommandLine.access$1200(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2332)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2326)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2291)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2159)
at maestro.cli.DisableAnsiMixin$Companion.executionStrategy(DisableAnsiMixin.kt:22)
at picocli.CommandLine.execute(CommandLine.java:2058)
at maestro.cli.AppKt.main(App.kt:136)
Dev environment:
react-native info
info Fetching system and libraries information...
System:
OS: macOS 13.4
CPU: (12) x64 Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
Memory: 22.93 MB / 16.00 GB
Shell: 5.9 - /bin/zsh
Binaries:
Node: 14.19.0 - ~/.nvm/versions/node/v14.19.0/bin/node
Yarn: 1.22.19 - /usr/local/bin/yarn
npm: 6.14.17 - ~/.nvm/versions/node/v14.19.0/bin/npm
Watchman: 2023.05.22.00 - /usr/local/bin/watchman
Managers:
CocoaPods: 1.11.2 - /usr/local/bin/pod
SDKs:
iOS SDK:
Platforms: DriverKit 22.4, iOS 16.4, macOS 13.3, tvOS 16.4, watchOS 9.4
Android SDK:
API Levels: 16, 28, 29, 30, 31, 32, 33
Build Tools: 28.0.3, 29.0.2, 30.0.2, 30.0.3, 31.0.0, 32.0.0, 33.0.0
System Images: android-29 | Android TV Intel x86 Atom, android-29 | Intel x86 Atom_64, android-29 | Google APIs Intel x86 Atom, android-29 | Google Play Intel x86 Atom, android-30 | Google APIs Intel x86 Atom
Android NDK: Not Found
IDEs:
Android Studio: 2022.2 AI-222.4459.24.2221.9971841
Xcode: 14.3/14E222b - /usr/bin/xcodebuild
Languages:
Java: 12.0.1 - /usr/bin/javac
npmPackages:
@react-native-community/cli: Not Found
react: 18.2.0 => 18.2.0
react-native: 0.71.7 => 0.71.7
react-native-macos: Not Found
npmGlobalPackages:
*react-native*: Not Found
P.S. As a quick temporary solution you can manually launch simulator and open the app, modify your test as follows and start test:
- launchApp:
stopApp: false
Same issue here. 😢
P.S. As a quick temporary solution you can manually launch simulator and open the app, modify your test as follows and start test:
This just fixes the issue with launchApp sadly. @Dalamar
@Leland-Takamine Any schedule to fix this by any chance? We were just about to setup maestro cloud but if all our tests get stuck, it's not really worth the effort.
@matt-dalton @keeslinp @konstantinlindner @Dalamar @bobinrinder Thank you all for bumping this issue back up! If any of you are able to send over a simulator build (Feel free to DM me on Slack) along with repro steps that would be very much appreciated!
Hey @matt-dalton @keeslinp @konstantinlindner @Dalamar @bobinrinder ,
Thank you for your valuable feedback!
Upon investigating the issue, we identified multiple contributing factors. Primarily, the main thread seems to get overwhelmed by certain XCTest API calls when specific screens are presented (notably during the presentation of the 'fit52' loading screen with an animation). Although XCTest has an internal API, waitForQuiescenceIncludingAnimationsIdle, meant to prevent XCTest queries when the app isn't idle, it appears ineffective in some scenarios.
Recommendations:
- Disable Animations: Turn off animations for screens that exhibit freezing behavior.
- Use View Flattening: For ReactNative apps, consider implementing View Flattening.
- Main Thread Usage: Ensure that the main thread is exclusively reserved for UI-related logic. Tools like the Instruments Time Profiler and the combination of Thread Sanitizer + Main Thread Checker (available in Debug scheme settings) can be instrumental in this verification.
Additionally, it would immensely aid our investigation if you could share more app examples, flow cases, maestro cloud upload links, where this issue arises. Please feel free to share these details with me or @Leland-Takamine privately in our Maestro Slack channel.
Should you have further queries or concerns, don't hesitate to reach out. We're here to help!
Hey @artem888 @Leland-Takamine
I believe can reliably reproduce this issue on screens that are using react-native-maps with a bunch of <Marker /> components. Seems to work okay-ish with only a few markers but when I'm running e2e tests against a production dataset with just under 300 markers, the app grinds to a hault and maestro studio refuses to open.
I've created a similar reproduction here: https://github.com/eeston/maestro-issue
Hope this is useful!
