rules_swift_package_manager icon indicating copy to clipboard operation
rules_swift_package_manager copied to clipboard

Address UI tests for `shake_ios_example` and `stripe_example` failing in CI for Xcode 15

Open cgrindel opened this issue 9 months ago • 1 comments

The UI tests fail in CI using Xcode 15.

Example error output:

[2023-11-19 15:46:38 +0000] Status=4, isTerminal=NO, Elapsed=05:01.
	Waiting on System App

[2023-11-19 15:46:38 +0000] Status=4294967295, isTerminal=YES, Elapsed=05:03.
	Finished


note: Using 'xcodebuild' because test host was provided
Command line invocation:
    /Applications/Xcode_15.0.1.app/Contents/Developer/usr/bin/xcodebuild test-without-building -destination-timeout 15 -xctestrun /var/folders/q4/3s60g5510ydbd3qc3p4xflm80000gn/T//test_tmp_dir.JzjB2z/tests.xctestrun -destination id=B9952EFE-5056-4FB6-AA8B-B2F550BCEF40

User defaults from command line:
    IDEPackageSupportUseBuiltinSCM = YES

2023-11-19 16:08:15.154 xcodebuild[14412:83671]  IDETestOperationsObserverDebug: Failure collecting diagnostics from simulator: Timed out after 600.0 seconds while waiting for a response from the invoked process
2023-11-19 16:08:15.429 xcodebuild[14412:75441] [MT] IDETestOperationsObserverDebug: 1235.302 elapsed -- Testing started completed.
2023-11-19 16:08:15.429 xcodebuild[14412:75441] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
2023-11-19 16:08:15.429 xcodebuild[14412:75441] [MT] IDETestOperationsObserverDebug: 1235.302 sec, +1235.302 sec -- end

Test session results, code coverage, and logs:
	/Users/runner/Library/Developer/Xcode/DerivedData/temporary-aatadcpmwpdhhsbcwmbshnmcgyoi/Logs/Test/Test-Transient Testing-2023.11.19_15-47-39-+0000.xcresult

Testing failed:
	PaymentSheetUITest-Runner (15545) encountered an error (Early unexpected exit, operation never finished bootstrapping - no restart will be attempted. (Underlying Error: Test crashed with signal kill before starting test execution.))

** TEST EXECUTE FAILED **

Testing started
error: tests exited with '65'

#760 was implemented to workaround the CI failures. It updated the set_up_macos GH action to check for the two problematic tests and run them on Xcode 14.3.1.

The goal of this issue is two-fold:

  • Revert the changes to set_up_macos action from #760.

and one of the following:

  1. Fix the underlying issue with the UI tests on Xcode 15.
  2. Update the CI workflow generation to include an Xcode version.

cgrindel avatar Nov 20 '23 13:11 cgrindel