getsentry
Fix flaky UI Tests Critical
Description
The UI tests critical sometimes fail: https://github.com/getsentry/sentry-cocoa/actions/runs/15692735287/job/44211425971
[22:10:39]: ▸ ✔ testCrash (59.618 seconds)
[22:10:39]: ▸ Executed 2 tests, with 1 failure (0 unexpected) in 515.488 (515.494) seconds
[22:10:39]: ▸ Test Suite 'SwiftUITestSampleUITests.xctest' failed at 2025-06-16 22:10:39.114.
[22:10:39]: ▸ Executed 2 tests, with 1 failure (0 unexpected) in 515.488 (515.497) seconds
[22:10:39]: ▸ Test Suite 'All tests' failed at 2025-06-16 22:10:39.115.
[22:10:39]: ▸ Executed 2 tests, with 1 failure (0 unexpected) in 515.488 (515.499) seconds
[22:11:00]: ▸ 2025-06-16 22:11:00.651 xcodebuild[6665:36592] [MT] IDETestOperationsObserverDebug: 687.277 elapsed -- Testing started completed.
[22:11:00]: ▸ 2025-06-16 22:11:00.651 xcodebuild[6665:36592] [MT] IDETestOperationsObserverDebug: 0.000 sec, +0.000 sec -- start
[22:11:00]: ▸ 2025-06-16 22:11:00.652 xcodebuild[6665:36592] [MT] IDETestOperationsObserverDebug: 687.277 sec, +687.277 sec -- end
[22:11:01]: ▸ Testing failed:
[22:11:01]: ▸ Simulator device failed to launch io.sentry.tests.SwiftUITestSample.
[22:11:01]: ▸ Failing tests:
[22:11:01]: ▸ SwiftUITestSampleUITests.testCorruptedEnvelope()
[22:11:01]: ▸ ** TEST FAILED **
[22:11:04]: Exit status: 65
We need to identify the root cause and fix it, because if testCorruptedEnvelope is actually broken the damage would be fatal. This test validates that the SDK doesn't crash when reading a corrupted envelope.
One idea from @noahsmartin is to let the one for crashing the application run in its own GH action job, because crashing the app can mess up the other UI tests.
From Cocoa Sync 06/18:
A potential fix for this issue could be switching the CI provider
It also fails for SwiftUITestSampleUITests.testCrash() here: https://github.com/getsentry/sentry-cocoa/actions/runs/15879487502/job/44775669032
@noahsmartin, what we can already do now is to have a sample app that crashes based on an env variable instead of using XCTest and then validate a stored crash report. I created a draft PR for this. Please let me know what you think: https://github.com/getsentry/sentry-cocoa/pull/5512
Calling the Sentry API to validate that the SDK sent a proper crash would be the next step.