Xcode 16 Degraded Performance Since 9/25

[Nathan-Molby]

Description

Xcode 16 seems to have had degraded performance on MacOS 14 since 9/25. We updated our pipelines to Xcode 16 on 9/23, and saw consistent successful test runs (unit and UI tests) on both 9/23 and 9/24.

Starting on 9/25, however, we have identified three separate degradations with Xcode 16.

1. UI tests fail consistently. The same UI tests that were working on 9/24 now fail every time since 9/25.
2. Unit and UI tests take ~50% longer to run. Before 9/25, our tests suite took about 20 minutes to run. Now it takes 30+ minutes.
3. Xcodebuild fails to find a simulator on this command due to an inability to find the iPhone 15 18.0 simulator:

xcodebuild -showBuildSettings -scheme ******** -project ********* -configuration Debug -destination platform\=iOS\ Simulator,name\=iPhone\ 15,OS\=18.0

On 9/23 and 9/24, this command succeeded and found the iPhone 15 18.0 simulator.

Due to the correlation of these three issues on the same date, I am fairly confident that a runner upgrade resulted in degraded performance for Xcode 16.

Platforms affected

• [ ] Azure DevOps
• [X] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 20.04
• [ ] Ubuntu 22.04
• [ ] Ubuntu 24.04
• [ ] macOS 12
• [ ] macOS 13
• [ ] macOS 13 Arm64
• [ ] macOS 14
• [X] macOS 14 Arm64
• [ ] macOS 15
• [ ] macOS 15 Arm64
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Image version and build link

Version of the successful runner: macos-14-arm64 20240918.8 Version of the failing runner: macos-14-arm64 20240922.1

Both runners indicate that they are version 2.319.1.

Is it regression?

Yes. Last successful runner was version 20240918.8

Expected behavior

1. UI tests succeed as they did on 9/24
2. Unit and UI tests take the same amount of time they did on 9/24
3. Fastlane scan succeeds

Actual behavior

1. UI tests fail consistently. The same UI tests that were working on 9/24 now fail every time since 9/25.
2. Unit and UI tests take ~50% longer to run. Before 9/25, our tests suite took about 20 minutes to run. Now it takes 30+ minutes.
3. Fastlane scan fails due to inability to find a simulator. On 9/24, this command:

Repro steps

Steps to reproduce the behavior:

1. Attempt to run a unit or UI test suite on MacOS 14 Xcode 16
2. Notice that it fails