runner-images icon indicating copy to clipboard operation
runner-images copied to clipboard

[Xcode 26] Build time regression in Azure pipeline: 15 mins → 40+ mins with xcode@5

Open yuzhongqi opened this issue 5 months ago • 24 comments

Description

After upgrading from Xcode 16.1 to Xcode 26.0, the build step using xcode@5 in our Azure pipeline has experienced a significant regression.

With Xcode 16.1, the build consistently completed in about 15 minutes.

With Xcode 26.0, the same build now takes at least 40 minutes under the same pipeline configuration.

This large increase in build time indicates a potential performance regression in the Xcode 26 build environment, which directly impacts our CI/CD efficiency.

The Compilation Phase is the most time-consuming stage, which corresponds to the xcode@5 step. Here are the specific details I summarized from the build logs.

📊 Absolute Build Time Differences: Xcode 16.1 vs 26.0

Overall Time Comparison

  • Xcode 16.1 total duration: 14m 14s (854s)

  • Xcode 26.0 total duration: 40m 55s (2455s)

  • Total difference: +26m 41s (1601s)


🔍 Stage-by-Stage Breakdown

  1. Package Resolution

  • Xcode 16.1:

    • Start: 08:09:55

    • End: 08:12:47

    • Duration: 2m 52s (172s)

  • Xcode 26.0:

    • Start: 07:59:21

    • End: 08:06:52

    • Duration: 7m 31s (451s)

  • Absolute difference: +4m 39s (+279s)


  1. Compilation Phase

  • Xcode 16.1:

    • Start: 08:13:22 (≈35s after package resolution)

    • End: 08:24:07

    • Duration: 10m 45s (645s)

  • Xcode 26.0:

    • Estimated start: 08:07:30 (≈38s after package resolution)

    • End: 08:40:10

    • Duration (est.): 32m 40s (1960s)

  • Absolute difference: +21m 55s (+1315s)


  1. SDK Compatibility Overhead
    Based on analysis, the new SDK introduces extra processing time due to:

  • Longer module precompilation

  • Extended dependency adaptation

  • Estimated additional time: +10m 42s (+642s)


Platforms affected

  • [x] Azure DevOps
  • [ ] GitHub Actions - Standard Runners
  • [ ] GitHub Actions - Larger Runners

Runner images affected

  • [ ] Ubuntu 22.04
  • [ ] Ubuntu 24.04
  • [ ] macOS 13
  • [ ] macOS 13 Arm64
  • [ ] macOS 14
  • [ ] macOS 14 Arm64
  • [x] macOS 15
  • [ ] macOS 15 Arm64
  • [ ] macOS 26 Arm64
  • [ ] Windows Server 2019
  • [ ] Windows Server 2022
  • [ ] Windows Server 2025

Image version and build link

Image: macos-15 Version: 20250917.1898

Is it regression?

Yes

Expected behavior

I think the overall build process should take roughly the same amount of time as it did with Xcode 16.1, with a total duration of around 15 minutes and a pure packaging time of approximately 10 minutes.

Actual behavior

total duration of around at lest 40+mins and a pure packaging time of approximately 30 minutes.

Repro steps

On a pipeline that normally builds successfully with macOS 14 arm64 and Xcode 16.1: 1.Select macOS 15 2.Select Xcode 26 3.Download the Metal tool 4.Trigger this build pipeline

yuzhongqi avatar Sep 29 '25 06:09 yuzhongqi

Hey @yuzhongqi!

ℹ️ We don't consider behavioral changes from version to version to be a regression without clear evidence. Almost every new Xcode release has been accompanied by longer build times or more resource-intensive simulators, and so on, which could be considered a performance issue, but is external to us.

ℹ️ We don't accept requests to increase available hardware resources on runners—the technically permissible maximum is allocated.


We may note an issue if the same resource-intensive project builds in relatively similar times for the specified Xcode versions in a local environment with comparable resources, but shows performance differences on our runners.

This behavior is generally consistent with the usual changes that occur with the release of a new major version.

Since I understand how frustrating such a noticeable change in runtime can be, I propose the following:

  • I'll keep this issue open for a week.
  • If during this time any evidence emerges that the behavior differs from the behavior in local environments, we will investigate.
  • If during this time any information emerges that it's possible to safely configure Xcode 26 to speed up build times, we will investigate.

erik-bershel avatar Sep 29 '25 08:09 erik-bershel

@erik-bershel Thank you for your clarification and understanding! 🙏 I fully understand that with new Xcode releases, build times and resource consumption may change. This is common and may also be beyond your control. I really appreciate you keeping the issue open and being willing to follow up together! I’ll also make sure to provide feedback if I discover anything further.

yuzhongqi avatar Sep 29 '25 09:09 yuzhongqi

We may note an issue if the same resource-intensive project builds in relatively similar times for the specified Xcode versions in a local environment with comparable resources, but shows performance differences on our runners.

Would it be considered an issue if there are performance differences across multiple runs on your runners? We upgraded our workflows to Xcode 26 and early indications are that the same workflow could take 7 mins, 40 mins or just time out after 6 hours. The code being tested in those 3 examples is unchanged.

pixlwave avatar Sep 30 '25 08:09 pixlwave

@pixlwave What simulator version are you using in your workflows? Just wondering if you have also bumped to iOS 26.

yasali avatar Oct 07 '25 20:10 yasali

Yes we have – Xcode 26 running an iOS 26 simulator.

pixlwave avatar Oct 08 '25 07:10 pixlwave

@pixlwave Thanks for your response I have tried using Xcode 26.0 with iOS 26.0 on macos-15-xlarge runner but I get

Running xcodes version 1.6.2
[08:02:03]: $ /opt/homebrew/bin/xcodes installed '26.0'
[08:02:03]: ▸ 26.0 is not installed.
[08:02:03]: Exit status of command '/opt/homebrew/bin/xcodes installed '26.0'' was 1 instead of 0.

could you share which runner are you using?

yasali avatar Oct 08 '25 08:10 yasali

The plain old macos-15 runner.

pixlwave avatar Oct 09 '25 15:10 pixlwave

Hey @pixlwave!

Would it be considered an issue if there are performance differences across multiple runs on your runners? We upgraded our workflows to Xcode 26 and early indications are that the same workflow could take 7 mins, 40 mins or just time out after 6 hours. The code being tested in those 3 examples is unchanged.

Well, it's not a regression, but it's an interesting case study. I'm currently working on a slew of performance issues, including those related to Xcode 26. There's a rather interesting underlying situation with runners, which also changes with every system and Xcode update. There's very high competition for disk resources from simulators and MDS.

Unfortunately, this is very difficult to fix on our end, since we cannot provide faster disks, and launches really become a lottery (if MDS decides to reindex your project while building, you will have to wait some time).

erik-bershel avatar Oct 09 '25 16:10 erik-bershel

Thanks for the reply and explanation @erik-bershel, your work is much appreciated and we're definitely noticing that day by day the number of runs that timeout are going down 🙏

pixlwave avatar Oct 10 '25 10:10 pixlwave

Hi @erik-bershel ,we are still consistently experiencing excessively long build time(The archive app step takes about 1 hour) and would greatly appreciate guidance or updates on potential solutions. Or do you have any suggestions for speeding up the archive process by modifying the pipeline YAML?Thanks

- task: Xcode@5
    displayName: "Archive App"
    inputs:
      actions: "${{ parameters.actions }}"
      configuration: "${{ parameters.buildConfiguration }}"
      packageApp: ${{ parameters.packageApp }}
      scheme: "copilotn"
      sdk: "$(sdk)"
      signingOption: "default"
      provisioningProfileUuid: "${{ parameters.devProvisionProfileId }}"
      args: "DEVELOPMENT_TEAM=${{ parameters.teamId }} -derivedDataPath $(outputFolder) ${{ parameters.additionalArgs }}"
      archivePath: "$(outputFolder)/ArchiveOutput/Release-$(sdk)"
      exportPath: "$(outputFolder)/Archive/Release-$(sdk)"
      exportOptions: plist
      exportOptionsPlist: "$(Build.SourcesDirectory)/.provisioning/${{ lower(parameters.buildConfiguration) }}/ExportOptions.plist"
      useXcpretty: false
      project: "CopilotNative/copilotn/copilotn.xcodeproj"
    continueOnError: ${{ parameters.continueOnError }}

Agent version: macos-15 Xcode version: Xcode_26.0.app example pipeline build link:https://msasg.visualstudio.com/CopilotKit/_build/results?buildId=61571054&view=results

yuzhongqi avatar Oct 21 '25 06:10 yuzhongqi

Are there any updates on this? This is really stopping us from moving to Xcode 26 runners and beginning to block some development in our teams. Any workarounds, hints, tips etc - I'll take anything at this point 😅

ThrownToys avatar Nov 02 '25 22:11 ThrownToys

+1 here, the build is more than 2 hours in macos 15 (Xcode 16 or 26)

badredaha avatar Nov 14 '25 14:11 badredaha

https://developer.apple.com/news/?id=6lxhtioi Starting April 2026, apps and games uploaded to App Store Connect need to meet the following minimum requirements. iOS and iPadOS apps must be built with the iOS 26 & iPadOS 26 SDK or later tvOS apps must be built with the tvOS 26 SDK or later visionOS apps must be built with the visionOS 26 SDK or later watchOS apps must be built with the watchOS 26 SDK or later Because of the official mandatory requirements above, we hope Azure can resolve the issue of excessively long Xcode 26 build times as soon as possible. @erik-bershel Thanks

yuzhongqi avatar Nov 17 '25 03:11 yuzhongqi

FYI I'm not using Azure but Bitrise as my CI and I'm witnessing the same kind of issue. Even building on my own macBook (M1 Pro), release mode gets stuck on some modules and take no less than 20 minutes vs a fraction of a second on debug or Xcode 16.x. For me it looks like an Apple issue so I filed a feedback to Apple FB21015261. Maybe you could do the same & reference this ticket for it to get some traction ^^'. (I'm on macOS 15 but the CI is macOS 26 so apparently in my case it makes no difference)

bioche avatar Nov 17 '25 10:11 bioche

Because of the official mandatory requirements above, we hope Azure can resolve the issue of excessively long Xcode 26 build times as soon as possible. @erik-bershel Thanks

There's nothing we can do - the degradation is on Apple's end. We're keeping these tickets open to raise awareness and to allow users to share custom workarounds for their specific cases.

erik-bershel avatar Nov 17 '25 11:11 erik-bershel

@erik-bershel with my all respect, its works very good in Action Github! The problem is just in Azure !

badredaha avatar Nov 17 '25 13:11 badredaha

Hi @badredaha Could you clarify how large the gap is between the two platforms? For example, if a build takes around 10 minutes on GitHub Actions, how long would the same build typically take on Azure— 20 minutes, 30 minutes, or even longer? Approximately how many times slower is it?

yuzhongqi avatar Nov 17 '25 14:11 yuzhongqi

The problem is just in Azure !

I'd like to clarify. This works worse in Azure DevOps because the default runners are Intel-based, while for GitHub, Apple Silicon is the default, with Intel being an optional feature. If you run the process on macos-15-intel on GitHub, you'll encounter the same issues as in Azure DevOps.

There are several reasons:

  • Intel-based runners are indeed slower for a number of tasks, including Xcode builds. This issue affects all Intel-based runners, and there's nothing we can do about it, as the maximum resources have already been allocated, and there's no optimization on the tool's developer side, and it likely won't happen due to the upcoming deprecation of Intel support. Only Intel/x64 runners are affected.

  • Xcode 26 is not backwards compatible with Xcode 16. New versions of the CoreSimulator runtime either cause critical slowdowns or even cause older simulators to stop working altogether. The newer, the more interesting. Both arm64 and x64 runners are affected.

  • At least Xcode 26 has problems with running runtimes in a virtualization environment (runners are not an on-host solution). Both arm64 and x64 runners are affected.

Within the framework of this project and this repository, no decisions are made about which architecture to use for free runners. We develop the source code, build and distribute base images, and respond to image issues related to the build process or build code, system configuration, or installed applications. 🤷

I would also like to note that information about the demand for Apple Silicon support as a primary platform has been communicated; this is definitely a goal for the Azure DevOps platform. Any such changes, if and when they occur, will be announced both here and through more official Microsoft channels.

If at least one of the problems is resolved, it will become easier for everyone to work, for sure

UPD:

I'd even add that comparing the performance of Intel and Apple Silicon runners is meaningless and won't be considered as a degradation argument, given that the images are different at the base level, and with each subsequent layer (hypervisor -> VM OS -> application), the behavioral differences grow. We might as well try to achieve absolute compatibility between the GitHub Ubuntu-24.04 and Partner Ubuntu-24.04 arm64 runners.

At the same time we are willing to discuss the issue and work on improving the performance of each image individually. We understand how pressing this issue is for the community.

erik-bershel avatar Nov 17 '25 14:11 erik-bershel

A little question i can select macos-14-arm64 on azure pipeline, but i can't select macos-15-arm64.Do we know whether Azure has any plans or a timeline for supporting macOS-15-arm64? @erik-bershel

yuzhongqi avatar Nov 20 '25 02:11 yuzhongqi

A little question i can select macos-14-arm64 on azure pipeline, but i can't select macos-15-arm64.Do we know whether Azure has any plans or a timeline for supporting macOS-15-arm64? @erik-bershel

There are definitely plans, and the work is definitely underway. If my colleagues had more detailed information about the timeframe, they would share it, as well as I do.

erik-bershel avatar Nov 20 '25 09:11 erik-bershel

A little question i can select macos-14-arm64 on azure pipeline, but i can't select macos-15-arm64.Do we know whether Azure has any plans or a timeline for supporting macOS-15-arm64? @erik-bershel

There are definitely plans, and the work is definitely underway. If my colleagues had more detailed information about the timeframe, they would share it, as well as I do.

Thanks

yuzhongqi avatar Nov 20 '25 09:11 yuzhongqi

As the number of Copilot app users on iOS 26 continues to grow, and with the requirement that starting April 2026, apps uploaded to App Store Connect must meet the following minimum: iOS and iPadOS apps must be built with iOS 26, we urgently need Azure agents to support macOS-15-arm64 for the Copilot iOS app CI/CD. Does Azure have a clear timeline for supporting macOS-15-arm64? @erik-bershel Thanks

yuzhongqi avatar Nov 27 '25 07:11 yuzhongqi

Well, it's not a regression, but it's an interesting case study. I'm currently working on a slew of performance issues, including those related to Xcode 26. There's a rather interesting underlying situation with runners, which also changes with every system and Xcode update. There's very high competition for disk resources from simulators and MDS.

Unfortunately, this is very difficult to fix on our end, since we cannot provide faster disks, and launches really become a lottery (if MDS decides to reindex your project while building, you will have to wait some time).

@erik-bershel MDS is Spotlight right? So if we disable it at the start of the run the performance of the build should improve?

Jon889 avatar Dec 03 '25 08:12 Jon889

Hey @Jon889!

I must warn you that any of the solutions listed are not official and may break your workflow - use at your own risk. Possible options:

  • PerfPowerServices disabling
  • Spotlight service disabling
  • To use third-party solutions like: https://github.com/biscuitehh/yeetd, usage example.

erik-bershel avatar Dec 03 '25 09:12 erik-bershel