[macOS] macos-15 (x86) runner is excruciatingly slow

[dumbmoron]

Description

I have recently noticed that the x86 macos-15 runner has become uselessly slow. I have no idea whether it's overloaded, or if it's a regression, but it is incapable of compiling software at a normal rate.

Platforms affected

• [ ] Azure DevOps
• [x] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 22.04
• [ ] Ubuntu 24.04
• [ ] macOS 13
• [ ] macOS 13 Arm64
• [ ] macOS 14
• [ ] macOS 14 Arm64
• [x] macOS 15
• [ ] macOS 15 Arm64
• [ ] Windows Server 2019
• [ ] Windows Server 2022
• [ ] Windows Server 2025

Image version and build link

Example job: https://github.com/imputnet/helium-macos/actions/runs/16128303494/job/45562831983

Current runner version: '2.325.0' Runner Image Provisioner Hosted Compute Agent Version: 20250703.357 Commit: 07daf62238a21140d93e045a38f3784d75c509e1 Build Date: 2025-07-03T14:39:09Z Operating System macOS 15.5 24F74 Runner Image Image: macos-15-arm64 Version: 20250630.1914 Included Software: https://github.com/actions/runner-images/blob/macos-15-arm64/20250630.1914/images/macos/macos-15-arm64-Readme.md Image Release: https://github.com/actions/runner-images/releases/tag/macos-15-arm64%2F20250630.1914

Is it regression?

Maybe

Expected behavior

The runner should be able to compile more than 400 translation units over 5 hours

Actual behavior

The runner is incapable of compiling more than 400 units into objects over 5 hours. Previously, it was able to compile thousands (>6000) objects per hour.

Repro steps

1. Try to compile a larger project
2. Fail spectacularly

[ijunaidm1]

Hi @dumbmoron - We will start investigating on this issue and will update you accordingly.

[amlcurran]

Can confirm that

Image: macos-15-arm64 Version: 20250630.1914

Is also running much slower for us than previous, similar to my previously reported issues #11509

[sangeeths03]

Hi @dumbmoron, thanks for the detailed report.

We've tested this thoroughly on our side using a synthetic workload that compiles over 6000 translation units into a single executable. This benchmark was executed across macos-13-arm64, macos-14-arm64, and macos-15-arm64 runners using the latest image version , and we did not observe any significant slowdown on macos-15-arm64 compared to earlier versions.

System conditions also appear normal:

1. ~35 GiB of free disk space (df -h)

2. Build times are within expected ranges

3. Full run log: - https://github.com/sangeeths03/runner-images-sangeeth/actions/runs/16216838686

Given these results, we’re currently unable to reproduce the performance issue on our side.

As a next step, we'll fork your repository and attempt to reproduce the problem using your exact build and configuration. This should help us identify whether the slowdown is related to specific build scripts, link step behavior, or other project-specific factors.

Additionally, a new macOS 15 ARM runner image is scheduled for release this week. Once it's live, we recommend trying your build again to see if the issue still persists under the updated environment.

We'll follow up after deeper testing with your repo. Also, could you please confirm:

1. What build duration you were observing previously for this same job (before the slowdown)?
2. If the slowness is consistent or intermittent across reruns?

Thanks again for your patience as we investigate this further.

[dumbmoron]

Hi @sangeeths03,

Thank you for looking into this!

What build duration you were observing previously for this same job (before the slowdown)?

The build normally takes around 20 hours to complete (4x 5 hour build jobs). Recently, I noticed that it can take easily up to 50 hours (10x 5 hour build jobs).

If the slowness is consistent or intermittent across reruns?

The slowness does seem to be intermittent - during some builds, the runner builds at a normal pace (maybe 1 object file per second, more or less, or much faster if it's using ccached objects). In other builds, the pipeline slows to a crawl, producing 1 object file per 10 to 20 minutes.

We currently have a x86 pipeline experiencing this: https://github.com/imputnet/helium-macos/actions/runs/16207265303/job/45787787964 It has been running for 2 hours so far, and only managed to produce 259 object files (normally, it is able to compile around 1200-1600 units per hour).

[sangeeths03]

Hi @dumbmoron, Thanks again for your detailed reporting and for your patience throughout the investigation.

We've been monitoring your recent workflow runs, and we’re happy to see that the build durations now appear to be back within the expected range (around 20 hours total). This aligns with the performance observed prior to the slowdown.

Additionally, based on our internal testing and environment health checks, we haven’t been able to reproduce the degradation on the macos-15 runners. No infrastructure-level regressions have been identified so far.

Given this, could you please confirm:

Are your builds consistently completing within the expected duration now?

Have you observed any further intermittent slowdowns recently?

If everything is running smoothly again, we’ll go ahead and consider this issue resolved. Otherwise, happy to dig deeper with more specific job links or details.

Thanks again!

[dumbmoron]

Hi @sangeeths03,

Are your builds consistently completing within the expected duration now?

The arm64 builds are more or less fine, and the x86 builds are still intermittently slow.

Have you observed any further intermittent slowdowns recently?

Yes, we currently have a running job that compiled only 500 units in 4 hours:

https://github.com/imputnet/helium-macos/actions/runs/16352765637/job/46232654075

[badredaha]

+1

[badredaha]

Hi,

I use AzureDevops pipeline and My iOS build app take more than 2 hours! and after that the build failed!

Any suggestions ?

[sangeeths03]

Hi @dumbmoron ,A new image was recently released for macOS 15 Intel (x86). Could you please confirm if the slowness reported still persists with the updated image?

[sangeeths03]

@badredaha , Thanks for reporting the issue. Could you please share the workflow or pipeline steps you're using for the iOS build? This will help me reproduce the problem and investigate it further.

[badredaha]

Hi @sangeeths03 , I use fastlane with build_app. In macos13 (40 minutes max) work fine but macos15 its exceed 2h and fail because Timeout Azuredevops not accept more than 189 minute.

[sangeeths03]

Hi @badredaha, did you get a chance to test this today? A new macOS 15 image was deployed two days ago with performance improvements.

[badredaha]

Hi, I still get this version: Agent name: 'Hosted Agent' Agent machine name: 'Mac-1754456033654' Current agent version: '4.258.1' Operating System macOS 15.5 24F5068b Runner Image Image: macos-15 Version: 20250728.1632 Included Software: https://github.com/actions/runner-images/blob/macos-15/20250728.1632/images/macos/macos-15-Readme.md Image Release: https://github.com/actions/runner-images/releases/tag/macos-15%2F20250728.1632 Runner Image Provisioner Current image version: '20250728.1632'

[sangeeths03]

Hi @badredaha, Thanks for confirming the runner version — you’re indeed using the latest macOS 15 image (20250728.1632). If you’re still seeing the timeout or slowness, I’ll take a closer look at this version and see if I can reproduce the issue on my end. I’ll get back to you shortly with my findings.

[badredaha]

Hi,

Thank you a lot @sangeeths03 🙏🏼 For the version that I shared with you is not good for me. The same problem

[sangeeths03]

@badredaha, We’ve released the new macOS 15 image. Could you try your workflow on it and let us know if you’re still seeing the issue?

If the problem persists, please share the relevant success or error logs so we can investigate further.

Thank you!

[sangeeths03]

Hi @dumbmoron, could you please confirm if the slowness issue is still occurring? From your repo, I am seeing successful and quick runs.

[badredaha]

Hi @sangeeths03

[sangeeths03]

Hi @badredaha , We faced a similar hang when resolving SPM dependencies with private repositories. Updating both Fastlane and the pipeline steps resolved it for us. In the Fastlane scan lane, add:

package_authorization_provider: "netrc"
use_system_scm: true

And before running the lane in your pipeline, ensure the .netrc file has the correct permissions:

chmod 600 ~/.netrc

Could you try this and see if it helps?

For reference - https://github.com/actions/runner-images/issues/12866#issuecomment-3221600818

[badredaha]

Hi,

I use CocoaPods not SPM.

[badredaha]

Hi,

Still not working! Always block into for more than 1 hours:

[15:56:37]: ▸ [App] Write Auxiliary File Script-00EEFC60759A1932668264C0.sh [15:56:37]: ▸ [App] Write Auxiliary File OutputFileList-00EEFC60759A1932668264C0-Pods-App-frameworks-Release-output-files-938190b10090936430bda70e8ffebcbd-resolved.xcfilelist [15:56:37]: ▸ [App] Write Auxiliary File InputFileList-00EEFC60759A1932668264C0-Pods-App-frameworks-Release-input-files-20522033a6315e1ea065b0cd5def1bd3-resolved.xcfilelist [15:56:37]: ▸ [App] Running script [CP] Embed Pods Frameworks

[sangeeths03]

Hi @badredaha,

Thanks for confirming it still hangs. To help reproduce this on our side, could you please share a bit more detail about your setup? Specifically:

1. The pipeline/workflow steps you are using (YAML or script that runs Fastlane)

2. The Fastlane lane (from your Fastfile) where the build is invoked

3. Whether you are running pod install in the pipeline before calling Fastlane

This will help us try to replicate the issue with CocoaPods and see if it’s related to the runner environment or the setup itself.

Additionally, could you also try running the same pipeline on the macos-14 runner image? This will help us confirm whether the slowdown is specific to a runner version or related to the project setup itself.

[badredaha]

Hi @sangeeths03

1. A: Basically I use workflow. I just tested yaml and doesn't work also. For information, the workflow and yaml work good in macos13 @ macos14.

2. A: the lane is lane beta but based in fastlane plugin also and an other lans that I share with you:

desc "Get version from BUILD_NUMBER"
  lane :increment_version do
    buildNumber = ENV["BUILD_NUMBER"]
    increment_build_number(build_number: buildNumber)
  end
desc "Prepare archive for AppStore or TestFlight"
  lane :prepare_archive do
    increment_version
    if ENV["APP_VERSION"]
      version = ENV["APP_VERSION"]
    else
      version = get_version_number()
    end
    increment_version_number(version_number: version)
  end


 desc "Deploy Beta for TestFlight - Optimized"
lane :beta do
  puts "=== Beta lane started at #{Time.now} ==="
  
 prepare_archive
  key = get_apple_key()
  
 match(
    api_key: key,
    type: "appstore", 
    app_identifier: "com.bunle.app
  )
  
  puts "=== Build phase started at #{Time.now} ==="
  build_app(
    scheme: "App",
    export_method: "app-store",
    skip_profile_detection: true,
    export_xcargs: "-parallelizeTargets",
    skip_archive: false,
    skip_codesigning: false
  )
  puts "=== Build phase completed at #{Time.now} ==="


  dsym_path = Actions.lane_context[SharedValues::DSYM_OUTPUT_PATH] || "./App.dSYM.zip"
  
  if File.exist?(dsym_path)
    upload_symbols_to_crashlytics(
      gsp_path: "./Assets/Configs/Firebase/Prod/GoogleService-Info.plist",
      dsym_path: dsym_path
    )
  else
    UI.important("No dSYM found at #{dsym_path}")
  end

  puts "=== Upload phase started at #{Time.now} ==="
  upload_to_testflight(
    notify_external_testers: false,
    app_identifier: "com.bunel.app",
    changelog: "Upload TestFlight",
    groups: ENV["DIFF_GROUP"] || "Team App",
    skip_waiting_for_build_processing: true,
    skip_submission: true,
    beta_app_description: "Description of the app"
  )
  puts "=== Upload phase completed at #{Time.now} ==="
end

3. A: I use also fastlane to run pod install:

platform :ios do

  before_all do |lane, options|
  setup_ci if [:beta, :distribution_deploy_dev, :distribution_deploy_recette, :distribution_deploy_staging].include?(lane)
  
  # Only run pods if Podfile changed or first time
  if !File.exist?("Pods/Manifest.lock") || File.mtime("Podfile") > File.mtime("Pods/Manifest.lock")
    puts "Installing/updating pods..."
    cocoapods(
      try_repo_update_on_error: true,
      podfile: "./Podfile"
    )
  else
    puts "Pods up to date, skipping installation"
  end
end

[badredaha]

Hi @sangeeths03

Any news, pls ?

[sangeeths03]

Hi @badredaha ,

I tested your case across macOS 14 (with Xcode 16.2 explicitly selected), and macOS 15 (with the default Xcode). The Fastlane lanes are running fine and the workflow builds correctly in these environments.

The only pending item is CocoaPods setup. I’ll work on adding that integration next so that the workflow fully covers projects using a Podfile.

Test workflow - https://github.com/bbq-beets/sangeeth-test/actions/runs/17857006663

Fastfile i used:

default_platform(:ios)

platform :ios do
  desc "Increment build number from CI BUILD_NUMBER"
  lane :increment_version do
    buildNumber = ENV["BUILD_NUMBER"] || "1"
    increment_build_number(build_number: buildNumber)
  end

  desc "Prepare archive (version + build number)"
  lane :prepare_archive do
    increment_version
    version = ENV["APP_VERSION"] || get_version_number()
    increment_version_number(version_number: version)
  end

  desc "Deploy Beta for TestFlight - Repro only (no signing, no upload)"
  lane :beta do
    UI.message("=== Beta lane started at #{Time.now} ===")

    # Step 1: bump version/build
    prepare_archive

    # Step 2: Simulated cert/keychain setup
    UI.message("=== Skipping match/api_key setup (repro only) ===")

    # Step 3: Build (no signing, no upload)
    UI.message("=== Build phase started at #{Time.now} ===")
    build_app(
      scheme: "Testapp",                
      export_method: "development",     
      skip_profile_detection: true,     
      export_xcargs: "-parallelizeTargets",
      skip_archive: false,              
      skip_codesigning: true,
      skip_package_ipa: true,    # :white_check_mark: avoid exportArchive
      skip_package_pkg: true
    )

    UI.message("=== Build phase completed at #{Time.now} ===")

    # Step 4: Simulated uploads
    UI.message("=== Skipping dSYM + Crashlytics upload ===")
    UI.message("=== Skipping TestFlight upload ===")

    UI.message("=== Beta lane finished at #{Time.now} ===")
  end
end

Everything else is working as expected.

[badredaha]

Hi @sangeeths03

Thx for the response. For me everything work good expect macOS15 (take more than 2h). You will release new version with cocoapods setup ?

[sangeeths03]

@badredaha CocoaPods 1.16.2 is already included in all macOS images. What I meant is that, for testing your case specifically, I need to add the CocoaPods setup steps in my Fastfile so your scenario can be replicated exactly.

Also, could you share any additional logs, links? That will help me investigate more effectively.

[badredaha]

@sangeeths03 please try : export_method: "ad-hoc" and give me your feedbacks

[sangeeths03]

https://github.com/bbq-beets/sangeeth-test/blob/main/Testapp/fastlane/Fastfile

@badredaha ,Tried that and it's also working fine - https://github.com/bbq-beets/sangeeth-test/actions/runs/17859523905

[badredaha]

@sangeeths03 404 for both urls