bazelbuild
Upgrade macOS version and Xcode version for Mac machines
- macOS version should be upgraded to 12.5.1
- need to fix Bazel test suites broken due to
pythonno longer available.
- need to fix Bazel test suites broken due to
- Xcode version should be upgraded to 13.4.1, we can provide multiple Xcode versions.
Failing test on macOS 12.5.1: https://buildkite.com/bazel-testing/bazel-bazel/builds/8481#018330f9-71a1-4bc5-a8be-942f5c5ac3f9
@meteorcloudy Is there any way we can install Xcode 14? It would unblock some tests in rules_apple. https://github.com/bazelbuild/continuous-integration/issues/1348#issuecomment-1408531680
I'm in the process of installing both 13.4.1 and 14.2. However, some of the machines require an OS update, too. I hope that the process will be finished by tomorrow EOD.
Hey @fweikert! Thanks for doing this. I assume the rollout has been completed but the default version hasn't been changed, so we need to add an override?
I also noticed some failures starting Tuesday last week in our scheduled builds on our master branch: https://buildkite.com/bazel/rules-apple-darwin/builds?branch=master
It looks like the main issue is the that the tvOS SDK isn't installed anymore. We should likely make sure that every machine has all SDKs installed (with something like xcodebuild -downloadAllPlatforms for example). The error I'm seeing is: The operation couldn’t be completed. Failed to locate any simulator runtime matching options: { "com.apple.platform.appletvsimulator"
We also realized that the we have a Activating Xcode 13.0... step which didn't fail, but subsequently we see DEBUG: /private/var/tmp/_bazel_buildkite/e6a8ceab815440f7ffea5ed855e0655d/external/bazel_tools/tools/osx/xcode_configure.bzl:243:14: No default Xcode version is set with 'xcode-select'; picking ':version14_2_0_14C18' in the log. This is likely because Xcode 13 can't run on macOS Ventura.
Hey, yeah, unfortunately the infra update broke some of our tests (https://github.com/bazelbuild/bazel/labels/macos-infra-update).
Technically our CI should now default to Xcode 14.2 on Ventura, but apparently there is a bug that I still need to find. Can you explicitly specify 14.2 in your test config?
I guess we could (provided the tvOS simulator runtimes are installed for that specific version), but not sure what other stuff will break in our repos.
Is there no way to specify that we want to run on Monterey? Or is the whole fleet assumed to be Xcode 14.2/Ventura now?
It's a bit tricky right now - all the iMacs run Ventura (platform macos), while all the MacStudios still run Monterey (macos_arm64). Our Monterey machines have Xcode versions 13.0, 13.4.1 and 14.2 installed.
To elaborate a bit on the background: Right now CI fleet management is quite painful since we have to manually maintain 40 Macs of different generations (trash can Mac Pros, iMac Pros, MacStudios).
We're working on virtualizing the fleet, which would allow us to offer multiple different OS versions at the same time.
We definitely have tests failing on arm64, so switching directly to Mac Studios might be tricky in the short-term. I'm seeing messages related to the simulator runtimes being a bit off on those machines as well:
Failed to find a suitable device for the type IBSimDeviceTypeiPad2x (com.apple.dt.Xcode.IBSimDeviceType.iPad-2x) with runtime iOS 15.0 (15.0 - 19A339) - com.apple.CoreSimulator.SimRuntime.iOS-15-0 (Failure reason: Failed to create SimDeviceSet at path /Users/buildkite/Library/Developer/Xcode/UserData/IB Support/Simulator Devices. You'll want to check the logs in ~/Library/Logs/CoreSimulator to see why creating the SimDeviceSet failed.): Failed to initialize simulator device set. (Failure reason: Failed to subscribe to notifications from CoreSimulatorService.): Unable to determine SimDeviceSet, set_path=(null): Failed to initialize simulator device set. (Failure reason: Allocation or initialization failed.)
I think the best path forward would making sure the simulator runtimes and SDKs are installed properly on all machines, and then we can proceed to see which option gets us back to green asap (I'm thinking Xcode 14.0 running on Ventura at the moment).
making sure the simulator runtimes and SDKs are installed properly on all machines
I guess neither me or Florian is very familiar with this. Can you give some instructions on how to do this?
@fweikert Can you try to run xcrun simctl list and check what's the output?
I think following this guide should be a good start: https://developer.apple.com/documentation/xcode/installing-additional-simulator-runtimes
Running xcodebuild -downloadAllPlatforms should at least provide all the SDKs that we need (watchOS, tvOS) that are optional in Xcode 14 and not bundled with Xcode anymore by default.
I ran xcodebuild -downloadAllPlatforms on all iMacs. I can see that we got tvOS 16.1 and watchOS 9.1:
% xcodebuild -showsdks
DriverKit SDKs:
DriverKit 22.2 -sdk driverkit22.2
iOS SDKs:
iOS 16.2 -sdk iphoneos16.2
iOS Simulator SDKs:
Simulator - iOS 16.2 -sdk iphonesimulator16.2
macOS SDKs:
macOS 13.1 -sdk macosx13.1
macOS 13.1 -sdk macosx13.1
tvOS SDKs:
tvOS 16.1 -sdk appletvos16.1
tvOS Simulator SDKs:
Simulator - tvOS 16.1 -sdk appletvsimulator16.1
watchOS SDKs:
watchOS 9.1 -sdk watchos9.1
watchOS Simulator SDKs:
Simulator - watchOS 9.1 -sdk watchsimulator9.1
However, xcrun simctl list still shows many "Unavailable" entries such as the aforementioned com.apple.CoreSimulator.SimRuntime.iOS-15-0.
I think that's good enough, we might not need other runtimes that are not bundled with Xcode by default (I don't think we run tests on lower iOS versions in our repos). Looks like I was able to get a rules_apple PR that switches us to Xcode 14.2 on the iMacs green now:
I only noticed one issue on bk-imacpro-6 (which I didn't see on other iMacs running the same build):
Failure Reason: Failed to spawn AssetCatalogSimulatorAgent on Apple Watch Series 7 (45mm) (9BAC5919-45D6-4DDD-AB6A-AAAF73F5FB43, (null), Shutdown)
It looks like the "Apple Watch Series 7 (45mm)" simulator is not correctly created. If you run xcrun simctl list and you see that device on that machine, maybe it was just a temporary error. Otherwise something might be different on that machine vs the other iMacs (for example this build on bk-imacpro-19 was successful).
I can see Apple Watch Series 7 (45mm) (com.apple.CoreSimulator.SimDeviceType.Apple-Watch-Series-7-45mm) under "Devices" on bk-imacpro-6. However, the watchOS 9.1 runtime contains an entry with a different value: Apple Watch Series 7 (45mm) (7FF9613B-A581-4703-B2D9-DF589ACAB733) (Shutdown)
@fweikert I'm still seeing issues with the Apple Watch simulators. If I run xcrun simctl list devices available on a build agent, I only get iOS devices:
== Devices ==
-- iOS 16.2 --
iPhone 14 (438498C7-2E9A-4A4A-9CC7-5DED49FAC8C4) (Shutdown)
iPhone 14 Plus (BB68AFF0-037F-4B23-9A0B-2C5DB99D743A) (Shutdown)
iPhone 14 Pro Max (94EE6ADE-C7B7-4E76-9DFD-78B8B9229449) (Shutdown)
iPad Air (5th generation) (710C70AD-5DE4-4A57-846F-132519AB808D) (Shutdown)
iPad Pro (11-inch) (4th generation) (9F4F019B-BBDA-4C8D-84B0-73649107A41F) (Shutdown)
-- tvOS 16.1 --
-- watchOS 9.1 --
Maybe since the watchOS and tvOS simulators were installed post-Xcode install, we need to create some defaults manually?
https://buildkite.com/bazel/rules-apple-darwin/builds/6669#01864a45-7a3e-42d7-befe-dd6cbb024800 on imacpro-5 fails with Failed to spawn AssetCatalogSimulatorAgent on Apple Watch Series 7 (45mm) (9C9DE839-E2E7-4734-893F-1ED67D94A37C, (null), Shutdown)
However, when I ssh into imacpro-5 it looks ok:
== Devices ==
-- iOS 16.2 --
iPhone SE (3rd generation) (23077071-4176-435F-98BA-39F5499042B0) (Shutdown)
iPhone 14 (37FA8814-1FD8-480A-8F58-33C4617A73F1) (Shutdown)
iPhone 14 Plus (428DB8C2-202C-49AA-8608-4D27D7CF17A3) (Shutdown)
iPhone 14 Pro (5CB291A7-2186-4228-AA88-4CFD5A1644F9) (Shutdown)
iPhone 14 Pro Max (1711DFB0-CE56-4BE9-A339-C8FD18A0D2E0) (Shutdown)
iPad Air (5th generation) (67AADEA6-205E-4D90-B7C3-AC0ED2BAB359) (Shutdown)
iPad (10th generation) (AD32E8BD-DD0F-4160-A427-3647665E8050) (Shutdown)
iPad mini (6th generation) (17BB1E83-17A4-4A71-B171-0458FDD07149) (Shutdown)
iPad Pro (11-inch) (4th generation) (A56AFB3B-B499-4324-8EE8-C3D12EC7345D) (Shutdown)
iPad Pro (12.9-inch) (6th generation) (50D57211-FFF5-4C72-909E-C1C08EA748CE) (Shutdown)
-- tvOS 16.1 --
Apple TV (4843400E-199F-4036-ADD0-8C104D2FD4BD) (Shutdown)
Apple TV 4K (3rd generation) (608D000A-BA58-41DA-9F66-7393C7971FEC) (Shutdown)
Apple TV 4K (3rd generation) (at 1080p) (795E3ACD-E19C-42EA-B891-B75E42C15B0C) (Shutdown)
-- watchOS 9.1 --
Apple Watch Series 5 (40mm) (B848B04C-069B-4233-9B82-A356213D0286) (Shutdown)
Apple Watch Series 5 (44mm) (0C433A12-7EFE-40AA-BB3C-B89534F1AD1B) (Shutdown)
Apple Watch Series 6 (40mm) (2FB53684-00DF-4AA7-8DB1-D8C75DE12E41) (Shutdown)
Apple Watch Series 6 (44mm) (B809C122-7EA6-4459-AD86-573F62C89A95) (Shutdown)
Apple Watch Series 7 (41mm) (3CC797C2-E64F-4BB4-BDFC-0E07F57C6A88) (Shutdown)
Apple Watch Series 7 (45mm) (FD37414F-5819-4070-9E78-ED8A3747C3B7) (Shutdown)
Apple Watch SE (40mm) (2nd generation) (D571509A-4DDD-47DF-88BB-44F947EA44AC) (Shutdown)
Apple Watch SE (44mm) (2nd generation) (C756171C-DA97-4337-9F98-8A74B94902BE) (Shutdown)
Apple Watch Series 8 (41mm) (7C377419-8B05-4A3F-AE47-5AB686ED3BAD) (Shutdown)
Apple Watch Series 8 (45mm) (D0DBE70A-F153-410F-A8E8-52E5A46CD6A1) (Shutdown)
Apple Watch Ultra (49mm) (F8943ED9-75A3-404D-A696-4B1B0BF68FE6) (Shutdown)
Note that the UDIDs are different. Maybe it's an issue that we're using a dedicated CI user account? I'm not sure.
What I could find from my research was that it's either some permission issue, or broken Xcode installation. Some people suggested deleting /Library/Developer/ for example first and later reinstall Xcode. I saw this failure on a schedule master build on Saturday, so I'm fairly sure it's happening on some agents on builds that aren't cached: https://buildkite.com/bazel/rules-apple-darwin/builds/6653#01863dc8-f095-4f31-be49-d1dbf22b08dc
Issue with a missing android setup here https://buildkite.com/bazel/bazel-bazel-github-presubmit/builds/14315#018651c9-b37a-4443-bbee-700e877557ed
Issue with a missing android setup here https://buildkite.com/bazel/bazel-bazel-github-presubmit/builds/14315#018651c9-b37a-4443-bbee-700e877557ed
That's unrelated to Xcode - for some reason imac11 randomly decided to delete its Android SDKs. I reinstalled them earlier today.
more issues https://buildkite.com/bazel/rules-apple-darwin/builds/6689#01865773-20b1-4c30-b629-4b9ad316132d
Yeah, getting a flakey test (covered in the above link as well): https://buildkite.com/bazel/rules-apple-darwin/builds/6720#0186754c-b8b7-49d2-b46d-7184fbe0fb7a
looks like machine number 1 doesn't have all the platforms installed so rules_apple tests don't work https://buildkite.com/bazel/bcr-presubmit/builds/1060#018675e2-6e91-46e8-b1d9-b96fde6531b1
can someone run xcodebuild -downloadAllPlatforms on it?
Tried to run xcodebuild -downloadAllPlatforms on all imacpros, getting the following error on machine 1 and a few other machines:
ci@bk-imacpro-1 ~ % xcodebuild -downloadAllPlatforms
Downloading tvOS 16.1 Simulator (20K67): Error: Error Domain=SimDiskImageErrorDomain Code=5 "Duplicate of C343EBAA-CF3F-4397-A071-388674B05A31" UserInfo={NSLocalizedDescription=Duplicate of C343EBAA-CF3F-4397-A071-388674B05A31, unusableErrorDetail=}
I ran xcodebuild -downloadAllPlatforms on all iMacs two weeks ago. This is really weird.