build
build copied to clipboard
nodejs
Add macOS 15 CI images with Xcode 16
V8 builds now need at least Xcode 16.3, which requires us to bump macOS to at least 15.2.
See https://github.com/nodejs/node/issues/58069 Example failing canary release: https://ci-release.nodejs.org/job/iojs+release/10961/
@ryanaslett Is this something that can be done? I don't know if we would have to use the newer version for all jobs or if can keep macOS 13 and 15 in parallel and select a version in the Jenkins jobs based on the Node.js version.
Our current mac infra has 2 intel nodes and 2 arm nodes that are serving both release infra and test infra.
Each node is capable of having 2 VM's at a time on them, for a total of 4 concurrent intel nodes and 4 concurrent arm nodes.
I have set the test infra to never have more than 3 at a time running so that release always has an available node.
If the node version determines which macOS version we test, then it should be relatively simple to do, as we're not trying to test both versions of xcode with the same test (thus not incurring more tests)
Update on this:
We need to upgrade the underlying cluster from our current 3.0 to at least 3.2 in order to have Sequoia images available.
Orka is releasing 3.3 on Jun 11, so we can either wait until the new release is out (to avoid having to upgrade again), or, alternately we can ask to upgrade to 3.2 if we need this sooner.
Hi Matteo, so the Orka cluster is upgraded now, but the images that were running (the macos13 ones) were broken, and required reconstructing.
Thats done, and we're back, and now, with an upgraded cluster, able to move forward with macos15 images with modern XCode builds, however the current priority is to reconstruct some ARM instances as our provider sprung a surprisingly short deadline on us (Friday) when they would be shutting down our Primary ARM hosts.
Once we have some temporary ARM hosts in place, to buy us some time to find a new permanent ARM home, I'll be able to pivot back to creating the new OSX Instances.
@ryanaslett ping. This is now urgent, as it's blocking the V8 update for the Node.js 25 release.
My plate just cleared as we've got the ARM machines in place now. This is next.
Could you make some progress? It's frustrating to see the deadline come closer while I can't do anything.
Sorry for the lack of updates, I have been working on this, just kept hitting snags and kept erroneously imagining I was going to update with "here, its done".
The current status: I have the arm64 test image built, and its actually running a build on the ci server now: https://ci.nodejs.org/job/node-test-commit-osx/66838/nodes=osx15-arm64/console
I'll have the x64 image, and the release variants built tomorrow (I have to do weird userid wrangling to allow the x64 ones to write to the shared ccache).
One thing that we're going to need to sort out is how we manage the osx images.. do we need the osx13 builds? We only have 2 arm and 2 intel machines capable of running 2 vms each (so 4 osx, and 4 x64 runners) -> those are shared with release so, effectively we only have 3 osx of each arch available for builds. If we need 2 per build it'll bottleneck quickly.
Thanks for the update!
I don't think it was discussed with other team members, but what I imagined is that we would continue to run osx13 only for Node.js 20-24, and switch to osx15 only for Node.js 25+. So we would not need 2 per build.
Can we take the opportunity to use the label names macos15-* instead of osx15-*?
I imagined is that we would continue to run osx13 only for Node.js 20-24, and switch to osx15 only for Node.js 25+.
Opened https://github.com/nodejs/build/pull/4154
The x64 builds are taking longer than I had hoped.. in order to make them work with the shared VM storage I've had to hack the user ID of the admin account to match the underlying UID/GID of the strange 9p mount that Orka/Macstadium uses. And that keeps tripping on strange things like the orka authentication only lasts for 1 hour, and I've started builds that fail to complete before that etc.
I have a build running now that I have greater confidence in actually completing.
Once that is done, the test images will be complete, and I'll use those to create the release images, which is just install xcode (likely manually) and add the signing keys.
The x64 image finally built, and is running a job: https://ci.nodejs.org/job/node-test-commit-osx/nodes=macos15-x64/66908/consoleFull
I went ahead and made the labels for these new macos ones "macos15" as that does seem like it'll be easier to pick out from the node selector script.
Remaining:
- [x] Build the release images from the test images
- [x] Update the jenkins cloud config of ci-release
- [x] Land #4154
- [x] Document the update method for building out all the macos infra.
Looks like SEA tests are failing on x64:
07:53:44 [process 31240]: --- stderr ---
07:53:44 dyld[31240]: failed to set up thread local variables for '/Users/admin/node-tmp/.tmp.4757/sea': unsupported thread-local, larger than 4GB
@nodejs/single-executable
I think we can just mark SEA on x64 + macOS as "unsupported" if no one ends up fixing it.
Status update:
The packer builds are built for the arm release image and its running a v8-canary release build now: https://ci-release.nodejs.org/job/iojs+release/11286/nodes=osx13-release-pkg/console
That build is going to take quite a while to finish, as its freshly populating both ccaches for release.
Same goes for the x64 build: https://ci-release.nodejs.org/job/iojs+release/nodes=osx13-x64-release-tar/11287/console : those have to re-populate the ccache and will take a while.
Once those finish, I'll adjust the tags/labels to indicate that the releases are exclusively being built on osx15 images, and we should be able to land #4154.
Tomorrow I'll push up all the packer changes I've made along with re-documenting the whole stack.
The images needed more work.. they were missing some things to make them complete (ssh keys to node-www/ssh configs/and the notarization tool credentials wouldnt properly save. Finally got that worked out.
Theres a build running now that should run closer to the cached speed, and should be much more likely to work: https://ci-release.nodejs.org/job/iojs+release/11293/
Hooray!, all three builds ran https://ci-release.nodejs.org/job/iojs+release/11293/nodes=osx13-arm64-release-tar/ , https://ci-release.nodejs.org/job/iojs+release/11293/nodes=osx13-release-pkg/ , https://ci-release.nodejs.org/job/iojs+release/11293/nodes=osx13-x64-release-tar/
(those are macos15 despote the label mismatch)
So, all images are built now, and running.
Documentation for this new management regime is in : #4163 .
Images are built and deployed. Lets open new issues for any new complications that arise.