[6.5.0] Fixes for using recent Xcodes and macOS 26

[mbland]

Upgrades Zlib to 1.3.1 and removes the -no_adhoc_codesign and -no_uuid linker flags for building the Clang wrapper. Fixes #27026.

Incorporates changes from the following commits and pull requests:

• Update third_party/zlib/BUILD{,.tools} for 1.3.1 bazelbuild/bazel@b29649fbdc983cd62a58b9b09ef699867e7c5b69 bazelbuild/bazel@2067a8b96f624ab5177f3120f802a1255de01ce9

• Fix upb build with Clang 16 bazelbuild/bazel@4a32d7b98423682be9c47f571afbfbb587605895 bazelbuild/bazel#23700

• Remove -Wl{-no_adhoc_codesign,-no_uuid} Required to build under macOS 26.0 to avoid the `missing LC_UUID` error from `wrapped_clang` and `xcode-locator`. bazelbuild/bazel#27014

Upgrades third_party/zlib using the methodology from #27026:

cd third_party
curl -L https://github.com/madler/zlib/releases/download/v1.3.1/zlib-1.3.1.tar.gz |
  gzip -dc | tar xf -
rsync -r zlib-1.3.1/* zlib/
rm -rf zlib-1.3.1
cd ..

Several third_party/zlib files show up as complete replacements because the previous files used CRLF line endings, and the 1.3.1 files only use LF.

To list the substantial changes outside of the Zlib 1.3.1 distribution upgrade:

$ git diff --stat HEAD^ -- ':!:third_party/zlib/[^B]*'
 distdir_deps.bzl                       |  4 ++++
 third_party/upb/01_remove_werror.patch | 19 +++++++++++++++++++
 third_party/upb/BUILD                  |  4 +++-
 third_party/zlib/BUILD                 | 45 ++++++++++++++++++++++++++++++++++++++-------
 third_party/zlib/BUILD.tools           | 15 +++++++++------
 tools/cpp/osx_cc_configure.bzl         |  2 --
 tools/osx/BUILD                        |  3 +--
 7 files changed, 74 insertions(+), 18 deletions(-)

[mbland]

@meteorcloudy Note that the bulk of the diff in that pull request is the zlib-1.3.1 upgrade. The rest of the changes are exactly from the patch from #27026 (which I copy and pasted, and applied).

The first CI run failed a lot of tests and several builds were DOA. The Windows and RBE builds, in particular, broke as soon as they started.

The second CI run after pushing a commit to update .bazelversion to 6.3.2 got a lot farther. The Windows builds ran to completion, and the RBE build actually passed. The Clang build passed as well.

I suppose we can figure out the rest of the failures between now and some time after BazelCon.

[meteorcloudy]

Yeah, our release policy does say we promise to backport fixes for OS-compatibility issues into this LTS release in Maintenance stage. Bazel 6 is still in maintenance until Dec 2025 ;)

I'm fine with disabling many of those failing tests if they look irrelevant, for example, android tests. Many of them are failing due to infrastructure changes.

[mbland]

@meteorcloudy I've been poking at this the past week and a half. Between cherrypicking fixes from master and applying my own test fixes, all the Linux builds are now passing. (The pull request has become a bit of a beast, however.)

The macOS and Windows builds continue to fail, and I'm pretty sure the lion's share of those are because those images contain Java 21, not Java 11. (i.e., Several of the failures resemble failures I fixed on Linux by ensuring Java 11.)

Tomorrow I can compile a detailed breakdown of what's still failing. Unless, of course, you want to take a look and can beat me to it. (And if you have a tip as to how we can get Java 11 on the macOS and Windows images and make it the default, that would be great!)

[mbland]

Progress! I'm down to only these tests failing on macOS and Windows:

macOS shard 2

//src/test/shell/bazel:bazel_example_test
//src/test/shell/bazel:bazel_proto_library_test  # passed, but timed out
//src/test/cpp:option_processor_test
//src/test/shell/bazel:bazel_coverage_java_jdk11_toolchain_head_test
//src/test/shell/bazel/remote:remote_execution_test  

macOS shard 3

//src/test/shell/bazel:bazel_java_tools_test  # passed, but timed out
//src/test/py/bazel:launcher_test
//src/test/shell/bazel:bazel_bootstrap_distfile_tar_test
//src/test/shell/bazel:bazel_coverage_java_jdk11_toolchain_released_test

macOS shard 4

//src/test/shell/bazel:bazel_bootstrap_distfile_test
//src/test/shell/bazel:bazel_java_test_jdk11_toolchain_head
//src/test/shell/bazel:bazel_sandboxing_test
//src/test/shell/bazel:external_starlark_execute_test
//src/test/shell/integration:bazel_command_log_test
//src/test/shell/bazel:bazel_repository_cache_test

macOS shard 5

//src/test/py/bazel:bazel_remote_cache_decompression_test
//src/test/shell/bazel:bazel_workspaces_test

Windows shard 2

//src/test/shell/bazel:bazel_bootstrap_distfile_test
//src/test/shell/bazel:bazel_java_test_jdk11_toolchain_head

Windows shard 3

//src/test/py/bazel:bazel_windows_cpp_test

Windows shard 4

//src/test/java/com/google/devtools/build/lib/bazel/repository/downloader:DownloaderTestSuite

[meteorcloudy]

you have a tip as to how we can get Java 11 on the macOS and Windows images and make it the default

Unfortunately this is no longer possible, we've moved forward with CI image. I'm fine with disabling those tests if they don't actually catch any regression for in new changes.

[loeffel-io]

any update here? we really could need something like 6.5.1 🙏

[mbland]

@meteorcloudy Modulo the last macOS test timeouts I'm squashing, I've got all the CI builds passing. The pull request has become a beast, so I can split it up or rebase it to squash/drop commits if you like.

(Oddly enough, the CLA check is failing because I cherry-picked a commit by @katre, even though his google.com address is in the commit.)

The core changes from the very first commit remain unchanged. I've cherry-picked a number of commits from master to backport fixes that didn't land on the 6.5.0 branch (applying manual tweaks where necessary). There are a bunch of debugging commits I can rebase out of existence if need be.

Some of the more significant recent commits (without going back to the very beginning) include:

• 312098f9c75365206a3e363a12b4abadb965e451 to replace ubuntu2004 CI images with ubuntu2004_java11

• ef45283ec54207cb79aecbe0f0705a6e9fb9c1e6 to set sandbox_default_allow_network=true on macOS CI to enable downloading of archives

• Several commits to set the Java language to 11 and runtime version to remotejdk_11 in all the tests that need it

• e41057f2f12fdf2cde1fc5f03fba82977e047e97 to load the Bash runfiles library in integration_test_setup.sh

• Several commits remove remote_helpers.sh from tests that don't need it, or add @local_jdk//:jdk to the deps of those that do. (setup_localjdk_javabase, invoked by remote_helpers.sh, requires @local_jdk//:jdk.)

• 298140ccf353ba10a0d36a6d297f87a159ccf5ef to fix bazel_proto_library_test

• 1f34f7358ee725ae60b001de4586be0994203562 to fix //src/test/cpp:option_processor_test by ignoring .bazelrc files

• bd93f03c05eaa00efc4ab6ac82a5b8330458af9f includes most of the original cherry-picked commit to fix and verify Java version compatibility of Java tools

• 9485f548c2d933826e64f9e8046f33e0e104ce00 to move release_archive constants to tools/build_defs

• cdd4af17f4a44b8692bd84c7fa333c228041304c to fix repository/downloader:DownloaderTestSuite for windows + JDK21

• e7d2fe5eaad1a393d1ee0ea52fd07adb1257afbb and a08c284d332cf738069eab4cb4e638933f8d4b0c to disable Android tests after the changes from bazelbuild/continuous-integration#2408 landed (https://github.com/bazelbuild/continuous-integration/pull/2408#issuecomment-3612425196)

• 954c16b7b411d8e71c1d53eb99f56b2d2e47dbac and 7b76014f7d70b71a8fcab79e8d8670b95e9f73ed (and others) to increase the timeout for a number of tests on macOS

• c75798db570186ad6da1c10c9e352d3f178e3669 to fix bazel_bootstrap_distfile{,_tar}_test for local JDKs > Java 11

• 4c7e9ddc0f82521e29ccd65dd8e38949576df498 to remove the last failing Windows test

• a210ec0da6f220d7000b3663c1fe4105aad11d20 to resolve IPv6-related errors on macOS

• 3f56f27badd04cfcbd0e538d54f308cbb8945875, f0c172d0785e128ef0e102b28f6890099ec23685, 0448fcf71aa0b645622e235d90f18da271d15975, and f143bf40edffe91f9af1cd050c54780d8acd0188 to shore up //src/test/shell/integration:bazel_command_log_test on macOS

The real pièce de résistance that restored my sanity was 8a5bd6f5faabbff2207742f6c4ff71f61c139f8e, to set the right JDK version for *_for_testing repos. Bazel was emitting JVM flags for Java >= 17 into test stubs using a Java 11 runtime. It took weeks to track down what was happening, which amounted to an omission from 7556e1107b666d10b660470a571631463c7eb4ec and 2d04c91327cadb3f0d53bceea117d3939a25e143.

Please let me know how you'd like to proceed from here.