Make Bazel clearly communicate issues with XCode install

[eugeneo]

I updated the OS but did not update XCode. Bazel was failing with the most cryptic error (see below).

Please make Bazel messaging more helpful, possibly suggesting the user to check XCode install and do bazel clean --expunge

Description of the bug:

Unable to build my branch on my local macOS environment. This branch builds on CI, including Mac.

FATAL: bazel crashed due to an internal error. Printing stack trace:
java.lang.RuntimeException: Unrecoverable error while evaluating node 'ActionLookupData0{actionLookupKey=ConfiguredTargetKey{label=//test/core/event_engine:common_closures_test_TEST_LIBRARY, config=BuildConfigurationKey[e7d0355b3d90789b06a9207238e0040e8b8404e05a204aa41ab02c9d9ba36c03]}, actionIndex=0}' (requested by nodes 'TargetCompletionKey{topLevelArtifactContext=com.google.devtools.build.lib.analysis.TopLevelArtifactContext@90904c3b, actionLookupKey=ConfiguredTargetKey{label=//test/core/event_engine:common_closures_test_TEST_LIBRARY, config=BuildConfigurationKey[e7d0355b3d90789b06a9207238e0040e8b8404e05a204aa41ab02c9d9ba36c03]}, willTest=false}', 'ActionLookupData1{actionLookupKey=ConfiguredTargetKey{label=//test/core/event_engine:common_closures_test_TEST_LIBRARY, config=BuildConfigurationKey[e7d0355b3d90789b06a9207238e0040e8b8404e05a204aa41ab02c9d9ba36c03]}, actionIndex=1}')
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:547)
	at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:435)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.IllegalArgumentException: com.google.devtools.build.lib.rules.apple.DottedVersion$InvalidDottedVersionException: Dotted version components must all start with the form \d+([a-z0-9]*?)?(\d+)? but got 'None'
	at com.google.devtools.build.lib.rules.apple.DottedVersion.fromStringUnchecked(DottedVersion.java:181)
	at com.google.devtools.build.lib.exec.local.XcodeLocalEnvProvider.rewriteLocalEnv(XcodeLocalEnvProvider.java:95)
	at com.google.devtools.build.lib.sandbox.DarwinSandboxedSpawnRunner.prepareSpawn(DarwinSandboxedSpawnRunner.java:219)
	at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.exec(AbstractSandboxSpawnRunner.java:112)
	at com.google.devtools.build.lib.sandbox.SandboxModule$SandboxFallbackSpawnRunner.exec(SandboxModule.java:479)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:158)
	at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:118)
	at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:45)
	at com.google.devtools.build.lib.rules.cpp.CppCompileAction.execute(CppCompileAction.java:1395)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1170)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1075)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:166)
	at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:95)
	at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:559)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:928)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:375)
	at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:216)
	at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:467)
	... 7 more
Caused by: com.google.devtools.build.lib.rules.apple.DottedVersion$InvalidDottedVersionException: Dotted version components must all start with the form \d+([a-z0-9]*?)?(\d+)? but got 'None'
	at com.google.devtools.build.lib.rules.apple.DottedVersion.fromString(DottedVersion.java:203)
	at com.google.devtools.build.lib.rules.apple.DottedVersion.fromStringUnchecked(DottedVersion.java:179)
	... 24 more

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

macOS 15.3.2 (24D81)

What is the output of bazel info release?

release 8.0.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

$ which c++
/usr/bin/c++

$ c++ --version
Apple clang version 15.0.0 (clang-1500.3.9.4)
Target: arm64-apple-darwin24.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[eugeneo]

Just noticed I can not build even default branch (https://github.com/grpc/grpc) so it looks like something on my env broke with some Bazel update or maybe OS update.

I'm a Googler, feel free to reach out internally.

[keith]

probably the same local env issues as discussed on https://github.com/bazelbuild/bazel/issues/23111

[eugeneo]

I needed to reinstall XCode.

I will change the title and description of this issue.

[Wyverald]

Either way, Bazel should not crash. @keith do you see a way for us to print a helpful error message instead of crashing?

[keith]

yea for sure we can check for None in that codepath and error. It just doesn't come up too much so folks haven't been super motivated to improve it

[Wyverald]

I'm also a bit personally curious -- why is this still in native code, actually? I thought this kind of stuff was taken care of by apple_support/rules_apple already.

[keith]

this specific logic is doing non-hermetic stuff to make sure the absolute path to Xcode doesn't appear in the action keys. specifically the resulting path here https://github.com/bazelbuild/bazel/blob/2d14aa2e3f4e7b7b9893c23b26fdb32f84e5fef3/src/main/java/com/google/devtools/build/lib/exec/local/XcodeLocalEnvProvider.java#L95-L96 is absolute and based on the host system's local installation location

[meteorcloudy]

Can we starlarkify XcodeLocalEnvProvider.java?

[fzakaria]

What does it mean to reinstall xcode ?

> sudo rm -rf /Library/Developer/CommandLineTools
> xcode-select --install

Not sure how to get past it.

[nat45928]

Also having this same issue with Bazel 8.2.1. I tried to reinstall xcode command line tools (and a clean --expunge) as described with no change.

[jmillikin]

I can reproduce this on macOS 15.1 with the command-line tools installed. It seems to affect any workspace where @apple_support is depended on before @rules_cc, including the Bazel build itself.

Minimal reproduction workspace:

# MODULE.bazel
bazel_dep(name = "apple_support", version = "1.21.0")
bazel_dep(name = "rules_cc", version = "0.1.1")

# BUILD.bazel
load("@rules_cc//cc:cc_binary.bzl", "cc_binary")
cc_binary(
    name = "hello",
    srcs = ["hello.cc"],
)

#include <stdio.h>
int main() {
	printf("Hello, world!\n");
	return 0;
}

Command line tools are installed and functional:

john@macbook2 ~ % xcode-select -p
/Library/Developer/CommandLineTools
john@macbook2 ~ % xcrun --show-sdk-version
15.1

I tracked the issue down to something going wrong in tools/osx/xcode_locator.m.

% cd src/github.com/bazelbuild/bazel/8.2.1
% xcrun --sdk macosx clang -mmacosx-version-min=10.13 -fobjc-arc -framework CoreServices -framework Foundation -o xcode-locator tools/osx/xcode_locator.m
% ./xcode-locator -v
error: Error Domain=NSOSStatusErrorDomain Code=-10814 "kLSApplicationNotFoundErr: E.g. no application claims the file" UserInfo={_LSLine=1734, _LSFunction=runEvaluator}

This error causes @@bazel_tools+xcode_configure_extension+local_config_xcode//:host_xcodes to be just xcode_config(name = "host_xcodes") (no attributes), which creates an apple_common.XcodeVersionConfig that returns None from its xcode_version() function.

The poison XcodeVersionConfig propagates through @bazel_tools -> @apple_support -> @bazel_tools until it eventually hits apple_common.apple_host_system_env(), which sets XCODE_VERSION_OVERRIDE=None:

https://github.com/bazelbuild/bazel/blob/615b63dd005923e12e1356c1368e80dd121f2c5a/src/main/starlark/builtins_bzl/common/objc/apple_env.bzl#L17-L20

... and then that environment variable ends up in the spawn machinery, and tries to get parsed as a dotted version and blows up.

[meteorcloudy]

@jmillikin Thanks so much for figuring this out.

@keith Can you please help decide a proper fix for this? I think this issue is making first time Bazel experience worse for macOS users.

[sluongng]

Sounds like https://developer.apple.com/documentation/coreservices/1449290-lscopyapplicationurlsforbundleid failed around this code block

https://github.com/bazelbuild/bazel/blob/615b63dd005923e12e1356c1368e80dd121f2c5a/tools/osx/xcode_locator.m#L133C38-L139

Might need to troubleshoot the Launch Service on that machine to see if XCode was registered there. How was this XCode installed?

[jmillikin]

I don't have XCode installed on that machine. I installed the Apple developer "command line tools", which lives in /Library/Developer/CommandLineTools and provides clang etc.

[cerisier]

I was actually debugging this yesterday...

Like @jmillikin, I have ONLY the CommandLineTools installed. No Xcode.

It's not clear to me who is the culprit between apple_support._run, apple_common.apple_host_system_env or XcodeLocalEnvProvider.java nor who is really responsible for constructing the action env on apple systems.

It seems like a mix up currently. Especially since apple_common.apple_host_system_env is removed in Bazel 9.

It's not a fix but since XCODE_VERSION_OVERRIDE is supposed to be none (?), commenting out this line in apple_support bypasses the DottedVersion parsing.

diff --git a/lib/apple_support.bzl b/lib/apple_support.bzl
index 86a0dcf..c29dfe1 100644
--- a/lib/apple_support.bzl
+++ b/lib/apple_support.bzl
@@ -184,7 +184,9 @@ def _kwargs_for_apple_platform(

     # Add the environment variables required for DEVELOPER_DIR and SDKROOT last to avoid clients
     # overriding these values.
-    merged_env.update(apple_common.apple_host_system_env(xcode_config))
+    # merged_env.update(apple_common.apple_host_system_env(xcode_config))
     merged_env.update(
         apple_common.target_apple_env(xcode_config, apple_fragment.single_arch_platform),
     )

[jfirebaugh]

A coworker was encountering this issue with full Xcode installed, not CLT. xcode-select -p prints /Applications/Xcode.app/Contents/Developer. I had them do a "partial expunge" -- rm -rf $(bazel info output_base)/external/{bazel_tools,apple_support}* && bazel shutdown -- and that resolved the issue.

[j2kun]

I can reproduce this on macOS 15.1 with the command-line tools installed. It seems to affect any workspace where @apple_support is depended on before @rules_cc, including the Bazel build itself.

Is a workaround simply to change the order of dependencies? Our collaborators regularly hit this issue.

[keith]

here's a fix for this crash https://github.com/bazelbuild/bazel/pull/27043 that I think is reasonable. But there are some outstanding UX issues we should fix.

1. How should just the command line tool install be handled. Currently it would crash, which is fixed by this PR, but now it will just fail because bazel doesn't know how to use the CLT like it's an Xcode install, even though it would be good enough for many cases of building simple macOS targeting objc. I've thought about how to fix this before and it will require plumbing that through in a few places
2. how do you invalidate the repository_rule that discovered the Xcode versions after you fix your install

For #2 here is the trivial repro case to get this issue:

• Have a working Xcode version
• Run bazel cquery @@bazel_tools+xcode_configure_extension+local_config_xcode//:all --output=starlark --starlark:expr 'providers(target)'
• sudo mv /Applications/Xcode*.app ~/.Trash
• Run the same bazel command above

Even after shutdowns or bazel fetch --configure calls this repository remains stuck with the old version of Xcode in the config file. This is why users have such trouble getting out of this state. Doing bazel clean --expunge should fix it but interestingly I've had reports where folks say that isn't enough. To check the current state of things you can inspect the generated file in question with:

cat bazel-out/../../../external/bazel_tools+xcode_configure_extension+local_config_xcode/BUILD

Running bazel fetch --force @@bazel_tools+xcode_configure_extension+local_config_xcode//... seems to work to cause the repo to refetch. Ideally we would be able to do something smarter to invalidate something here. Given that the build can't proceed until the user fixes this issue, it wouldn't hurt anything to re-fetch this everytime until it was fixed. It seems like 1 option is if we flip local=True on the repository rule, it will re-fetch when the server restarts, but no longer works with the fetch --force command (might be a bug in fetch)

[fmeum]

bazel fetch --configure --force is a sight improvement as it should work whenever bazel clean --expunge does.

Are there any (potentially non-existent) directories that the repo rule could watch and that would be created by any suitable xcode installation?

[keith]

i can't vouch for bazel fetch --configure --force working because https://github.com/bazelbuild/bazel/issues/27042, but that would at least be nicer syntax if it does!

should work whenever bazel clean --expunge does.

is it possible the repo contents cache is taking over and caching the bad value between cleans? that's my guess as to why it's not working for some folks but I can't repro it offhand.

Are there any (potentially non-existent) directories that the repo rule could watch and that would be created by any suitable xcode installation?

There are so many intermediate states here that I think things get complicated. The case I see this in the most is people updating Xcode, and (my guess) running bazel while the update is ongoing, so spotlight doesn't return any results. But I believe during that time the /Applications/Xcode.app dir would be present the entire time.

I did find that /Library/Preferences/com.apple.dt.Xcode.plist is the file that is written when you accept the license agreement, so we could potentially watch that which might be better than nothing?

We're actually already trying really hard to invalidate things when Xcode changes, here are the current recommendations, summarized as:

• Set a startup option with the absolute path to Xcode to invalidate bazel's in memory cache of version -> path
• Set DEVELOPER_DIR and XCODE_VERSION as repo env, attempting to invalidated the repository rule here and the crosstool
• Set --xcode_version

My assumption as to why this isn't enough is that the path doesn't change on an upgrade, so the startup option nor the DEVELOPER_DIR change, and I guess given that we see this often I assume that xcodebuild -version returns the new version while spotlight would return no Xcode, so by the time the Xcode is valid you've already run bazel with the new XCODE_VERSION, and therefore it doesn't change and cause re-evaluation.

[fmeum]

Fixing #27042 shouldn't be hard, so let's do that.

is it possible the repo contents cache is taking over and caching the bad value between cleans?

I don't think that can happen since the relevant repo rule isn't marked as reproducible.

[keith]

The module extension is marked reproducible

[fmeum]

The module extension is marked reproducible

That just means that it is tracked in the persistent lockfile, not the one in the workspace. The repo rule itself would need to be marked reproducible for the repo cache to be used for it.

[keith]

ah 1 other case that I forgot to mention, but is probably the most common one we see, and related to my clean --expunge confusion above:

Users who only have a version of Xcode installed that is incompatible with their current macOS version. In that case xcode_locator returns nothing (arguably correct even though it might not matter) and they are often surprised by that. Maybe we should have a special warning for that case. I often try to steer folks to determine that this isn't the issue by asking them to open the Xcode gui

[jastice]

This reproduces on Mac when importing the Bazel repo into IntelliJ IDEA, as this will effectively invoke (a subset of) bazel build --keep_going //...

plugin issue: BAZEL-2747 Importing bazelbuild/bazel without configured projectview crashes Bazel