Upgrade or installation should stop if command line developer tools are missing

[tangix]

Provide a detailed description of the proposed feature

After upgrading to Monterey 12.6 a couple of days ago I just ran brew upgrade and got into a mess with multiple packages being removed that were scheduled for update. The reason seems to be a missing xcode CLI tools update that many packages required for the installation. The upgrade process never the less continued to remove the old version although the new version failed to install.

==> Upgrading node
  18.8.0 -> 18.9.0

==> Pouring node--18.9.0.monterey.bottle.tar.gz
2022-09-14 10:19:29.179 xcodebuild[11721:1312411] [MT] DVTPlugInLoading: Failed to load code for plug-in com.apple.dt.IDESimulatorAvailability (/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin), error = Error Domain=NSCocoaErrorDomain Code=3588 "dlopen(/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin/Contents/MacOS/IDESimulatorAvailability, 0x0109): Symbol not found: (_OBJC_CLASS_$_SimDiskImage)
  Referenced from: '/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin/Contents/MacOS/IDESimulatorAvailability'
  Expected in: '/Library/Developer/PrivateFrameworks/CoreSimulator.framework/Versions/A/CoreSimulator'" UserInfo={NSLocalizedFailureReason=The bundle couldn’t be loaded., NSLocalizedRecoverySuggestion=Try reinstalling the bundle., NSFilePath=/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin/Contents/MacOS/IDESimulatorAvailability, NSDebugDescription=dlopen(/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin/Contents/MacOS/IDESimulatorAvailability, 0x0109): Symbol not found: (_OBJC_CLASS_$_SimDiskImage)
  Referenced from: '/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin/Contents/MacOS/IDESimulatorAvailability'
  Expected in: '/Library/Developer/PrivateFrameworks/CoreSimulator.framework/Versions/A/CoreSimulator', NSBundlePath=/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin, NSLocalizedDescription=The bundle “IDESimulatorAvailability” couldn’t be loaded.}, dyldError = dlopen(/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin/Contents/MacOS/IDESimulatorAvailability, 0x0000): Symbol not found: (_OBJC_CLASS_$_SimDiskImage)
  Referenced from: '/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin/Contents/MacOS/IDESimulatorAvailability'
  Expected in: '/Library/Developer/PrivateFrameworks/CoreSimulator.framework/Versions/A/CoreSimulator'
2022-09-14 10:19:29.263 xcodebuild[11721:1312411] [MT] DVTAssertions: ASSERTION FAILURE in /System/Volumes/Data/SWE/Apps/DT/BuildRoots/BuildRoot2/ActiveBuildRoot/Library/Caches/com.apple.xbs/Sources/DVTFrameworks/DVTFrameworks-21303/DVTFoundation/PlugInArchitecture/DataModel/DVTPlugIn.m:374
Details:  Failed to load code for plug-in com.apple.dt.IDESimulatorAvailability (/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin)
Please ensure Xcode packages are up-to-date — try running 'xcodebuild -runFirstLaunch'.

NSBundle error: Error Domain=NSCocoaErrorDomain Code=3588 "dlopen(/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin/Contents/MacOS/IDESimulatorAvailability, 0x0109): Symbol not found: (_OBJC_CLASS_$_SimDiskImage)
  Referenced from: '/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin/Contents/MacOS/IDESimulatorAvailability'
  Expected in: '/Library/Developer/PrivateFrameworks/CoreSimulator.framework/Versions/A/CoreSimulator'" UserInfo={NSLocalizedFailureReason=The bundle couldn’t be loaded., NSLocalizedRecoverySuggestion=Try reinstalling the bundle., NSFilePath=/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin/Contents/MacOS/IDESimulatorAvailability, NSDebugDescription=dlopen(/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin/Contents/MacOS/IDESimulatorAvailability, 0x0109): Symbol not found: (_OBJC_CLASS_$_SimDiskImage)
  Referenced from: '/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin/Contents/MacOS/IDESimulatorAvailability'
  Expected in: '/Library/Developer/PrivateFrameworks/CoreSimulator.framework/Versions/A/CoreSimulator', NSBundlePath=/Applications/Xcode.app/Contents/PlugIns/IDESimulatorAvailability.ideplugin, NSLocalizedDescription=The bundle “IDESimulatorAvailability” couldn’t be loaded.}
Object:   <DVTPlugIn: 0x600001187de0>
Method:   -loadAssertingOnError:error:
Thread:   <_NSMainThread: 0x6000035d0440>{number = 1, name = main}
Hints:

Backtrace:
  0  0x000000010c327f81
  1  0x000000010c3274b5
  2  0x000000010c3276c1
  3  0x000000010c19e3ba
  4  0x000000010c159c73
  5  0x000000010c157a1f
  6  0x00007ff81f399317
  7  0x00007ff81f3a67ee
  8  0x000000010c3765a1
  9  0x000000010c34c0d8
 10  0x000000010c1578a3
 11  0x000000010c157af5
 12  0x000000010d23cf66
 13  0x000000010d23c526
 14  0x000000010d23b919
 15  0x000000010abc49a4
 16  0x000000010a94720a
sh: line 1: 11723 Abort trap: 6           /Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -find clang 2> /dev/null
clang: error: sh -c '/Applications/Xcode.app/Contents/Developer/usr/bin/xcodebuild -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk -find clang 2> /dev/null' failed with exit code 34304: (null) (errno=Invalid argument)
xcode-select: Failed to locate 'clang', requesting installation of command line developer tools.
🍺  /usr/local/Cellar/node/18.9.0: 1,997 files, 49.7MB
==> Running `brew cleanup node`...
Removing: /usr/local/Cellar/node/18.8.0... (1,984 files, 49.7MB)
Removing: /Users/msa/Library/Caches/Homebrew/node--18.8.0... (13.2MB)

What is the motivation for the feature?

Failed upgrades of packages should not trigger removal of old version making the system useless.

How will the feature be relevant to at least 90% of Homebrew users?

Many users will upgrade their OS but not think about upgrading xcode propely.

What alternatives to the feature have been considered?

brew upgrade should trigger installation of the CLI tools, not an individual package.

[carlocab]

This seems like a bug report and not a feature request.

What is the output of brew config and brew doctor?

[carlocab]

Failed upgrades of packages should not trigger removal of old version making the system useless.

node seemed to install just fine, despite the noisy error messages, though.

Which formulae failed to upgrade but were cleaned up regardless?

[tangix]

Failed upgrades of packages should not trigger removal of old version making the system useless.

node seemed to install just fine, despite the noisy error messages, though.

Which formulae failed to upgrade but were cleaned up regardless?

Can't get node or php to run after the upgrade. I also get regular prompts for installation of CLI tools when trying to get brew config or brew doctor output.

Will see if I can get the cli tools to install from xcode.

[tangix]

OK, launched xcode (I don't normally use it) and clicked through the prompt of installing additional SDKs (Watch OS 9 etc) and allowed xcode to restart. Restarted terminal and could capture doctor and config:

msa@verdandi ~ % brew doctor
Please note that these warnings are just used to help the Homebrew maintainers
with debugging if you file an issue. If everything you use Homebrew for is
working fine: please don't worry or file an issue; just ignore this. Thanks!

Warning: Some installed kegs have no formulae!
This means they were either deleted or installed manually.
You should find replacements for the following formulae:
  [email protected]
  gem-cfn-nag

Warning: Some installed formulae are deprecated or disabled.
You should find replacements for the following formulae:
  [email protected]
  [email protected]
  [email protected]
  [email protected]

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and which additional flags to use when
compiling and linking.

Having additional scripts in your path can confuse software installed via
Homebrew if the config script overrides a system or Homebrew-provided
script of the same name. We found the following "config" scripts:
  /Users/msa/.pyenv/shims/python-config
  /Users/msa/.pyenv/shims/python3-config
  /Users/msa/.pyenv/shims/python3.8-config

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause formulae that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  [email protected]

Warning: Some installed formulae are not readable:
  [email protected]: cannot load such file -- /usr/local/opt/[email protected]/lib/php_extension_formula

Warning: Homebrew's "sbin" was not found in your PATH but you have installed
formulae that put executables in /usr/local/sbin.
Consider setting your PATH for example like so:
  echo 'export PATH="/usr/local/sbin:$PATH"' >> ~/.zshrc
msa@verdandi ~ % brew config
HOMEBREW_VERSION: 3.6.1-33-g65da808
ORIGIN: https://github.com/Homebrew/brew
HEAD: 65da808a104d1cab0a6d1c5dde46b6ce85d60696
Last commit: 20 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 379fea565f21d730cf01987f5571fa0dc4823792
Core tap last commit: 85 minutes ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 12
Homebrew Ruby: 2.6.8 => /usr/local/Homebrew/Library/Homebrew/vendor/portable-ruby/2.6.8_1/bin/ruby
CPU: dodeca-core 64-bit kabylake
Clang: 14.0.0 build 1400
Git: 2.37.0 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Curl: 7.79.1 => /usr/bin/curl
macOS: 12.6-x86_64
CLT: 14.0.0.0.1.1661618636
Xcode: 14.0
msa@verdandi ~ %

[tangix]

Got continued prompts for installation of the CLI tools and ended up rebooting the system. Reinstalled php and node formulæ and all is well now. Maybe I've learned that I should not run $ brew update && brew upgrade but instead run the separately as I saw that brew update triggered the CLI install process but then continued with the upgrade of several packages.

¯_(ツ)_/¯

@carlocab Thanks for your help!

[lolychee]

xcodebuild -runFirstLaunch can fix it.

[MikeMcQuaid]

We should probably detect this state and tell the user to run xcodebuild -runFirstLaunch

[Bo98]

This appears to be a documented known issue:

Xcode 14 can fail to find tools using xcodebuild -find (which is used by xcrun and the wrappers in /usr/bin such as /usr/bin/clang) if first launch content isn’t installed. (98008921)

Workaround: Run xcodebuild -runFirstLaunch, or launch Xcode.app first.

https://developer.apple.com/documentation/xcode-release-notes/xcode-14-release-notes

[ifsheldon]

I encountered the same issue after Xcode was auto updated by App Store.

[MikeMcQuaid]

@Bo98 Good catch. I wonder if we can detect this somehow? I know brew doctor will try to run e.g. clang so perhaps we could try that first and check the stderr?

[tangix]

@Bo98 Good catch. I wonder if we can detect this somehow? I know brew doctor will try to run e.g. clang so perhaps we could try that first and check the stderr?

Could xcodebuild -checkFirstLaunchStatus help in checking the state of Xcode?

Seems to returning 69 if something needs to be done - https://github.com/xamarin/xamarin-macios/issues/3908

[MikeMcQuaid]

@tangix Good catch! Can someone with Xcode installed send the output of /usr/bin/time xcodebuild -checkFirstLaunchStatus here?

[tangix]

Hmmm. Checking my Mac Studio that is running brew but don't have xcode installed indicates there may be an issue with this approach:

msa@msa-studio ~ % xcodebuild -checkFirstLaunchStatus && echo $?
xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance
msa@msa-studio ~ % brew doctor
Your system is ready to brew.
msa@msa-studio ~ % brew config
HOMEBREW_VERSION: 3.6.1
ORIGIN: https://github.com/Homebrew/brew
HEAD: 6e2b162c4786e075323f038d46bfb566d91889e7
Last commit: 5 days ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: d5f09305793eb6caa97f4ec70a18d6f71d81bcd6
Core tap last commit: 89 minutes ago
Core tap branch: master
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_MAKE_JOBS: 10
Homebrew Ruby: 2.6.8 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 14.0.0 build 1400
Git: 2.37.0 => /Library/Developer/CommandLineTools/usr/bin/git
Curl: 7.79.1 => /usr/bin/curl
macOS: 12.6-arm64
CLT: 14.0.0.0.1.1661618636
Xcode: N/A
Rosetta 2: false

I guess the requirements for homebrew shouldn't include installing the full Xcode package?

[MikeMcQuaid]

@tangix Yeh, we can detect if Xcode isn't installed and only do this if it is, with existing code.

[tangix]

@MikeMcQuaid sounds like a good approach! The docs around -checkFirstLaunchStatus seems to be scarce but the Distributed Intelligence of Internet seems to agree that an exit code of 69 is bad and should trigger the user to open xcode and check what's going on (like agreeing to license agreement or click through an install SDK prompt)

[tangix]

@tangix Good catch! Can someone with Xcode installed send the output of /usr/bin/time xcodebuild -checkFirstLaunchStatus here?

msa@verdandi ~ % /usr/bin/time xcodebuild -checkFirstLaunchStatus
        0.37 real         0.27 user         0.06 sys

[zhixiangyao]

xcodebuild -runFirstLaunch can fix it.

Thx!

[ramsey]

I've been running into this issue for a while and finally found this issue. Running xcodebuild -runFirstLaunch appears to have resolved it for me, too.

Thanks!

[chrisfinazzo]

Related: According to brew config, downloading Xcode - outside of MAS - does not detect whether or not CLT is actually installed.

brew config
HOMEBREW_VERSION: 3.6.4-33-gc49f2e8
ORIGIN: https://github.com/Homebrew/brew
HEAD: c49f2e8b4c343844130b7517c84c3b1bec16a332
Last commit: 8 hours ago
Core tap ORIGIN: https://github.com/Homebrew/homebrew-core
Core tap HEAD: 77c33368d69c968b0fd3002671c1188265ae1bd6
Core tap last commit: 65 minutes ago
Core tap branch: master
HOMEBREW_PREFIX: /usr/local
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.n4Y2O0HAh1/org.xquartz:0
HOMEBREW_EDITOR: /usr/local/bin/bbedit
HOMEBREW_MAKE_JOBS: 8
Homebrew Ruby: 2.6.8 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: octa-core 64-bit icelake
Clang: 13.1.6 build 1316
Git: 2.38.0 => /usr/local/bin/git
Curl: 7.79.1 => /usr/bin/curl
macOS: 12.5.1-x86_64
CLT: N/A
Xcode: 13.4.1

While I am planning to move my Python install out of Homebrew proper and into Conda - which is itself managed by Homebrew - reporting that the tools are not installed seems incorrect.

This doesn't address the fact that the alternative brew suggests (compiling from source) isn't supported for some reason in cases where the OS thinks the tools aren't installed although they clearly are.

[MikeMcQuaid]

This doesn't address the fact that the alternative brew suggests (compiling from source) isn't supported for some reason in cases where the OS thinks the tools aren't installed although they clearly are.

They are not. The Xcode CLT and the CLT installed by xcode-select --install are not the same thing and we often need the latter, even if you have Xcode installed.

[tparvu]

Upgraded MacBook Pro to Ventura 13.0

brew upgrade was broken with missing xcrun.

xcode-select --install fixed my issue.

[robacarp]

xcode-select --install didn't fix my issue (missing clang) but xcodebuild -runFirstLaunch did.

[Bo98]

I recognise those who have already had the issue before won't be able to tell: but if anyone else has this issue it would be good to know if it still happens when upgrading directly to Xcode 14.1.

I tested an Xcode 13.4.1 -> Xcode 14.1 upgrade myself and did not have the issue occur.

[MikeMcQuaid]

Closing this out as there's no continued issue here.