Python 3.12 fails to build on M1 - 'Modules/_ssl.cpython-312-darwin.so': No such file or directory

[kyhyco]

Describe the bug There are similar issues for pyenv - https://github.com/pyenv/pyenv/issues/2805 and https://github.com/pyenv/pyenv/issues/2823

Can't install python 3.12. Full error log

Error snippet

 ...
/opt/homebrew/bin/ginstall -c -m 755 Modules/_sha3.cpython-312-darwin.so /Users/spacefuture/.local/share/rtx/installs/python/3.12.0/lib/python3.12/lib-dynload/_sha3.cpython-312-darwin.so
/opt/homebrew/bin/ginstall -c -m 755 Modules/_blake2.cpython-312-darwin.so /Users/spacefuture/.local/share/rtx/installs/python/3.12.0/lib/python3.12/lib-dynload/_blake2.cpython-312-darwin.so
ginstall: cannot stat 'Modules/_blake2.cpython-312-darwin.so': No such file or directory
make: *** [sharedinstall] Error 1
BUILD FAILED (OS X 14.0 using python-build 20180424)
Inspect or clean up the working tree at /var/folders/kh/b7d8n69d25j56c1nw5tb0p800000gn/T/python-build.20231110142528.37566
Results logged to /var/folders/kh/b7d8n69d25j56c1nw5tb0p800000gn/T/python-build.20231110142528.37566.log
Last 10 log lines:
/opt/homebrew/bin/ginstall -c -m 755 Modules/_dbm.cpython-312-darwin.so /Users/spacefuture/.local/share/rtx/installs/python/3.12.0/lib/python3.12/lib-dynload/_dbm.cpython-312-darwin.so
/opt/homebrew/bin/ginstall -c -m 755 Modules/_gdbm.cpython-312-darwin.so /Users/spacefuture/.local/share/rtx/installs/python/3.12.0/lib/python3.12/lib-dynload/_gdbm.cpython-312-darwin.so
/opt/homebrew/bin/ginstall -c -m 755 Modules/readline.cpython-312-darwin.so /Users/spacefuture/.local/share/rtx/installs/python/3.12.0/lib/python3.12/lib-dynload/readline.cpython-312-darwin.so
/opt/homebrew/bin/ginstall -c -m 755 Modules/_md5.cpython-312-darwin.so /Users/spacefuture/.local/share/rtx/installs/python/3.12.0/lib/python3.12/lib-dynload/_md5.cpython-312-darwin.so
/opt/homebrew/bin/ginstall -c -m 755 Modules/_sha1.cpython-312-darwin.so /Users/spacefuture/.local/share/rtx/installs/python/3.12.0/lib/python3.12/lib-dynload/_sha1.cpython-312-darwin.so
/opt/homebrew/bin/ginstall -c -m 755 Modules/_sha2.cpython-312-darwin.so /Users/spacefuture/.local/share/rtx/installs/python/3.12.0/lib/python3.12/lib-dynload/_sha2.cpython-312-darwin.so
/opt/homebrew/bin/ginstall -c -m 755 Modules/_sha3.cpython-312-darwin.so /Users/spacefuture/.local/share/rtx/installs/python/3.12.0/lib/python3.12/lib-dynload/_sha3.cpython-312-darwin.so
/opt/homebrew/bin/ginstall -c -m 755 Modules/_blake2.cpython-312-darwin.so /Users/spacefuture/.local/share/rtx/installs/python/3.12.0/lib/python3.12/lib-dynload/_blake2.cpython-312-darwin.so
ginstall: cannot stat 'Modules/_blake2.cpython-312-darwin.so': No such file or directory
make: *** [sharedinstall] Error 1
Error:
   0: /Users/spacefuture/Library/Caches/rtx/python/pyenv/plugins/python-build/bin/python-build exited with non-zero status: exit code 1

Location:
   src/cmd.rs:265

Version:
   2023.11.2 macos-arm64 (2023-11-09)

OS: macOS Sonama 14.0 Arch: arm64 (M1) Shell: zsh

To Reproduce

1. Install rtx - brew install rtx
2. Add eval "$(rtx activate zsh)" to .zshrc
3. Restart terminal and run rtx install [email protected]

Expected behavior Python 3.12 should install without issues.

rtx doctor output

rtx version:
  2023.11.2 macos-arm64 (2023-11-09)

build:
  Target: aarch64-apple-darwin
  Features: BREW, DEFAULT, NATIVE_TLS
  Built: Thu, 9 Nov 2023 19:04:17 +0000
  Rust Version: rustc 1.72.1 (d5c2e9c34 2023-09-13) (Homebrew)
  Profile: release

shell:
  /bin/zsh
  zsh 5.9 (x86_64-apple-darwin23.0)

rtx data directory:
  /Users/spacefuture/.local/share/rtx

rtx environment variables:
  RTX_SHELL=zsh

settings:
  {"always_keep_download": "false", "always_keep_install": "false", "asdf_compat": "false", "disable_default_shorthands": "false", "disable_tools": "[]", "experimental": "false", "jobs": "4", "legacy_version_file": "true", "legacy_version_file_disable_tools": "[]", "log_level": "INFO", "missing_runtime_behavior": "warn", "plugin_autoupdate_last_check_duration": "10080", "raw": "false", "trusted_config_paths": "[]", "verbose": "false", "yes": "false"}

config files:

plugins:
  go       (core)
  java     (core)
  node     (core)
  python   (core)
  ruby     (core)
  rust     https://github.com/code-lever/asdf-rust.git#95acf4f

toolset:


No problems found

Dependency Info Xcode info:

❯ pkgutil --pkg-info=com.apple.pkg.CLTools_Executables
package-id: com.apple.pkg.CLTools_Executables
version: 15.0.0.0.1.1694021235
volume: /
location: /
install-time: 1697401497

[kyhyco]

As reference rtx install [email protected] works

[kyhyco]

Updated to macOS Sonoma 14.1.1 and Xcode 15.0.1 and get the same error.

[kyhyco]

Only one solution worked out of all the recommendations from similar threads:

brew unlink pkg-config
rtx install [email protected]
brew link pkg-config

Check with this command to see which packages has dependencies to pkg-config

brew uses --installed pkg-config 

[kyhyco]

Here is the explanation - https://github.com/pyenv/pyenv/issues/2823#issuecomment-1769081965

Homebrew installs pkg-config under a name that shadows the system's one.

This stops one from being able to build software using the system toolchain -- because the flags it produces only work out of the box if using a Homebrew-provided compiler.

Pyenv doesn't use packages from Homebrew in Linux by default. That's because Homebrew is more problematic to use than distro-provided packages. Unlike most distros, Homebrew is rolling-release -- and as such, its dependency package upgrades break existing Pyenv-provided installations whenever their .so names change.

To build with the system's toolchain,

uninstall or at least brew unlink Homebrew-provided pkg-config; or remove Homebrew from PATH To build with Homebrew's toolchain,

install a Homebrew-provided compiler and direct any builds to use it with export CC=<compiler executable to use>

[BoscoDomingo]

This happened to me too on Ubuntu 22.04 (WSL) due to using Homebrew. The solution was to run:

brew unlink pkg-config && \
CFLAGS="-I$(brew --prefix openssl)/include" \
LDFLAGS="-L$(brew --prefix openssl)/lib" \
rtx install python@latest; \
brew link pkg-config

This will temporarily unlink pkg-config and undo it after the installation (even if it fails). The CFLAGS and LDFLAGS might not be necessary in your case, but otherwise I get openssl errors and whatnot. It took about 2 minutes, so be patient

[BoscoDomingo]

@jdx Might want to include this in the docs as a workaround for common errors. This may happen often with Python 3.12, idk

[jdx]

you're welcome to make some modifications to https://github.com/jdx/rtx/blob/main/docs/python.md

[BoscoDomingo]

you're welcome to make some modifications to https://github.com/jdx/rtx/blob/main/docs/python.md

Done in the mise-docs repo :+1:

[jdx]

thanks!