pypa
[BUG] can't link c++ so if python is compiled with ccache since 72.2.0
setuptools version
setuptools >= 72.2.0
Python version
python 3.10
OS
debian 12
Additional environment information
No response
Description
I have a python compiled by pyenv with ccache, and since 72.2.0, setuptools can't link so as expected.
(bencode2) trim21@omv ~/proj/bencode2 $ python setup.py build_ext -i --force
running build_ext
building 'bencode2.__bencode' extension
ccache /usr/bin/c++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -O3 -Wall -march=native -mtune=native -fPIC -DFMT_HEADER_ONLY= -I./src/bencode2 -I./vendor/fmt/include -I./vendor/small_vector/source/include -I/home/trim21/proj/bencode2/.venv/lib/python3.10/site-packages/pybind11/include -I/home/trim21/proj/bencode2/.venv/include -I/home/trim21/.pyenv/versions/3.10.15/include/python3.10 -c ./src/bencode2/bencode.cpp -o build/temp.linux-x86_64-cpython-310/./src/bencode2/bencode.o -fvisibility=hidden -g0 -std=c++17
ccache /usr/bin/c++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -O3 -Wall -march=native -mtune=native /usr/bin/c++ -shared build/temp.linux-x86_64-cpython-310/./src/bencode2/bencode.o -L/home/trim21/.pyenv/versions/3.10.15/lib -o build/lib.linux-x86_64-cpython-310/bencode2/__bencode.cpython-310-x86_64-linux-gnu.so
/usr/bin/ld: cannot use executable file '/usr/bin/c++' as input to a link
collect2: error: ld returned 1 exit status
error: command '/usr/bin/ccache' failed with exit code 1
(bencode2) trim21@omv ~/proj/bencode2 $ python -m sysconfig|grep ccache
BLDSHARED = "ccache /usr/bin/cc -shared -L/home/trim21/.pyenv/versions/3.10.15/lib -Wl,-rpath,/home/trim21/.pyenv/versions/3.10.15/lib -L/home/trim21/.pyenv/versions/3.10.15/lib -Wl,-rpath,/home/trim21/.pyenv/versions/3.10.15/lib"
CC = "ccache /usr/bin/cc"
CONFIG_ARGS = "'--prefix=/home/trim21/.pyenv/versions/3.10.15' '--libdir=/home/trim21/.pyenv/versions/3.10.15/lib' '--enable-shared' 'CC=ccache /usr/bin/cc' 'CFLAGS= -march=native -mtune=native' 'LDFLAGS=-L/home/trim21/.pyenv/versions/3.10.15/lib -Wl,-rpath,/home/trim21/.pyenv/versions/3.10.15/lib' 'LIBS=-L/home/trim21/.pyenv/versions/3.10.15/lib -Wl,-rpath,/home/trim21/.pyenv/versions/3.10.15/lib' 'CPPFLAGS=-I/home/trim21/.pyenv/versions/3.10.15/include'"
CXX = "ccache /usr/bin/c++"
LDCXXSHARED = "ccache /usr/bin/c++ -shared"
LDSHARED = "ccache /usr/bin/cc -shared -L/home/trim21/.pyenv/versions/3.10.15/lib -Wl,-rpath,/home/trim21/.pyenv/versions/3.10.15/lib -L/home/trim21/.pyenv/versions/3.10.15/lib -Wl,-rpath,/home/trim21/.pyenv/versions/3.10.15/lib"
LINKCC = "ccache /usr/bin/cc"
MAINCC = "ccache /usr/bin/cc"
Expected behavior
it should build so as expected
How to Reproduce
you will need to install pyenv, compile python with https://github.com/pyenv/pyenv-ccache
then use this python to build binary module,
for example
git clone https://github.com/trim21/bencode-py
cd bencode-py
pyenv local 3.10
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python setup.py build_ext -i --force
Output
(bencode2) trim21@omv ~/proj/bencode2 $ python setup.py build_ext -i --force -g
running build_ext
building 'bencode2.__bencode' extension
ccache /usr/bin/c++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -O3 -Wall -march=native -mtune=native -fPIC -g -DFMT_HEADER_ONLY= -I./src/bencode2 -I./vendor/fmt/include -I./vendor/small_vector/source/include -I/home/trim21/proj/bencode2/.venv/lib/python3.10/site-packages/pybind11/include -I/home/trim21/proj/bencode2/.venv/include -I/home/trim21/.pyenv/versions/3.10.15/include/python3.10 -c ./src/bencode2/bencode.cpp -o build/temp.linux-x86_64-cpython-310/./src/bencode2/bencode.o -fvisibility=hidden -g0 -std=c++17
ccache /usr/bin/c++ -Wno-unused-result -Wsign-compare -DNDEBUG -g -O3 -Wall -march=native -mtune=native /usr/bin/c++ -shared -g build/temp.linux-x86_64-cpython-310/./src/bencode2/bencode.o -L/home/trim21/.pyenv/versions/3.10.15/lib -o build/lib.linux-x86_64-cpython-310/bencode2/__bencode.cpython-310-x86_64-linux-gnu.so
/usr/bin/ld: cannot use executable file '/usr/bin/c++' as input to a link
collect2: error: ld returned 1 exit status
error: command '/usr/bin/ccache' failed with exit code 1
some testing:
54936c73b is good
64f10c582 is bad
this is caused by https://github.com/pypa/setuptools/pull/4539
I ran into this problem myself today (cpython 3.13.1), the symptom is somewhat different though, even though ccache was used to compile cpython (manual build and install of cpython from source), the linking command does not reference ccache, but it repeats the name of the compiler twice:
clang++ -isystem /opt-3/sundials-6.7.0-asan/include -fsanitize=address -stdlib++-isystem /opt-2/libcxx19-asan/include/c++/v1 -ferror-limit=5 -fPIC -DPYCVODES_NO_KLU=0 -DPYCVODES_NO_LAPACK=0 -DANYODE_NO_LAPACK=0 -I/tmp/pip-build-env-uzzybwax/overlay/lib/python3.13/site-packages/numpy/_core/include -Ipycvodes/include -Iexternal/anyode/include -I/opt-3/cpython-v3.13.1-asan/include/python3.13d -c pycvodes/_cvodes.cpp -o build/temp.linux-x86_64-cpython-313-pydebug/pycvodes/_cvodes.o -DVERSION_INFO=\"0.15.0.dev0+git\" -std=c++17
In file included from pycvodes/_cvodes.cpp:1280:
In file included from /tmp/pip-build-env-uzzybwax/overlay/lib/python3.13/site-packages/numpy/_core/include/numpy/arrayobject.h:5:
In file included from /tmp/pip-build-env-uzzybwax/overlay/lib/python3.13/site-packages/numpy/_core/include/numpy/ndarrayobject.h:12:
In file included from /tmp/pip-build-env-uzzybwax/overlay/lib/python3.13/site-packages/numpy/_core/include/numpy/ndarraytypes.h:1913:
/tmp/pip-build-env-uzzybwax/overlay/lib/python3.13/site-packages/numpy/_core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
17 | #warning "Using deprecated NumPy API, disable it with " \
| ^
1 warning generated.
clang++ -isystem /opt-3/sundials-6.7.0-asan/include -fsanitize=address -stdlib++-isystem /opt-2/libcxx19-asan/include/c++/v1 -ferror-limit=5 clang++ -shared -fsanitize=address -Wl,-rpath=/opt-3/cpython-v3.13.1-asan/lib -lsundials_nvecserial -lsundials_cvodes -lsundials_sunlinsolspgmr -lsundials_sunlinsolspbcgs -lsundials_sunlinsolsptfqmr -lsundials_sunmatrixdense -lsundials_sunmatrixband -lsundials_sunlinsollapackdense -lsundials_sunlinsollapackband -lsundials_sunlinsolklu -Wl,--disable-new-dtags -Wl,-rpath,/opt-3/sundials-6.7.0-asan/lib -L/opt-3/sundials-6.7.0-asan/lib -lopenblas -fsanitize=address -Wl,-rpath,/opt-2/libcxx19-asan/lib -L/opt-2/libcxx19-asan/lib -lc++ -lc++abi -stdlib=libc++ -isystem /opt-3/sundials-6.7.0-asan/include -fsanitize=address -stdlib++-isystem /opt-2/libcxx19-asan/include/c++/v1 -ferror-limit=5 build/temp.linux-x86_64-cpython-313-pydebug/pycvodes/_cvodes.o -lsundials_nvecserial -lsundials_cvodes -lsundials_sunlinsolspgmr -lsundials_sunlinsolspbcgs -lsundials_sunlinsolsptfqmr -lsundials_sunmatrixdense -lsundials_sunmatrixband -lsundials_sunlinsollapackdense -lsundials_sunlinsollapackband -lsundials_sunlinsolklu -o build/lib.linux-x86_64-cpython-313-pydebug/pycvodes/_cvodes.cpython-313d-x86_64-linux-gnu.so
clang++: error: no such file or directory: 'clang++'
[end of output]
If I pin "setuptools==72.1.0" in pyproject.toml, the extension compiles and links just fine.
I'm also running into this issue when I run pip install jupyter-ai-magics[all] on Python 3.13.2:
Install the project...
-- Install configuration: ""
-- Installing: /tmp/pip-install-ql1aronn/sentencepiece_a62c3b2ebfc242dfad476333de2070db/build/root/lib/pkgconfig/sentencepiece.pc
-- Installing: /tmp/pip-install-ql1aronn/sentencepiece_a62c3b2ebfc242dfad476333de2070db/build/root/lib/libsentencepiece.a
-- Installing: /tmp/pip-install-ql1aronn/sentencepiece_a62c3b2ebfc242dfad476333de2070db/build/root/lib/libsentencepiece_train.a
-- Installing: /tmp/pip-install-ql1aronn/sentencepiece_a62c3b2ebfc242dfad476333de2070db/build/root/bin/spm_encode
-- Installing: /tmp/pip-install-ql1aronn/sentencepiece_a62c3b2ebfc242dfad476333de2070db/build/root/bin/spm_decode
-- Installing: /tmp/pip-install-ql1aronn/sentencepiece_a62c3b2ebfc242dfad476333de2070db/build/root/bin/spm_normalize
-- Installing: /tmp/pip-install-ql1aronn/sentencepiece_a62c3b2ebfc242dfad476333de2070db/build/root/bin/spm_train
-- Installing: /tmp/pip-install-ql1aronn/sentencepiece_a62c3b2ebfc242dfad476333de2070db/build/root/bin/spm_export_vocab
-- Installing: /tmp/pip-install-ql1aronn/sentencepiece_a62c3b2ebfc242dfad476333de2070db/build/root/include/sentencepiece_trainer.h
-- Installing: /tmp/pip-install-ql1aronn/sentencepiece_a62c3b2ebfc242dfad476333de2070db/build/root/include/sentencepiece_processor.h
## cflags=-std=c++17 -I./build/root/include -Wl,-strip-all
## libs=./build/root/lib/libsentencepiece.a ./build/root/lib/libsentencepiece_train.a -Wl,-strip-all -Wl,-Bsymbolic
building 'sentencepiece._sentencepiece' extension
creating build/temp.linux-x86_64-cpython-313/src/sentencepiece
ccache /usr/bin/c++ -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -march=native -mtune=native -fPIC -I/home/joren/.pyenv/versions/3.13.2/include/python3.13 -c src/sentencepiece/sentencepiece_wrap.cxx -o build/temp.linux-x86_64-cpython-313/src/sentencepiece/sentencepiece_wrap.o -std=c++17 -I./build/root/include -Wl,-strip-all
c++: warning: -Wl,-strip-all: 'linker' input unused [-Wunused-command-line-argument]
src/sentencepiece/sentencepiece_wrap.cxx:3349:13: warning: unused function 'ConvertToUnicodeSpans<sentencepiece::ImmutableNBestSentencePieceText>' [-Wunused-function]
3349 | inline void ConvertToUnicodeSpans(sentencepiece::ImmutableNBestSentencePieceText *proto) {
| ^~~~~~~~~~~~~~~~~~~~~
1 warning generated.
ccache /usr/bin/c++ -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -march=native -mtune=native /usr/bin/c++ -shared -L/home/joren/.pyenv/versions/3.13.2/lib -Wl,-rpath,/home/joren/.pyenv/versions/3.13.2/lib -L/home/joren/.pyenv/versions/3.13.2/lib -Wl,-rpath,/home/joren/.pyenv/versions/3.13.2/lib build/temp.linux-x86_64-cpython-313/src/sentencepiece/sentencepiece_wrap.o -L/home/joren/.pyenv/versions/3.13.2/lib -o build/lib.linux-x86_64-cpython-313/sentencepiece/_sentencepiece.cpython-313-x86_64-linux-gnu.so ./build/root/lib/libsentencepiece.a ./build/root/lib/libsentencepiece_train.a -Wl,-strip-all -Wl,-Bsymbolic
/usr/bin/ld: cannot use executable file '/usr/bin/c++' as input to a link
c++: error: linker command failed with exit code 1 (use -v to see invocation)
error: command '/usr/bin/ccache' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for sentencepiece
Failed to build sentencepiece
ERROR: Failed to build installable wheels for some pyproject.toml based projects (sentencepiece)
Using update-alternatives --config c++ to switch from /usr/bin/g++ to /usr/bin/clang++ does not make a difference.
edit:
this might also be an issue with sentencepiece: https://github.com/google/sentencepiece/issues/1083
🤷🏻
edit2:
nevermind, I'm also facing this when I pip install pytype
I send a PR and never get review, so I close it when I stop using setuptools in my projects.
Fell free to cherry pick if you find it useful.
@trim21 understandable. I don't know why CI runs on such an important repo as setuptools does not include configurations using ccache, especially since this is not even the first time there have been regressions with respect to it. (I've reported at least one regression a while back). Out of curiosity: what are/will you be using instead? (I feel that I too might need to move on to something more actively maintained).
@trim21 understandable. I don't know why CI runs on such an important repo as setuptools include configurations using ccache, especially since this is not even the first time there have been regressions with respect to it. (I've reported at least one regression a while back). Out of curiosity: what are/will you be using instead? (I feel that I too might need to move on to something more actively maintained).
https://github.com/mesonbuild/meson with it's pep517 build system https://github.com/mesonbuild/meson-python