pytorch
`error: 'Python.h' file not found` building on Windows with Python installed to `C:\Program Files\Python312`
🐛 Describe the bug
We're building torchvision downstream with ROcm on Windows support (https://github.com/ROCm/TheRock/issues/910#issuecomment-3099560950) using the clang-cl compiler, and I encountered issues as long as I used "Install Python 3.12 for all users" and Python from the default C:\Program Files\Python312 path (note the space in Program Files).
[!IMPORTANT] Using a "user install" of Python at a path like
C:\Users\<username>\AppData\Local\Programs\Python\Python312(note no spaces) does work.
Our build script is https://github.com/ROCm/TheRock/blob/main/external-builds/pytorch/build_prod_wheels.py, with this code for torchvision: https://github.com/ROCm/TheRock/blob/4b31640a70d73333d9cf7abde549a68dc07ccdbc/external-builds/pytorch/build_prod_wheels.py#L694-L727. That bottoms out at python setup.py bdist_wheel from the torchvision source directory with a few environment variables set.
Error logs look like this:
[1/8] D:\projects\TheRock\external-builds\pytorch\.venv\Lib\site-packages\_rocm_sdk_devel\lib\llvm\bin\clang-cl.exe /showIncludes /nologo /O2 /W3 /GL /DNDEBUG /MD -DWITH_HIP -Dtorchvision_EXPORTS -ID:\projects\TheRock\external-builds\pytorch\.venv\Lib\site-packages\torch\include -ID:\projects\TheRock\external-builds\pytorch\.venv\Lib\site-packages\torch\include\torch\csrc\api\include -ID:\projects\TheRock\external-builds\pytorch\.venv\include -IC:\Program\Files\Python312\include -IC:\Program\Files\Python312\Include -IC:\Program\Files\(x86)\Microsoft\Visual\Studio\2022\BuildTools\VC\Tools\MSVC\14.42.34433\include -IC:\Program\Files\(x86)\Microsoft\Visual\Studio\2022\BuildTools\VC\Tools\MSVC\14.42.34433\ATLMFC\include -IC:\Program\Files\(x86)\Microsoft\Visual\Studio\2022\BuildTools\VC\Auxiliary\VS\include -IC:\Program\Files\(x86)\Windows\Kits\10\include\10.0.26100.0\ucrt -IC:\Program\Files\(x86)\Windows\Kits\10\\include\10.0.26100.0\\um -IC:\Program\Files\(x86)\Windows\Kits\10\\include\10.0.26100.0\\shared -IC:\Program\Files\(x86)\Windows\Kits\10\\include\10.0.26100.0\\winrt -IC:\Program\Files\(x86)\Windows\Kits\10\\include\10.0.26100.0\\cppwinrt -IC:\Program\Files\(x86)\Windows\Kits\NETFXSDK\4.8\include\um /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /wd4624 /wd4067 /wd4068 /EHsc -c D:\b\vision_main\torchvision\csrc\io\image\cpu\decode_webp.cpp /FoD:\b\vision_main\build\temp.win-amd64-cpython-312\Release\b\vision_main\torchvision\csrc\io\image\cpu\decode_webp.obj -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fms-runtime-lib=dll -fms-runtime-lib=dll -D__HIP_PLATFORM_AMD__=1 -DUSE_ROCM=1 -DHIPBLAS_V2 -fms-runtime-lib=dll -fms-runtime-lib=dll /MP -g0 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=image /std:c++17
FAILED: D:/b/vision_main/build/temp.win-amd64-cpython-312/Release/b/vision_main/torchvision/csrc/io/image/cpu/decode_webp.obj
...
In file included from D:\projects\TheRock\external-builds\pytorch\.venv\Lib\site-packages\torch\include\torch/csrc/Device.h:4:
D:\projects\TheRock\external-builds\pytorch\.venv\Lib\site-packages\torch\include\torch/csrc/python_headers.h(12,10): warning: #include resolved using non-portable Microsoft search rules as: D:\projects\TheRock\external-builds\pytorch\.venv\Lib\site-packages\torch\include\torch\csrc\api\include\torch\Python.h [-Wmicrosoft-include]
12 | #include <Python.h>
| ^
D:\projects\TheRock\external-builds\pytorch\.venv\Lib\site-packages\torch\include\torch/csrc/python_headers.h(12,10): error: 'Python.h' file not found with <angled> include; use "quotes" instead
12 | #include <Python.h>
| ^~~~~~~~~~
| "Python.h"
D:\projects\TheRock\external-builds\pytorch\.venv\Lib\site-packages\torch\include\torch/csrc/python_headers.h(12,10): warning: non-portable path to file '<python.h>'; specified path differs in case from file name on disk [-Wnonportable-include-path]
12 | #include <Python.h>
| ^~~~~~~~~~
| <python.h>
D:\projects\TheRock\external-builds\pytorch\.venv\Lib\site-packages\torch\include\torch/csrc/python_headers.h(13,10): fatal error: 'frameobject.h' file not found
13 | #include <frameobject.h>
| ^~~~~~~~~~~~~~~
Note the -IC:\Program\Files\Python312\include that has \ where a space should be.
More complete logs here: https://gist.github.com/ScottTodd/5f8900ec2ff30d0d5c9d7ecf8c3be70e
I tried a few variations on setting the include path to pick up the right system path without much luck... the fix seems simple on the surface - just escape the space in that path.
- Set
TORCHVISION_INCLUDEtosysconfig.get_path("include")(wrapped in quotes or not) - Try finding the Python development package and including it via CMake:
find_package(Python3 REQUIRED COMPONENTS Development) # tried both of these include_directories("${Python3_INCLUDE_DIRS}") # include_directories(${Python3_INCLUDE_DIRS})
Versions
main branch from https://github.com/pytorch/pytorch and https://github.com/pytorch/vision, after running HIPIFY and applying some (I believe) unrelated patches from https://github.com/ROCm/TheRock/tree/main/external-builds/pytorch/patches/pytorch/main/pytorch/hipified.
Approximately https://github.com/pytorch/pytorch/commit/a7f3bdf5506 and https://github.com/pytorch/vision/commit/98f8b3757c0648724064ca95434b18281c43c5f6
Hi @ScottTodd ,
Sorry you're experiencing build issues. I won't be able to debug this on my end, but if you can find a simple fix e.g. from our setup.py, I think we would be able to consider merging it
Thanks! I'm unblocked for now, since I reinstalled Python to the users path (without spaces). I at least wanted to file the issue for tracking and so other users (or future-me) can see the error messages and find the workaround.
I'm curious if this is unique to our build configuration (PyTorch + ROCm, using clang-cl as the compiler).