Include directory prefers the more distant Python headers

[jaraco]

Following #200 (#178) (setuptools>=67.2), should the order of precedence be changed to prefer the nearest Python?

Environment:

• macOS
• ChimeraX
• System Python in /Library/Frameworks

What goes wrong:

• Builds in ChimeraX prefer Python.h from /Library instead of the headers in ChimeraX.

Expected:

• Builds should prefer headers from the local Python.

Originally posted by @zjp in https://github.com/pypa/distutils/issues/178#issuecomment-1419517007

[jaraco]

I don't know how to build ChimeraX, but I do know how to build indygreg/python-build-standalone, so I attempted the same repro:

$ git clone -q gh://indygreg/python-build-standalone
$ cd python-build-standalone
 python-build-standalone main $ git checkout -b stable 20230116
Switched to a new branch 'stable'
 python-build-standalone stable $ py build-macos.py e>o > /dev/null
 python-build-standalone stable $ cd dist
 dist stable $ pip-run jaraco.zstd -- -m jaraco.zstd -e *.zst
 dist stable $ cd python
 python stable $ install/bin/python3 -m pip install -q -U 'setuptools<67.2' pip
 python stable $ install/bin/python3 -c "import distutils.sysconfig as sc; print(sc.get_python_inc())"
/install/include/python3.10
 python stable $ install/bin/python3 -m pip install -q -U 'setuptools>=67.2'
 python stable $ install/bin/python3 -c "import distutils.sysconfig as sc; print(sc.get_python_inc())"
/Users/jaraco/draft/python-build-standalone/dist/python/install/include/python3.10
 python stable $ py -3.12 -c "import distutils.sysconfig as sc; print(sc.get_python_inc())"
/Library/Frameworks/Python.framework/Versions/3.12/include/python3.12

As you can see, at least for python-build-standalone, the Library/Frameworks name doesn't get picked up even though it's present on the system.

I suspect the issue is that ChimeraX has /Library/Frameworks/... set in sysconfigdata (configured). Can you confirm?

The whole point of #173 was to give precedence to the configured value so that it could take precedence over the unconfigured value (IIRC).

It seems there are lots of competing concerns and no coherent design that addresses all of these concerns.

If there's any hope of addressing these concerns, we're going to need some one to step up and devise a solution that understands these concerns and addresses them in a holistic way, ideally with tests that capture these concerns (though admittedly, that may be difficult given the level of integration required to replicate them). Is someone willing to take on this epic task?

[zjp]

I suspect our environment is misconfigured, since CONFINCLUDEPY, CONFINCLUDEDIR, INCLUDEDIR, and INCLUDEPY point to locations in /Library. Thank you for taking the time to double-check. Changing those values to the empty string lets us find the local includes, and only those.