Adapt python3-config to produce correct output in a virtual environment

[A-CGray]

Related to https://github.com/pyenv/pyenv-virtualenv/issues/462

Steps to reproduce

1. Create a new virtualenv and activate it:

$ pyenv virtualenv 3.12.11 test
$ pyenv activate test

2. Check that python3-config points to pyenv's shim

$ which python3-config
/home/ali/.pyenv/shims/python3-config

3. Check where python3-config include directories are (same directory is listed twice for some reason?):

$ python3-config --includes
-I/home/ali/.pyenv/versions/3.12.11/envs/test/include/python3.12 -I/home/ali/.pyenv/versions/3.12.11/envs/test/include/python3.12

4. Check what's in that directory (nothing)

ls -al /home/ali/.pyenv/versions/3.12.11/envs/test/include/python3.12
total 8
drwxr-xr-x 2 ali ali 4096 Aug 28 10:53 .
drwxr-xr-x 3 ali ali 4096 Aug 28 10:53 ..

Thus, whenever I try to compile something that needs to use python headers, I get the following error:

libsplinemodule.c:18:10: fatal error: Python.h: No such file or directory
   18 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.

I can get around this by linking the virtualenv include directory to the original pyenv include directory:

$ rm -rf /home/ali/.pyenv/versions/3.12.11/envs/test/include
$ ln -s /home/ali/.pyenv/versions/3.12.11/include /home/ali/.pyenv/versions/3.12.11/envs/test/include

Presumably pyenv-virtualenv should do something like this by default when you create the environment?

[native-api]

We've patched the build and added python-config to envs in https://github.com/pyenv/pyenv-virtualenv/issues/459 .

But in all appearance, python-config does not seem to be designed to run inside a virtual environment. It contains no code to detect it and alter its output accordingly.

So I'm marking this as a feature request.

[A-CGray]

Forgive me if I'm misunderstanding your point (I'm not at all familiar this kind of internal python stuff), but doesn't the pyenv shim for python3-config do exactly what you describe?

$ cat $(which python3-config)
#!/usr/bin/env bash
set -e
[ -n "$PYENV_DEBUG" ] && set -x

program="${0##*/}"

export PYENV_ROOT="/home/ali/.pyenv"
exec "/home/ali/.pyenv/libexec/pyenv" exec "$program" "$@"

This shim does correctly alter its output according to the currently activated environment:

$ python3-config --includes
-I/home/ali/.pyenv/versions/3.12.11/envs/test/include/python3.12 -I/home/ali/.pyenv/versions/3.12.11/envs/test/include/python3.12

The issue is just that there is nothing in these directories.

[native-api]

The issue is just that there is nothing in these directories.

And there shouldn't be. In a virtual environment, python-config should print include directories from the base installation.

[A-CGray]

Gotcha, so it should be pointing to /home/ali/.pyenv/versions/3.12.11/include in this case like I'm manually linking it to do at the moment?

[native-api]

https://github.com/python/cpython/blob/025a2135eff848abf24f9dc52c81386eea9da397/Lib/venv/init.py#L178-L188 :

        # PEP 405 says venvs should create a local include directory.
        # See https://peps.python.org/pep-0405/#include-files
        # XXX: This directory is not exposed in sysconfig or anywhere else, and
        #      doesn't seem to be utilized by modern packaging tools. We keep it
        #      for backwards-compatibility, and to follow the PEP, but I would
        #      recommend against using it, as most tooling does not pass it to
        #      compilers. Instead, until we standardize a site-specific include
        #      directory, I would recommend installing headers as package data,
        #      and providing some sort of API to get the include directories.
        #      Example: https://numpy.org/doc/2.1/reference/generated/numpy.get_include.html
        incpath = os.path.join(env_dir, 'Include' if os.name == 'nt' else 'include')