setuptools

setuptools copied to clipboard

setuptools version

setuptools==69.0.2

Python version

Python 3.11.6

OS

MacOS Sonoma (14.1.2)

Additional environment information

No response

Description

I am working with a project that uses Bazel to build a C++ extension via a setuptools.Extension and subclassing the build_ext.build_ext class.

In the current setup (which worked for a long time until about a month ago in CI), Bazel is symlinking its build directories directly into build_ext.build_temp.

Unfortunately, during the course of an editable install via pip (calling python -m pip install -e .), setuptools tries to copy the built extension from the build_ext.build_lib directory, which it evidently expects to contain the extensions. Here's the relevant source:

https://github.com/pypa/setuptools/blob/dd5f15a600a071ba00859e84fa497a9d1a25f521/setuptools/command/build_ext.py#L99-L101

Later in the build_ext.copy_extensions_to_source() API, these extensions are copied even if the assumed extension path (given by build_ext._get_inplace_equivalent()) does not exist.

https://github.com/pypa/setuptools/blob/dd5f15a600a071ba00859e84fa497a9d1a25f521/setuptools/command/build_ext.py#L105-L112

(L111 will still execute if the extension does not exist in the expected location, but is required).

This is exactly the case if, for example, a build tool with its own hermetic output directory (like Bazel) is used, and thus the install will fail.

This was a head scratcher for me until debugging a bit more, since python -m pip install . (fixed, not editable) worked just fine.

I am not sure whether this really qualifies as a bug - my hunch is that if I set an alternative build directory on the build_ext or the setuptools.Extension in the correct location, it might have continued working. But the sudden breakage, the editable vs. non-editable discrepancy and the customization docs not mentioning it made me go ahead with this report.

Is there a canonical way to correctly wire output directories of build tools in build_ext subclasses?

Expected behavior

It continues working.

How to Reproduce

NB: Having Bazel installed is a prerequisite for running the repro below.

git clone https://github.com/google/benchmark.git
cd benchmark
git checkout HEAD~5 # bug is already fixed on main, so rewind a couple of commits
python3 -m venv venv --system-site-packages --upgrade-deps # should fetch latest setuptools
source venv/bin/activate
python -m pip install -e .

Output

      Traceback (most recent call last):
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 156, in run
          self._create_wheel_file(bdist_wheel)
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 345, in _create_wheel_file
          files, mapping = self._run_build_commands(dist_name, unpacked, lib, tmp)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 268, in _run_build_commands
          self._run_build_subcommands()
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/command/editable_wheel.py", line 295, in _run_build_subcommands
          self.run_command(name)
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "<string>", line 56, in run
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 91, in run
          self.copy_extensions_to_source()
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 112, in copy_extensions_to_source
          self.copy_file(regular_file, inplace_file, level=self.verbose)
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 350, in copy_file
          return file_util.copy_file(
                 ^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/_distutils/file_util.py", line 115, in copy_file
          raise DistutilsFileError(
      distutils.errors.DistutilsFileError: can't copy '/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/tmpfg283hkc.build-lib/google_benchmark/_benchmark.cpython-311-darwin.so': doesn't exist or not a regular file
      /private/var/folders/rw/mj164vm16k10x_byww6my1s00000gn/T/pip-build-env-rimjgckl/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py:988: _DebuggingTips: Problem in editable installation.

NB: I fixed this in the project by overloading build_ext.copy_extensions_to_source() to pass, since we copy the extension into the source tree by hand right after the bazel build execution.

Guidance on how to make setuptools aware of alternate build locations to use would still be appreciated.

NB2: The build_ext.build_lib directory is never even created during the python -m pip install -e .

NB3: I did in fact not overload the initialize_options and finalize_options APIs as the docs suggested, but I'm not sure that would have made any difference.