[BUG] breaks numpy in cross-compiling environment

[th0ma7]

setuptools version

setuptools >= 64.x

Python version

python 3.10.x

OS

Linux

Additional environment information

Environment is using the Synology DSM toolchain for cross-compiling for multiple archs (armv5, v7, v8, ppc, x64, i686).

Description

While preparing python 3.10.6 update I've took care to update all build requirements, including setuptools. It hapens that it made numpy to fail to build starting with setuptools >= 64.x (also fails on 65.x). Full build logs can be found here: https://github.com/SynoCommunity/spksrc/runs/7917533394?check_suite_focus=true Associated SynoCommunity PR: https://github.com/SynoCommunity/spksrc/pull/5395

Expected behavior

numpy to build properly (like all other regular wheel being cross-compiled)

How to Reproduce

Can be reproduced easily using the SynoCommunity (spksrc) cross-compiling environment.

Output

Output is mostly hidden under pip call:

===>  _PYTHON_HOST_PLATFORM=x86_64-pc-linux-gnu /home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/cross/bin/pip wheel --disable-pip-version-check --no-binary :all: --cache-dir /home/spksrc/py310.6/spksrc/spk/bazarr/../../distrib/pip --no-deps --wheel-dir /home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/wheelhouse --no-build-isolation greenlet==1.1.2
Collecting greenlet==1.1.2
  Using cached greenlet-1.1.2.tar.gz (91 kB)
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Building wheels for collected packages: greenlet
  Building wheel for greenlet (setup.py): started
  Building wheel for greenlet (setup.py): finished with status 'done'
  Created wheel for greenlet: filename=greenlet-1.1.2-cp310-cp310-linux_x86_64.whl size=142558 sha256=d3ac376cd9b322f76ee60208a103be9f52d989d3b3a8a1185fb9857c4a28de25
  Stored in directory: /home/spksrc/py310.6/spksrc/distrib/pip/wheels/51/01/b7/16995feb30f4eed2504aed2260a4058a1164472c9be5ab44b1
Successfully built greenlet
make[3]: Leaving directory '/home/spksrc/py310.6/spksrc/spk/bazarr'
===>  [numpy]           
make[3]: Entering directory '/home/spksrc/py310.6/spksrc/spk/bazarr'
===>  _PYTHON_HOST_PLATFORM=x86_64-pc-linux-gnu /home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/cross/bin/pip wheel --disable-pip-version-check --no-binary :all: --cache-dir /home/spksrc/py310.6/spksrc/spk/bazarr/../../distrib/pip --no-deps --wheel-dir /home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/wheelhouse --no-build-isolation numpy==1.22.4
Collecting numpy==1.22.4
  Using cached numpy-1.22.4.zip (11.5 MB)
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'error'
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [75 lines of output]
      Running from numpy source directory.
      running dist_info
      running build_src
      INFO: build_src
      creating /tmp/pip-modern-metadata-8a5sb6si/numpy.egg-info
      writing /tmp/pip-modern-metadata-8a5sb6si/numpy.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-modern-metadata-8a5sb6si/numpy.egg-info/dependency_links.txt
      writing entry points to /tmp/pip-modern-metadata-8a5sb6si/numpy.egg-info/entry_points.txt
      writing top-level names to /tmp/pip-modern-metadata-8a5sb6si/numpy.egg-info/top_level.txt
      writing manifest file '/tmp/pip-modern-metadata-8a5sb6si/numpy.egg-info/SOURCES.txt'
      /home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/egg_info.py:643: SetuptoolsDeprecationWarning: Custom 'build_py' does not implement 'get_data_files_without_manifest'.
      Please extend command classes from setuptools instead of distutils.
        warnings.warn(
      INFO: unifing config_cc, config, build_clib, build_ext, build commands --compiler options
      INFO: unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
      Traceback (most recent call last):
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/build_meta.py", line 369, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/build_meta.py", line 474, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/build_meta.py", line 334, in run_setup
          exec(code, locals())
        File "<string>", line 461, in <module>
        File "<string>", line 453, in setup_package
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 973, in run_commands
          self.run_command(cmd)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 992, in run_command
          cmd_obj.run()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/dist_info.py", line 96, in run
          self.egg_info.run()
        File "/tmp/pip-wheel-zx645048/numpy_62d5a1a8355144f781d4c843ede91f36/numpy/distutils/command/egg_info.py", line 25, in run
          _egg_info.run(self)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 308, in run
          self.find_sources()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 316, in find_sources
          mm.run()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 560, in run
          self.add_defaults()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 597, in add_defaults
          sdist.add_defaults(self)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/sdist.py", line 107, in add_defaults
          self._add_defaults_build_sub_commands()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/sdist.py", line 127, in _add_defaults_build_sub_commands
          self.filelist.extend(chain.from_iterable(files))
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 503, in extend
          self.files.extend(filter(self._safe_path, paths))
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/sdist.py", line 126, in <genexpr>
          files = (c.get_source_files() for c in cmds if hasattr(c, "get_source_files"))
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/command/sdist.py", line 125, in <genexpr>
          cmds = (self.get_finalized_command(c) for c in missing_cmds)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 306, in get_finalized_command
          cmd_obj.ensure_finalized()
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 109, in ensure_finalized
          self.finalize_options()
        File "/tmp/pip-wheel-zx645048/numpy_62d5a1a8355144f781d4c843ede91f36/numpy/distutils/command/config_compiler.py", line 69, in finalize_options
          v = getattr(c, a)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/Cython/Distutils/old_build_ext.py", line 157, in __getattr__
          return _build_ext.build_ext.__getattr__(self, name)
        File "/home/spksrc/py310.6/spksrc/spk/bazarr/work-x64-6.1/crossenv/build/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 105, in __getattr__
          raise AttributeError(attr)
      AttributeError: fcompiler. Did you mean: 'compiler'?
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.
make[3]: *** [../../mk/spksrc.wheel.mk:139: cross-compile-wheel-numpy] Error 1
make[3]: Leaving directory '/home/spksrc/py310.6/spksrc/spk/bazarr'
make[2]: *** [../../mk/spksrc.wheel.mk:81: build_wheel_target] Error 1
make[2]: Leaving directory '/home/spksrc/py310.6/spksrc/spk/bazarr'
make[1]: *** [../../mk/spksrc.spk.mk:603: build-arch-x64-6.1] Error 1
make[1]: Leaving directory '/home/spksrc/py310.6/spksrc/spk/bazarr'
make: *** [../../mk/spksrc.spk.mk:597: arch-x64-6.1] Error 2

[boegel]

@th0ma7 I'm hitting this same problem when installing numpy from source (no cross-compiling, but fine).

Did you make any progress on this?

[freemedom]

@th0ma7 I'm hitting this same problem when installing numpy from source (no cross-compiling, but fine).

Did you make any progress on this?

Try downgrading the setuptools version?

[th0ma7]

Downgrading setuptools is all i could find for now

[mboisson]

Hitting the same issue. Need to downgrade setuptools to <64 on our system.

[jeanbez]

Hitting the same issue in MacOS, the solution to downgrade setuptools to <64 worked.

[mboisson]

This seems to have disappeared for me with recent versions of setuptools and/or numpy

[th0ma7]

@mboisson Thnx! indeed it did and can confirm https://github.com/SynoCommunity/spksrc/pull/5866

[mboisson]

To be more specific, after more testing, setuptools>63 is broken to build numpy<1.24, but numpy>=1.24 builds fine with the latest setuptools.

[th0ma7]

I concur, you are absolutely and had to revert back to older setuptools due to a need for an older numpy. But indeed now working with 1.24+

[ReluXingZeng]

I am having this issue with setuptools==65.5.1 as well as numpy==1.26.0, so I am not sure if the newest numpy (as of writing, 1.26.2) has already fixed for setuptools.

update: upgrade to newest setuptools setuptools==69.0.2 and numpy==1.26.2 also does not work for me. Ubuntu 22 is my operating system so unfortunately I can't downgrade to numpy<1.24