Cannot install gensim with pip 23.1

[kennysong]

Problem description

Installing gensim==3.8.3 fails with pip 23.1 (the latest version).

Steps/code/corpus to reproduce

This works:

> python -m venv tmp_venv1 && \
source tmp_venv1/bin/activate && \
pip install --upgrade pip==23.0.1 && \
pip install gensim==3.8.3

Collecting pip==23.0.1
  Using cached pip-23.0.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Uninstalling pip-22.0.4:
      Successfully uninstalled pip-22.0.4
Successfully installed pip-23.0.1
Collecting gensim==3.8.3
  Using cached gensim-3.8.3.tar.gz (23.4 MB)
  Preparing metadata (setup.py) ... done
Collecting numpy>=1.11.3
  Using cached numpy-1.24.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (14.0 MB)
Collecting scipy>=0.18.1
  Using cached scipy-1.10.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (31.0 MB)
Collecting six>=1.5.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting smart_open>=1.8.1
  Using cached smart_open-6.3.0-py3-none-any.whl (56 kB)
Installing collected packages: smart_open, six, numpy, scipy, gensim
  DEPRECATION: gensim is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for gensim ... done
Successfully installed gensim-3.8.3 numpy-1.24.2 scipy-1.10.1 six-1.16.0 smart_open-6.3.0

[notice] A new release of pip is available: 23.0.1 -> 23.1
[notice] To update, run: pip install --upgrade pip

This fails:

> python -m venv tmp_venv2 && \
source tmp_venv2/bin/activate && \
pip install --upgrade pip==23.1 && \
pip install gensim==3.8.3

Collecting pip==23.1
  Using cached pip-23.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 22.0.4
    Uninstalling pip-22.0.4:
      Successfully uninstalled pip-22.0.4
Successfully installed pip-23.1
Collecting gensim==3.8.3
  Using cached gensim-3.8.3.tar.gz (23.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [55 lines of output]
      running dist_info
      creating /tmp/pip-modern-metadata-2_uusn3i/gensim.egg-info
      writing /tmp/pip-modern-metadata-2_uusn3i/gensim.egg-info/PKG-INFO
      writing dependency_links to /tmp/pip-modern-metadata-2_uusn3i/gensim.egg-info/dependency_links.txt
      writing requirements to /tmp/pip-modern-metadata-2_uusn3i/gensim.egg-info/requires.txt
      writing top-level names to /tmp/pip-modern-metadata-2_uusn3i/gensim.egg-info/top_level.txt
      writing manifest file '/tmp/pip-modern-metadata-2_uusn3i/gensim.egg-info/SOURCES.txt'
      Traceback (most recent call last):
        File "/workspaces/citadel-lens-2/tmp_venv2/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/workspaces/citadel-lens-2/tmp_venv2/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/workspaces/citadel-lens-2/tmp_venv2/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 377, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 484, in run_setup
          super(_BuildMetaLegacyBackend,
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 367, in <module>
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 1221, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/command/dist_info.py", line 98, in run
          self.egg_info.run()
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 318, in run
          self.find_sources()
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 326, in find_sources
          mm.run()
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 570, in run
          self.add_defaults()
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 608, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/command/sdist.py", line 106, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/_distutils/command/sdist.py", line 335, in _add_defaults_ext
          build_ext = self.get_finalized_command('build_ext')
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 305, in get_finalized_command
          cmd_obj.ensure_finalized()
        File "/tmp/pip-build-env-yeoyhj63/overlay/lib/python3.8/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
          self.finalize_options()
        File "<string>", line 111, in finalize_options
      AttributeError: 'dict' object has no attribute '__NUMPY_SETUP__'
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Versions

These are version from the first venv with pip 23.0.1 which was able to install gensim.

Python 3.8.16 (default, Mar  5 2023, 08:24:22) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform; print(platform.platform())
Linux-5.15.49-linuxkit-aarch64-with-glibc2.29
>>> import sys; print("Python", sys.version)
Python 3.8.16 (default, Mar  5 2023, 08:24:22) 
[GCC 10.2.1 20210110]
>>> import struct; print("Bits", 8 * struct.calcsize("P"))
Bits 64
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.24.2
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.10.1
>>> import gensim; print("gensim", gensim.__version__)
gensim 3.8.3
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION 1

[piskvorky]

@kennysong sorry, old versions are Gensim are no longer supported. Any reason why you're not using the latest Gensim = v4.3.1?

[kennysong]

Hi @piskvorky, gensim 3.8.3 is a dependency of PyCaret 2.3.10, and we need to run that specific version for compatibility with old models from 2022.

It is possible for us to pin the version of pip as a workaround, though, if the fix can't be backported to an older version of gensim.

[piskvorky]

What's the problem with those "old models from 2022", specifically?

The model migration (if needed at all) should be cosmetic. There have been many optimizations and fixes since 3.8.3, so moving on to 4.3.1 is recommended. It's not just about Gensim as such, but its dependencies (numpy, pip…) that have changed.

[kennysong]

Sorry for the confusion. It's a PyCaret model from 2022 which uses gensim internally (or at least depends on the package gensim==3.8.3).