Install issue when installing with poetry

[jaymegordo]

poetry add gensim

https://github.com/RaRe-Technologies/gensim/blob/ded78776284ad7b55b6626191eaa8dcea0dd3db0/setup.py#L107

getting AttributeError: 'dict' object has no attribute '__NUMPY_SETUP__' when trying to install gensim with poetry.

Seems to work on intel Macs, fails on M1

       Traceback (most recent call last):
          File "/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/tmpgt9xpffh_in_process.py", line 363, in <module>
            main()
          File "/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/tmpgt9xpffh_in_process.py", line 345, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/tmpgt9xpffh_in_process.py", line 164, in prepare_metadata_for_build_wheel
            return hook(metadata_directory, config_settings)
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 188, in prepare_metadata_for_build_wheel
            self.run_setup()
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 281, in run_setup
            super(_BuildMetaLegacyBackend,
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 174, in run_setup
            exec(compile(code, __file__, 'exec'), locals())
          File "setup.py", line 335, in <module>
            setup(
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 87, in setup
            return distutils.core.setup(**attrs)
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 177, in setup
            return run_commands(dist)
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 193, in run_commands
            dist.run_commands()
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
            self.run_command(cmd)
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1217, in run_command
            super().run_command(command)
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
            cmd_obj.run()
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/command/dist_info.py", line 35, in run
            egg_info.run()
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 307, in run
            self.find_sources()
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 314, in find_sources
            mm.run()
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 549, in run
            self.add_defaults()
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/command/egg_info.py", line 586, in add_defaults
            sdist.add_defaults(self)
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 252, in add_defaults
            self._add_defaults_ext()
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
            build_ext = self.get_finalized_command('build_ext')
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 304, in get_finalized_command
            cmd_obj.ensure_finalized()
          File "/private/var/folders/06/vq0z1ctd6yx5c3kgfz1bh4n00000gn/T/pip-build-env-yx8vfexi/overlay/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 107, in ensure_finalized
            self.finalize_options()
          File "setup.py", line 107, in finalize_options
            __builtins__.__NUMPY_SETUP__ = False
        AttributeError: 'dict' object has no attribute '__NUMPY_SETUP__'
        [end of output]

[piskvorky]

Thanks for reporting. This must be some new development, installations have been working for a decade…

Are you able to check what changed? And if it's fix-able, open a PR?? Many thanks.

[jaymegordo]

Yeah no idea, I have it failing on 3 colleague's M1 mbps also.

Seems other projects have used a workaround like: https://www.programcreek.com/python/example/67000/builtins.NUMPY_SETUP

def finalize_options(self):
    _build_ext.finalize_options(self)
    # Prevent numpy from thinking it is still in its setup process:
    try:
        __builtins__.__NUMPY_SETUP__ = False
    except:
        try:
            # For python 3
            import builtins
            builtins.__NUMPY_SETUP__ = False
        except:
            warn("Skipping numpy hack; if installation fails, try installing numpy first")

    import numpy

I could open a quick PR with this if u like?

[vigenere23]

Also related to https://github.com/RaRe-Technologies/gensim/issues/3225

[vigenere23]

I also have the same problem with poetry with the 3.8.3 version, with latest python 3.9

[vigenere23]

Upgrading to latest (4.2.0) did fix the problem. The migration guide was really helpfull : https://github.com/RaRe-Technologies/gensim/wiki/Migrating-from-Gensim-3.x-to-4

[Jebq]

Hi, I don't think this has to do with gensim version. I encounter the same issue with gensim 4.2.0.

After a bit of investigation, it appears to be the --use-pep517 flag in pip which causes the issue. The bug appears with poetry because it adds this arg to pip when installing a package (see source).

I could reproduce the behaviour (crash with AttributeError: 'dict' object has no attribute '__NUMPY_SETUP__') by running the following command: pip install gensim --use-pep517 --disable-pip-version-check --no-deps --no-cache-dir --no-binary gensim (similar to the command ran by poetry)

and the installation works without --use-pep517: pip install gensim --disable-pip-version-check --no-deps --no-cache-dir --no-binary gensim

[eduardosan]

Just adding a few more remarks over @Jebq comments:

• Pip installer complains about the usage of --no-use-pep517 for python 3.10:

$ poetry run python -m pip install gensim --disable-pip-version-check --no-deps --no-cache-dir --no-binary gensim

DEPRECATION: --no-binary currently disables reading from the cache of locally built wheels. In the future --no-binary will not influence the wheel cache. pip 23.1 will enforce this behaviour change. A possible replacement is to use the --no-cache-dir option. You can use the flag --use-feature=no-binary-enable-wheel-cache to test the upcoming behaviour. Discussion can be found at https://github.com/pypa/pip/issues/11453

• There is a discussion on poetry package (which is the reason I got this error in first place) about not complaining about this error as there is an approved PEP to change package installation method: https://github.com/python-poetry/poetry/issues/3433

It seems like the solution would be generating a pyproject.toml file for this package.

[eduardosan]

More information about new package metadata declaration can be found on PEP-621: https://peps.python.org/pep-0621/