pip install failure on homebrew Python 3.11 with clang error

[alick]

Problem description

Tried to install gensim on macOS with Python 3.11 from homebrew, but it failed due to compilation error.

pip3.11 install --upgrade gensim runs into

...
      copying gensim/models/word2vec_inner.pxd -> build/lib.macosx-12-arm64-cpython-311/gensim/models
      copying gensim/models/word2vec_inner.pyx -> build/lib.macosx-12-arm64-cpython-311/gensim/models
      copying gensim/corpora/_mmreader.c -> build/lib.macosx-12-arm64-cpython-311/gensim/corpora
      copying gensim/corpora/_mmreader.pyx -> build/lib.macosx-12-arm64-cpython-311/gensim/corpora
      running build_ext
      building 'gensim.models.word2vec_inner' extension
      creating build/temp.macosx-12-arm64-cpython-311
      creating build/temp.macosx-12-arm64-cpython-311/gensim
      creating build/temp.macosx-12-arm64-cpython-311/gensim/models
      clang -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/taoz/brew/opt/[email protected]/Frameworks/Python.framework/Versions/3.11/include/python3.11 -I/Users/taoz/brew/lib/python3.11/site-packages/numpy/core/include -c gensim/models/word2vec_inner.c -o build/temp.macosx-12-arm64-cpython-311/gensim/models/word2vec_inner.o
      gensim/models/word2vec_inner.c:217:12: fatal error: 'longintrepr.h' file not found
        #include "longintrepr.h"
                 ^~~~~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

Same issue reported at https://stackoverflow.com/q/74979674/1166587 by others.

Python 3.10 and pip3.10 install gensim: work.

Steps/code/corpus to reproduce

full tracebacks: https://gist.github.com/alick/d7a06116736cf36d4838572a1b8a1873

Versions

Please provide the output of:

In [3]: import platform; print(platform.platform())
   ...: import sys; print("Python", sys.version)
   ...: import struct; print("Bits", 8 * struct.calcsize("P"))
   ...: import numpy; print("NumPy", numpy.__version__)
   ...: import scipy; print("SciPy", scipy.__version__)
macOS-12.6.1-arm64-arm-64bit
Python 3.11.1 (main, Jan  7 2023, 20:18:46) [Clang 14.0.0 (clang-1400.0.29.202)]
Bits 64
NumPy 1.24.1
SciPy 1.10.0

macOS 12.6.1 Homebrew 3.6.17 pip 22.3.1

[gojomo]

I see other projects have hit similar with Python 3.11. It might be an issue requiring a cython upgrade before our .c files are compiled (if I'm reading https://github.com/aio-libs/aiohttp/issues/6600 right) – perhaps to cython-0.29.25 or higher? Or, perhaps it's fixable with some extra CFLAGS under Python 3.11, if comment at https://github.com/termux/termux-packages/issues/13327#issuecomment-1321155920 applies – though conversation there also implies that maybe this header file shouldn't be referenced at all by external code?

[jc-louis]

Hello, any updates on this issue? Thanks!

[nikaro]

It should be fixed by https://github.com/RaRe-Technologies/gensim/pull/3430. Waiting for a review/merge.