pip install hnswlib fail on python 3.9

[taqihaider7]

`Building wheels for collected packages: hnswlib Building wheel for hnswlib (pyproject.toml) ... error error: subprocess-exited-with-error

× Building wheel for hnswlib (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> [5 lines of output] running bdist_wheel running build running build_ext building 'hnswlib' extension error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/ [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for hnswlib Failed to build hnswlib ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects`

I have installed the latest Microsoft Visual C++ redistributions despite of that it is not installing

[yurymalkov]

It seems like something is wrong with the environment. E.g. not running from the command line with MS VC paths set.

[taqihaider7]

any solution to fix that problem

[yurymalkov]

Can you check your path variables? Does cl.exe work?

[dcydrob]

I don't know if this is at all helpful to you all, but I (on a Mac) also got failures running pip install hnswlib and python3.9 -m pip install hnswlib but it worked when I used python 3.7: python3.7 -m pip install hnswlib (I don't happen to have 3.8 on my machine, so I didn't try that...)

[anentropic]

I have this problem with Python 3.10.3 on macOS 13.2.1 (M1)

...
        /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__memory/unique_ptr.h:610:45: error: unknown type name 'nullptr_t'
        operator!=(const unique_ptr<_T1, _D1>& __x, nullptr_t) _NOEXCEPT
                                                    ^
        fatal error: too many errors emitted, stopping now [-ferror-limit=]
        20 errors generated.
        error: command '/usr/bin/clang' failed with exit code 1
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed building wheel for hnswlib
  Failed to build hnswlib
  ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

I'm not using hnswlib directly, but it's needed as a dependency of another package (https://github.com/chroma-core/chroma/issues/250)

Are there some instructions for how to install the python lib successfully?

[anentropic]

more details around the error:

building 'hnswlib' extension
creating build
creating build/temp.macosx-12.3-arm64-cpython-310
creating build/temp.macosx-12.3-arm64-cpython-310/python_bindings
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-tyw8alra/overlay/lib/python3.10/site-packages/pybind11/include -I/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-tyw8alra/overlay/lib/python3.10/site-packages/numpy/core/include -I./hnswlib/ -I/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/include -I/Users/anentropic/.pyenv/versions/3.10.3/include/python3.10 -c ./python_bindings/bindings.cpp -o build/temp.macosx-12.3-arm64-cpython-310/./python_bindings/bindings.o -O3 -stdlib=libc++ -mmacosx-version-min=10.7 -DVERSION_INFO=\"0.7.0\" -std=c++14 -fvisibility=hidden
In file included from ./python_bindings/bindings.cpp:1:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iostream:41:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/ios:221:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__locale:18:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/memory:841:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/copy.h:12:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__algorithm/unwrap_iter.h:13:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__iterator/iterator_traits.h:14:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__iterator/incrementable_traits.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/concepts:133:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__concepts/arithmetic.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/type_traits:421:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__functional/invoke.h:15:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__type_traits/apply_cv.h:16:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/__type_traits/remove_reference.h:13:
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/cstddef:50:9: error: no member named 'nullptr_t' in the global namespace
using ::nullptr_t;
      ~~^

I know basically nothing about C++

but googling I have learned that nullptr_t was added to the standard in C++11

and we are calling clang with -stdlib=libc++ -std=c++14, which seems like it should work

[anentropic]

no idea if this is relevant, but I found a similar looking issue here: https://github.com/chipsalliance/verible/issues/260

and they fixed it by adding #include <cstddef> at the top of one of their header files and prefixing as std::nullptr_t https://github.com/chipsalliance/verible/commit/241da929600ea0e458580df6f8bfe95b288cff08

[bdunsmuir]

I have the same problem using python 3.9

Failed to build hnswlib ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

[snayan06]

same is happening as @anentropic mentioned I was installing chromadb and got a failure with python 3.9, any solutions people have found on this? which can help

[dcydrob]

Hey all. I looked in the code and saw there’s a check for an environment variable. Try export HNSWLIB_NO_NATIVE=1 and try install again.

[anentropic]

UsingHNSWLIB_NO_NATIVE=1 I get a different error:

  Command ['/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10', '--no-deps', '/Users/anentropic/Library/Caches/pypoetry/artifacts/d2/32/ed/8674b10968a423c6f1b64821759e55a782ff3b28da34e2a886e85d54a9/hnswlib-0.7.0.tar.gz'] errored with the following return code 1, and output:
  Processing /Users/anentropic/Library/Caches/pypoetry/artifacts/d2/32/ed/8674b10968a423c6f1b64821759e55a782ff3b28da34e2a886e85d54a9/hnswlib-0.7.0.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    error: subprocess-exited-with-error

    × Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [16 lines of output]
        Traceback (most recent call last):
          File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
            main()
          File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/Users/anentropic/Library/Caches/pypoetry/virtualenvs/experiments-langchain-FPovFLPB-py3.10/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 118, in get_requires_for_build_wheel
            return hook(config_settings)
          File "/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-29nj4h4p/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
            return self._get_build_requires(config_settings, requirements=['wheel'])
          File "/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-29nj4h4p/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
            self.run_setup()
          File "/private/var/folders/w1/_vgkxyln4c7bk8kr29s1y1k00000gn/T/pip-build-env-29nj4h4p/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
            exec(code, locals())
          File "<string>", line 74, in <module>
          File "<string>", line 91, in BuildExt
        ValueError: list.remove(x): x not in list
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

[snayan06]

@dcydrob i did that , and added it in ~/.bashrc file still the same error . did it worked for u . am i missing something .

this is how my bashrc looks like . i also tried directly in terminal as well same results

# .bashrc

# User specific aliases and functions

alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'

# Source global definitions
if [ -f /etc/bashrc ]; then
	. /etc/bashrc
fi
export HNSWLIB_NO_NATIVE=1

Building wheels for collected packages: hnswlib
  Building wheel for hnswlib (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for hnswlib (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [7 lines of output]
      running bdist_wheel
      running build
      running build_ext
      creating tmp
      gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/opt/engati/faq-indexer/venv/include -I/usr/local/include/python3.9 -c /tmp/tmpfi1kk7tb.cpp -o tmp/tmpfi1kk7tb.o -std=c++14
      gcc: error: unrecognized command line option '-std=c++14'
      error: command '/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for hnswlib
Failed to build hnswlib

[jeffleft]

Might be that gcc can't find cc1plus. I had the same problem on Amazon linux and this stackoverflow post fixed it.

[josepmy]

Hi, Same problem here, python3.9 on macosx 13.3.1 export HNSWLIB_NO_NATIVE=1 from terminal before install works for me. Thanks 👍

[tourbut]

HI. The same issue occurred in Windows 10, Python 3.9, and VSCODE. I resolved the issue by installing Microsoft C++ Build Tools (selecting the C++ build package during installation).

[OmarSultan85]

HI. The same issue occurred in Windows 10, Python 3.9, and VSCODE. I resolved the issue by installing Microsoft C++ Build Tools (selecting the C++ build package during installation).

Hey @tourbut , can you please advise which option from the vs installer exactly did u choose , thanks

[shivamMg]

For windows:

1. Download Build Tools: https://visualstudio.microsoft.com/visual-cpp-build-tools/
2. Open, and then in Workloads select "Desktop development with C++" to install:

[bgarza]

I have the same problem using python 3.9

Failed to build hnswlib ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

I have the same error on MacOS 13.1 and python 3.10. Setting the environment variable HNSWLIB_NO_NATIVE did not work for me. I do not have this problem on Windows 11.

[bgarza]

I have the same problem using python 3.9 Failed to build hnswlib ERROR: Could not build wheels for hnswlib, which is required to install pyproject.toml-based projects

I have the same error on MacOS 13.1 and python 3.10. Setting the environment variable HNSWLIB_NO_NATIVE did not work for me. I do not have this problem on Windows 11.

I was able to resolve this by upgrading XCode. I then built hnswlib by running setup.py, which I was not able to do prior. Upgrading XCode may have upgraded or installed Clang, which is now at 14.0.3 for me.

[yurymalkov]

@bgarza what was the initial environment? I do not know a way to reproduce this error on my mac.

[anentropic]

I have clang 14.0.3 and all the problems above:

$ clang --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.4.0

I have not tried cloning hnswlib repo and building from source though

[bgarza]

@bgarza what was the initial environment? I do not know a way to reproduce this error on my mac.

I believe that XCode was out of date, but not sure at what version. Not sure why, but the XCode upgrade took a very long time via the App Store. After the upgrade I was able to build the project.

[yurymalkov]

@anentropic I see. Do you use default system python or anaconda? pip install hnswlibshould be pretty much the same thing as install from source (pip install git+https://github.com/nmslib/hnswlib.git)

[anentropic]

@anentropic I see. Do you use default system python or anaconda? pip install hnswlibshould be pretty much the same thing as install from source (pip install git+https://github.com/nmslib/hnswlib.git)

I'm not using anaconda

I am using pyenv python with a poetry virtualenv, I think my pyenv is installed via homebrew

[armalko]

Was getting the same error while installing on Linux inside Docker Container:

In file included from ./python_bindings/bindings.cpp:6:
      ./hnswlib/hnswlib.h: At global scope:
      ./hnswlib/hnswlib.h:80:13: warning: ‘bool AVX512Capable()’ defined but not used [-Wunused-function]
         80 | static bool AVX512Capable() {
            |             ^~~~~~~~~~~~~
      gcc: fatal error: Killed signal terminated program cc1plus
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

It revealed that problem was in low RAM (1GB). As I created swap-file for another 1GB everything worked just fine.

I used this commands to create SWAP-memory, if anyone needs:

sudo fallocate -l 1G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

And deleted it afterwards with

sudo swapoff /swapfile
sudo rm /swapfile

[konykwong]

how to set the environment var on Windows? export HNSWLIB_NO_NATIVE=1

what should i type in the terminal?

[ikebo]

export HNSWLIB_NO_NATIVE=1 export ARCHFLAGS="-arch x86_64" and try again!!

[mariusorani]

Was getting the same error on mac using python 3.11 in a venv. Switching back to python 3.10 solved the problem

[sanskar-mk2]

how to set the environment var on Windows? export HNSWLIB_NO_NATIVE=1

what should i type in the terminal?

set HNSWLIB_NO_NATIVE=1 is the equivalent on windows terminal.

[dyashuni]

MACOS installation is fixed in the develop branch Could you try it? pip install git+https://github.com/nmslib/hnswlib.git@develop