Pip installation failed on macOS Big Sur due to clang error

[sutao]

Clang failed with errors complaining the compiler does not support -march=native

clang: error: the clang compiler does not support '-march=native'

Not sure if this is a Big Sur only issue due to changes in clang. But when I tried clang --help on Big Sur, the march argument is not there.

I've forked the repo here: https://github.com/sutao/annoy/commit/6d33b98b501433cfec02d7d45772b6b3ecf215a1

Full error:

  Complete output (14 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.14.6-x86_64-3.8
  creating build/lib.macosx-10.14.6-x86_64-3.8/annoy
  copying annoy/__init__.py -> build/lib.macosx-10.14.6-x86_64-3.8/annoy
  running build_ext
  building 'annoy.annoylib' extension
  creating build/temp.macosx-10.14.6-x86_64-3.8
  creating build/temp.macosx-10.14.6-x86_64-3.8/src
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -I/Users/tao/workspace/repo/facedepot/env/include -I/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c src/annoymodule.cc -o build/temp.macosx-10.14.6-x86_64-3.8/src/annoymodule.o -D_CRT_SECURE_NO_WARNINGS -march=native -O3 -ffast-math -fno-associative-math -std=c++11 -mmacosx-version-min=10.9
  clang: error: the clang compiler does not support '-march=native'
  error: command 'clang' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for annoy

[erikbern]

weird, not sure what's going on. are you able to get it to compile if you remove that flag?

[sutao]

weird, not sure what's going on. are you able to get it to compile if you remove that flag?

Yes. I was able to get it compiled without issues after removing that flag. Haven't seen this issue on Catalina. I did a fresh install of Big Sur last week and this started to happen.

[erikbern]

ok, strange

[fgvenegas]

weird, not sure what's going on. are you able to get it to compile if you remove that flag?

Yes. I was able to get it compiled without issues after removing that flag. Haven't seen this issue on Catalina. I did a fresh install of Big Sur last week and this started to happen.

Hi, how did you remove the flag? I'm having the same issue on Catalina

[erikbern]

There's been a few similar issues reported so maybe we need to reduce the use of -march=native to certain platforms

[sutao]

weird, not sure what's going on. are you able to get it to compile if you remove that flag?

Yes. I was able to get it compiled without issues after removing that flag. Haven't seen this issue on Catalina. I did a fresh install of Big Sur last week and this started to happen.

Hi, how did you remove the flag? I'm having the same issue on Catalina

See my change in my forked version here: https://github.com/sutao/annoy/commit/6d33b98b501433cfec02d7d45772b6b3ecf215a1

[sutao]

There's been a few similar issues reported so maybe we need to reduce the use of -march=native to certain platforms

Actually, I was able to narrow down the issue. This seems to happen only with the Python shipped with macOS (only tested Big Sur in my case). The official annoy wheel installs successfully with the Python downloaded from python.org. Haven't had time to try any Python distrubutions from Homebrew.

The version of Python I tested with is:

Python 3.8.2 (default, Oct  2 2020, 10:45:42) 
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin

Here is the full log when creating a virtualenv and install annoy using macOS's stock Python:

➜  test-annoy python3 -m virtualenv -p /Library/Developer/CommandLineTools/usr/bin/python3 menv
created virtual environment CPython3.8.2.final.0-64 in 355ms
  creator CPython3macOsFramework(dest=/Users/tao/workspace/temp/test-annoy/menv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/tao/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.4, setuptools==50.3.2, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
➜  test-annoy source menv/bin/activate
(menv) ➜  test-annoy python3                                                                          
Python 3.8.2 (default, Oct  2 2020, 10:45:42) 
[Clang 12.0.0 (clang-1200.0.32.27)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
(menv) ➜  test-annoy pip install --no-cache-dir annoy
Collecting annoy
  Downloading annoy-1.17.0.tar.gz (646 kB)
     |████████████████████████████████| 646 kB 872 kB/s 
Building wheels for collected packages: annoy
  Building wheel for annoy (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/tao/workspace/temp/test-annoy/menv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/90/g86_y2wj5fg3bbhg64zlw00h0000gn/T/pip-install-wao37wxj/annoy/setup.py'"'"'; __file__='"'"'/private/var/folders/90/g86_y2wj5fg3bbhg64zlw00h0000gn/T/pip-install-wao37wxj/annoy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/90/g86_y2wj5fg3bbhg64zlw00h0000gn/T/pip-wheel-h6oieong
       cwd: /private/var/folders/90/g86_y2wj5fg3bbhg64zlw00h0000gn/T/pip-install-wao37wxj/annoy/
  Complete output (14 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.14.6-x86_64-3.8
  creating build/lib.macosx-10.14.6-x86_64-3.8/annoy
  copying annoy/__init__.py -> build/lib.macosx-10.14.6-x86_64-3.8/annoy
  running build_ext
  building 'annoy.annoylib' extension
  creating build/temp.macosx-10.14.6-x86_64-3.8
  creating build/temp.macosx-10.14.6-x86_64-3.8/src
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -I/Users/tao/workspace/temp/test-annoy/menv/include -I/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c src/annoymodule.cc -o build/temp.macosx-10.14.6-x86_64-3.8/src/annoymodule.o -D_CRT_SECURE_NO_WARNINGS -march=native -O3 -ffast-math -fno-associative-math -DANNOYLIB_MULTITHREADED_BUILD -std=c++14 -mmacosx-version-min=10.12
  clang: error: the clang compiler does not support '-march=native'
  error: command 'clang' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for annoy
  Running setup.py clean for annoy
Failed to build annoy
Installing collected packages: annoy
    Running setup.py install for annoy ... error
    ERROR: Command errored out with exit status 1:
     command: /Users/tao/workspace/temp/test-annoy/menv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/90/g86_y2wj5fg3bbhg64zlw00h0000gn/T/pip-install-wao37wxj/annoy/setup.py'"'"'; __file__='"'"'/private/var/folders/90/g86_y2wj5fg3bbhg64zlw00h0000gn/T/pip-install-wao37wxj/annoy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/90/g86_y2wj5fg3bbhg64zlw00h0000gn/T/pip-record-7051j5h2/install-record.txt --single-version-externally-managed --compile --install-headers /Users/tao/workspace/temp/test-annoy/menv/include/site/python3.8/annoy
         cwd: /private/var/folders/90/g86_y2wj5fg3bbhg64zlw00h0000gn/T/pip-install-wao37wxj/annoy/
    Complete output (14 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.14.6-x86_64-3.8
    creating build/lib.macosx-10.14.6-x86_64-3.8/annoy
    copying annoy/__init__.py -> build/lib.macosx-10.14.6-x86_64-3.8/annoy
    running build_ext
    building 'annoy.annoylib' extension
    creating build/temp.macosx-10.14.6-x86_64-3.8
    creating build/temp.macosx-10.14.6-x86_64-3.8/src
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -I/Users/tao/workspace/temp/test-annoy/menv/include -I/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c src/annoymodule.cc -o build/temp.macosx-10.14.6-x86_64-3.8/src/annoymodule.o -D_CRT_SECURE_NO_WARNINGS -march=native -O3 -ffast-math -fno-associative-math -DANNOYLIB_MULTITHREADED_BUILD -std=c++14 -mmacosx-version-min=10.12
    clang: error: the clang compiler does not support '-march=native'
    error: command 'clang' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/tao/workspace/temp/test-annoy/menv/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/90/g86_y2wj5fg3bbhg64zlw00h0000gn/T/pip-install-wao37wxj/annoy/setup.py'"'"'; __file__='"'"'/private/var/folders/90/g86_y2wj5fg3bbhg64zlw00h0000gn/T/pip-install-wao37wxj/annoy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/90/g86_y2wj5fg3bbhg64zlw00h0000gn/T/pip-record-7051j5h2/install-record.txt --single-version-externally-managed --compile --install-headers /Users/tao/workspace/temp/test-annoy/menv/include/site/python3.8/annoy Check the logs for full command output.
WARNING: You are using pip version 20.2.4; however, version 20.3.1 is available.
You should consider upgrading via the '/Users/tao/workspace/temp/test-annoy/menv/bin/python -m pip install --upgrade pip' command.

[erikbern]

Thanks. Maybe we should disable that flag. Not sure if it makes a huge difference in terms of performance.

[soarcreator]

@sutao Is there no way to "pip3 install annoy" with removing the flag of -march=native? I'm having the same issue on Catalina.

I'm unfamiliar with installing python package directly from the GitHub repository here. So, I don't know how to apply sutao@6d33b98 to solve this issue. Thank you

[sutao]

@sutao Is there no way to "pip3 install annoy" with removing the flag of -march=native? I'm having the same issue on Catalina.

I'm unfamiliar with installing python package directly from the GitHub repository here. So, I don't know how to apply sutao/annoy@6d33b98 to solve this issue. Thank you

You can basically: pip3 install git+https://github.com/sutao/annoy.git@master

[soarcreator]

I tried the command you put above, and then I got a different error. This may be supposed to be a new issue, but let me paste the new error I got.

In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/_types.h:27: /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:55:9: error: unknown type name '__int64_t' typedef __int64_t __darwin_blkcnt_t; /* total blocks */ ^ /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:56:9: error: unknown type name '__int32_t'; did you mean '__int128_t'? typedef __int32_t __darwin_blksize_t; /* preferred block size */ ^ note: '__int128_t' declared here /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:57:9: error: unknown type name '__int32_t'; did you mean '__int128_t'? typedef __int32_t __darwin_dev_t; /* dev_t */ ^ note: '__int128_t' declared here /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:60:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'? typedef __uint32_t __darwin_gid_t; /* [???] process and group IDs */ ^ note: '__uint128_t' declared here /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:61:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'? typedef __uint32_t __darwin_id_t; /* [XSI] pid_t, uid_t, or gid_t*/ ^ note: '__uint128_t' declared here /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:62:9: error: unknown type name '__uint64_t' typedef __uint64_t __darwin_ino64_t; /* [???] Used for 64 bit inodes */ ^ /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:68:9: error: unknown type name '__darwin_natural_t' typedef __darwin_natural_t __darwin_mach_port_name_t; /* Used by mach */ ^ /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:70:9: error: unknown type name '__uint16_t'; did you mean '__uint128_t'? typedef __uint16_t __darwin_mode_t; /* [???] Some file attributes */ ^ note: '__uint128_t' declared here /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:71:9: error: unknown type name '__int64_t' typedef __int64_t __darwin_off_t; /* [???] Used for file sizes */ ^ /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:72:9: error: unknown type name '__int32_t'; did you mean '__int128_t'? typedef __int32_t __darwin_pid_t; /* [???] process and group IDs */ ^ note: '__int128_t' declared here /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:73:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'? typedef __uint32_t __darwin_sigset_t; /* [???] signal set */ ^ note: '__uint128_t' declared here /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:74:9: error: unknown type name '__int32_t'; did you mean '__int128_t'? typedef __int32_t __darwin_suseconds_t; /* [???] microseconds */ ^ note: '__int128_t' declared here /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:75:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'? typedef __uint32_t __darwin_uid_t; /* [???] user IDs */ ^ note: '__uint128_t' declared here /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/sys/_types.h:76:9: error: unknown type name '__uint32_t'; did you mean '__uint128_t'? typedef __uint32_t __darwin_useconds_t; /* [???] microseconds */ ^ note: '__uint128_t' declared here

It seems a problem related to bit number or something like that. I omited some lines above because they are all similar.

This problem was solved by the following command. brew link --overwrite [email protected]

[afgarcia86]

Bump, I am having this issue with [email protected] installed with brew on Big Sur

[wmg-expired]

Thank you @sutao 🙏 saved my day!!

[scalleri]

for me it worked when I reinstalled Anaconda and made a new environment - not ideal but at least it works