pipenv icon indicating copy to clipboard operation
pipenv copied to clipboard

Published 20 hours ago • verified publisher icon pypa

Issue with NumPy, macOS 11 Big Sur, Python 3.9.1 Does pipenv not use the latest pip?

Open shanewilkins opened this issue 4 years ago • 27 comments

People are probably already aware of this issue, but adding my $0.02 as to what I think may be going on.

In a nutshell: NumPy fails to build on OS 11 and python 3.9.1. The error message is attached below, but NumPy is installed and works. (you can import numpy as np from a python interpreter) However, if you try to import numpy in a notebook, you get an error message saying that numpy is using a buggy "accelerate" backend.

The error about the backend occurs when numpy is being built from source incorrectly instead of using the provided wheel. That known issue was supposedly fixed in a pip upgrade (20.3). However, I get the same error below, even when I have updated pip to 20.3.

However, when i create a new pipenv environment, it defaults to pip 20.2. So my guess is that even if I subsequently update pip to 20.3, that pipenv install numpy is using the old version of pip.

Can anybody confirm and/or offer a workaround?

(blog)  shane@Olympus  ~/github/blog   dev ●  pipenv install numpy
Installing numpy...
Adding numpy to Pipfile's [packages]...
✔ Installation Succeeded
Pipfile.lock (16c839) out of date, updating to (49bf85)...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed!
ERROR:pip.subprocessor:Command errored out with exit status 1:
 command: /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/bin/python /usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/tmpk03mt0_f
     cwd: /private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy
Complete output (209 lines):
Running from numpy source directory.
setup.py:480: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
  run_build = parse_setuppy_commands()
Cythonizing sources
Processing numpy/random/_bounded_integers.pxd.in
Processing numpy/random/_philox.pyx
Processing numpy/random/_bounded_integers.pyx.in
Processing numpy/random/_sfc64.pyx
Processing numpy/random/_mt19937.pyx
Processing numpy/random/bit_generator.pyx
Processing numpy/random/mtrand.pyx
Processing numpy/random/_generator.pyx
Processing numpy/random/_pcg64.pyx
Processing numpy/random/_common.pyx
blas_opt_info:
blas_mkl_info:
customize UnixCCompiler
  libraries mkl_rt not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

blis_info:
  libraries blis not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

openblas_info:
  libraries openblas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_3_10_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries tatlas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_3_10_blas_info:
  libraries satlas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_blas_threads_info:
Setting PTATLAS=ATLAS
  libraries ptf77blas,ptcblas,atlas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_blas_info:
  libraries f77blas,cblas,atlas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

accelerate_info:
  libraries accelerate not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
Library accelerate was not found. Ignoring
  libraries veclib not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
Library veclib was not found. Ignoring
  FOUND:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

  FOUND:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

non-existing path in 'numpy/distutils': 'site.cfg'
lapack_opt_info:
lapack_mkl_info:
  libraries mkl_rt not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

openblas_lapack_info:
  libraries openblas not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

openblas_clapack_info:
  libraries openblas,lapack not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

flame_info:
  libraries flame not found in ['/Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib', '/usr/local/lib', '/usr/lib']
  NOT AVAILABLE

atlas_3_10_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries tatlas,tatlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries tatlas,tatlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/lib
  libraries tatlas,tatlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_3_10_threads_info'>
  NOT AVAILABLE

atlas_3_10_info:
  libraries lapack_atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries satlas,satlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries satlas,satlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/lib
  libraries satlas,satlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_3_10_info'>
  NOT AVAILABLE

atlas_threads_info:
Setting PTATLAS=ATLAS
  libraries lapack_atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries ptf77blas,ptcblas,atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/lib
  libraries ptf77blas,ptcblas,atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_threads_info'>
  NOT AVAILABLE

atlas_info:
  libraries lapack_atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries f77blas,cblas,atlas not found in /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/lib
  libraries lapack_atlas not found in /usr/local/lib
  libraries f77blas,cblas,atlas not found in /usr/local/lib
  libraries lapack_atlas not found in /usr/lib
  libraries f77blas,cblas,atlas not found in /usr/lib
<class 'numpy.distutils.system_info.atlas_info'>
  NOT AVAILABLE

  FOUND:
    extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers']
    extra_link_args = ['-Wl,-framework', '-Wl,Accelerate']
    define_macros = [('NO_ATLAS_INFO', 3), ('HAVE_CBLAS', None)]

/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py:274: UserWarning: Unknown distribution option: 'define_macros'
  warnings.warn(msg)
running dist_info
running build_src
build_src
building py_modules sources
creating build
creating build/src.macosx-11-x86_64-3.9
creating build/src.macosx-11-x86_64-3.9/numpy
creating build/src.macosx-11-x86_64-3.9/numpy/distutils
building library "npymath" sources
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
    main()
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 110, in prepare_metadata_for_build_wheel
    return hook(metadata_directory, config_settings)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 157, in prepare_metadata_for_build_wheel
    self.run_setup()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 248, in run_setup
    super(_BuildMetaLegacyBackend,
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 142, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 508, in <module>
    setup_package()
  File "setup.py", line 500, in setup_package
    setup(**metadata)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/core.py", line 169, in setup
    return old_setup(**new_attr)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 165, in setup
    return distutils.core.setup(**attrs)
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-build-env-5hhkc_mx/overlay/lib/python3.9/site-packages/setuptools/command/dist_info.py", line 31, in run
    egg_info.run()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/egg_info.py", line 24, in run
    self.run_command("build_src")
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/build_src.py", line 144, in run
    self.build_sources()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/build_src.py", line 155, in build_sources
    self.build_library_sources(*libname_info)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/build_src.py", line 288, in build_library_sources
    sources = self.generate_sources(sources, (lib_name, build_info))
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/build_src.py", line 378, in generate_sources
    source = func(extension, build_dir)
  File "numpy/core/setup.py", line 658, in get_mathlib_info
    st = config_cmd.try_link('int main(void) { return 0;}')
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/command/config.py", line 241, in try_link
    self._check_compiler()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/command/config.py", line 80, in _check_compiler
    self.fcompiler = new_fcompiler(compiler=self.fcompiler,
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 880, in new_fcompiler
    compiler = get_default_fcompiler(plat, requiref90=requiref90,
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 851, in get_default_fcompiler
    compiler_type =  _find_existing_fcompiler(matching_compiler_types,
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 802, in _find_existing_fcompiler
    c.customize(dist)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 526, in customize
    linker_so_flags = self.flag_vars.linker_so
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/environment.py", line 37, in __getattr__
    return self._get_var(name, conf_desc)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/environment.py", line 53, in _get_var
    var = self._hook_handler(name, hook)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/__init__.py", line 705, in _environment_hook
    return hook()
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/gnu.py", line 346, in get_flags_linker_so
    flags = GnuFCompiler.get_flags_linker_so(self)
  File "/private/var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/pip-resolver-id_2dgms/numpy/numpy/distutils/fcompiler/gnu.py", line 136, in get_flags_linker_so
    os.environ['MACOSX_DEPLOYMENT_TARGET'] = target
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 684, in __setitem__
    value = self.encodevalue(value)
  File "/usr/local/Cellar/[email protected]/3.9.1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/os.py", line 756, in encode
    raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not int
----------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 764, in <module>
    main()
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 758, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 741, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 702, in resolve_packages
    results, resolver = resolve(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/resolver.py", line 684, in resolve
    return resolve_deps(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/utils.py", line 1395, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/utils.py", line 823, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 229, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 350, in get_legacy_dependencies
    results, ireq = self.resolve_reqs(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 306, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 40, in prepare_distribution_metadata
    self.req.prepare_metadata()
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 564, in prepare_metadata
    self.metadata_directory = self._generate_metadata()
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 549, in _generate_metadata
    return generate_metadata(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/build/metadata.py", line 36, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 167, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/wrappers.py", line 245, in _call_hook
    self._subprocess_runner(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 271, in runner
    call_subprocess(
  File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Users/shane/.local/share/virtualenvs/blog-kKpxKzEP/bin/python /usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/yz/l6qh44196z70lty4rs6dm3nc0000gn/T/tmpk03mt0_f Check the logs for full command output.

shanewilkins avatar Dec 12 '20 19:12 shanewilkins

Seem related to the bug reported at NumPy here: #17784

shanewilkins avatar Dec 13 '20 18:12 shanewilkins

+1 exact same issue and error trace for me

@shanewilkins thank you for pointing me in the right direction. The issue was in fact that pipenv was using pip 2.2 instead of pip 2.3.

You can just manually update pipenv's pip with pipenv run python -m pip install -U pip and confirm it worked with pipenv run pip --version which should output pip 20.3.1 from ......

After this the numpy install works for me.

**Edit - Actually, the numpy install only works with pipenv install numpy --skip-lock, while pipenv lock still fails...

jaymegordo avatar Dec 13 '20 20:12 jaymegordo

+1 exact same issue and error trace for me

@shanewilkins thank you for pointing me in the right direction. The issue was in fact that pipenv was using pip 2.2 instead of pip 2.3.

You can just manually update pipenv's pip with pipenv run python -m pip install -U pip and confirm it worked with pipenv run pip --version which should output pip 20.3.1 from ......

After this the numpy install works for me.

**Edit - Actually, the numpy install only works with pipenv install numpy --skip-lock, while pipenv lock still fails...

@jaymegordo Upgrading pipenv's pip did not work.

MCFreddie777 avatar Dec 15 '20 17:12 MCFreddie777

I can also confirm that upgrading pipenv pip did not work for me.

mzagaja avatar Dec 30 '20 23:12 mzagaja

+1 exact same issue and error trace for me

@shanewilkins thank you for pointing me in the right direction. The issue was in fact that pipenv was using pip 2.2 instead of pip 2.3.

You can just manually update pipenv's pip with pipenv run python -m pip install -U pip and confirm it worked with pipenv run pip --version which should output pip 20.3.1 from ......

After this the numpy install works for me.

**Edit - Actually, the numpy install only works with pipenv install numpy --skip-lock, while pipenv lock still fails...

Any clue why pipenv lock fails?

jo3p avatar Jan 01 '21 15:01 jo3p

@jo3p I think it has something to do with how Big Sur changed how some system libraries are stored now. FWIW I'm still having the same issue, here's the start of my tb:

Locking [dev-packages] dependencies...
Building requirements...
Resolving dependencies...
✘ Locking Failed! 
ERROR:pip.subprocessor:Command errored out with exit status 1:
 command: /Users/Jayme/.local/share/virtualenvs/Jambot-uzRuMVcQ/bin/python /usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip install --ignore-installed --no-user --prefix /private/var/folders/h3/zjctchbd6jb6vsgm_nq8fwnw0000gn/T/pip-build-env-z40x6oqf/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.21,<3' 'numpy==1.16.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.5; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy; python_version>='"'"'3.9'"'"''
     cwd: None
Complete output (233 lines):
Ignoring numpy: markers 'python_version == "3.7" and platform_system != "AIX"' don't match your environment
Ignoring numpy: markers 'python_version == "3.8" and platform_system != "AIX"' don't match your environment
Ignoring numpy: markers 'python_version == "3.7" and platform_system == "AIX"' don't match your environment
Ignoring numpy: markers 'python_version == "3.8" and platform_system == "AIX"' don't match your environment
Collecting setuptools
  Using cached setuptools-51.1.1-py3-none-any.whl (2.0 MB)
Collecting wheel
  Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Collecting Cython<3,>=0.29.21
  Using cached Cython-0.29.21-py2.py3-none-any.whl (974 kB)
Collecting numpy
  Using cached numpy-1.19.4.zip (7.3 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /Users/Jayme/.local/share/virtualenvs/Jambot-uzRuMVcQ/bin/python /usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py prepare_metadata_for_build_wheel /var/folders/h3/zjctchbd6jb6vsgm_nq8fwnw0000gn/T/tmpto9f1e3i
         cwd: /private/var/folders/h3/zjctchbd6jb6vsgm_nq8fwnw0000gn/T/pip-install-qba16rxe/numpy
    Complete output (209 lines):
    Running from numpy source directory.
    setup.py:480: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
      run_build = parse_setuppy_commands()
    Cythonizing sources
    Processing numpy/random/_bounded_integers.pxd.in
    Processing numpy/random/_philox.pyx
    Processing numpy/random/_bounded_integers.pyx.in
    Processing numpy/random/_sfc64.pyx
    Processing numpy/random/_mt19937.pyx
    Processing numpy/random/bit_generator.pyx
    Processing numpy/random/mtrand.pyx
    Processing numpy/random/_generator.pyx
    Processing numpy/random/_pcg64.pyx
    Processing numpy/random/_common.pyx
    blas_opt_info:
    blas_mkl_info:
    customize UnixCCompiler
      libraries mkl_rt not found in ['/Users/Jayme/.local/share/virtualenvs/Jambot-uzRuMVcQ/lib', '/usr/local/lib', '/usr/lib']
      NOT AVAILABLE

    blis_info:
      libraries blis not found in ['/Users/Jayme/.local/share/virtualenvs/Jambot-uzRuMVcQ/lib', '/usr/local/lib', '/usr/lib']
      NOT AVAILABLE

Then a bunch more lib errors after that, exact same error trace as @shanewilkins's original post

jaymegordo avatar Jan 04 '21 15:01 jaymegordo

I think pipenv's vendored packaging within patched/notpip needs to be updated with the same Big Sur wheel support (accepting 10_9 wheels even when the OS reports 11_0) that went out in packaging 20.7 and pip 20.3: https://github.com/pypa/pip/pull/9170/files

That will at least allow use of the precompiled wheel for numpy, kicking the separate compiling-from-source issue down the road a bit.

jonblum avatar Jan 04 '21 17:01 jonblum

Hi, this helped me: export SYSTEM_VERSION_COMPAT=1

Source: https://stackoverflow.com/questions/63972113/big-sur-clang-invalid-version-error-due-to-macosx-deployment-target

radekerben avatar Jan 08 '21 08:01 radekerben

@radekerben thank you! That actually worked for me too, I can now successfully run pipenv lock without issue.

jaymegordo avatar Jan 08 '21 15:01 jaymegordo

This also worked for me as well. Thanks @radekerben 🥇

lashomb avatar Jan 08 '21 22:01 lashomb

You need packaging 20.9 for full Big Sur (Intel and Apple Silicon) support. 20.7+ for just Intel Big Sur. Or, of course, you can tell macOS to report a version number of 10.16 instead of 11 using SYSTEM_VERSION_COMPAT=1.

Any chance we can get a bump (master has 20.8) and version release?

henryiii avatar Mar 29 '21 19:03 henryiii

Hi, this helped me: export SYSTEM_VERSION_COMPAT=1

Source: https://stackoverflow.com/questions/63972113/big-sur-clang-invalid-version-error-due-to-macosx-deployment-target

Thanks for the investigation. It works for me and I just upgraded to the latest Big Sur 11.3

lqflqf avatar Apr 29 '21 12:04 lqflqf

This is because packaging expects 10.x, so SYSTEM_VERSION_COMPAT=1 causes macOS to call itself 10.16 instead of 11.0, which temporarily fixes older versions of packaging. This should be properly fixed and updated in pipenv, though. It should just be an update to 20.9 and then a patch release, I would expect.

henryiii avatar Apr 29 '21 12:04 henryiii

It is already done. I am seeking for people help testing the master branch before i can make a release

frostming avatar Apr 29 '21 12:04 frostming

How can we help test?

mzagaja avatar May 05 '21 02:05 mzagaja

Let me know how can I help test

lqflqf avatar May 05 '21 02:05 lqflqf

This issue seems fixed in the latest python release 3.9.5

lqflqf avatar May 07 '21 12:05 lqflqf

Just tested with python 3.9.5 and still having the problem (setting export SYSTEM_VERSION_COMPAT=1 fixes it).

zedrdave avatar May 11 '21 14:05 zedrdave

Does anyone have a good Universal2 wheel to test? I have a couple, like boost-histogram, but they require NumPy, and NumPy doesn't have a universal2 or arm64 for macOS wheel yet.

It looks like packaging is vendored at 20.8; 20.9 is required to support Apple Silicon property. But I need a good non-NumPy requiring package to test.

pipx run --spec git+https://github.com/pypa/pipenv.git pipenv install numpy -v

Starts building from source, which of course it should, as there are no supported wheels.

henryiii avatar May 21 '21 15:05 henryiii

@frostming could you please bump packaging to 20.9? Also, are you on the PyPA discord server?

Would it make sense to use the vendoring package from @pradyunsg like modern Pip does instead of using an old copy of the vendoring tooling from Pip? I tried on my M1 but couldn't install cffi, can try again on Intel later.

henryiii avatar May 21 '21 16:05 henryiii

If this is an issue on M1 + MacOS, that's possibly also because of https://github.com/numpy/numpy/issues/18143.

pradyunsg avatar May 21 '21 23:05 pradyunsg

pipenv was just released without bumping packaging to 20.9. The docs clearly said all dependencies would be bumped before releasing, so bump PRs were discouraged. 😭

I'm trying to help with Universal2 wheel rollout, and packaging 20.9 (and pip 21.0.1) are needed to properly handle some universal2 wheels.

henryiii avatar May 29 '21 04:05 henryiii

If this is an issue on M1 + MacOS, that's possibly also because of numpy/numpy#18143.

It's not exclusively. I have an older MacBook with an Intel chip and run into the same issue.

HWiese1980 avatar Jun 21 '21 14:06 HWiese1980

A lot of work has gone into making macOS Big Sur work, but it requires the latest dependencies. Pipenv was released without the "bump all dependencies" step described in its docs, so sadly, it still doesn't understand that "11.0" is a valid macOS version. If you set export SYSTEM_VERSION_COMPAT=1, then macOS will report itself as "10.16" instead, which temporarily fixes the problem (only on Intel machines, I believe).

henryiii avatar Jun 21 '21 14:06 henryiii

A lot of work has gone into making macOS Big Sur work, but it requires the latest dependencies. Pipenv was released without the "bump all dependencies" step described in its docs, so sadly, it still doesn't understand that "11.0" is a valid macOS version. If you set export SYSTEM_VERSION_COMPAT=1, then macOS will report itself as "10.16" instead, which temporarily fixes the problem (only on Intel machines, I believe).

works on apple silicon with terminal started under rosetta simulation too

igor-cheb avatar Aug 22 '21 12:08 igor-cheb

A lot of work has gone into making macOS Big Sur work, but it requires the latest dependencies. Pipenv was released without the "bump all dependencies" step described in its docs, so sadly, it still doesn't understand that "11.0" is a valid macOS version. If you set export SYSTEM_VERSION_COMPAT=1, then macOS will report itself as "10.16" instead, which temporarily fixes the problem (only on Intel machines, I believe).

It works. However, it's not exactly the ideal solution in my humble opinion.

HWiese1980 avatar Aug 23 '21 07:08 HWiese1980

I have faced the same problem but, it seems works when I am using latest version of numpy. In other words, whenever tried to install older version like 1.15.2 it gives me installation failure. So technically it should work when you are using the latest one.

ghost avatar Jan 28 '22 10:01 ghost

This should be resolvable on pipenv==2022.8.19 because we rely exclusively on the vendor'd pip which is the latest version 22.2.2 at the time of writing.

matteius avatar Aug 21 '22 11:08 matteius

In my case the issue was that I hadn't agreed to the xcode licensing agreement.

I fixed the issue by typing sudo xcodebuild -license and agreeing to the license.

I was then prompted to update my xcode CLI tools, which was also necessary since further runs of pipenv install produced errors like xcode-select: Failed to locate 'clang', requesting installation of command line developer tools.

I tried repeatedly to run the software installation, but the same error kept popping up until I found this Stack Overflow thread which suggested running the following command:

sudo xcode-select -switch /Library/Developer/CommandLineTools

After that it finally worked.

chasedenecke avatar Sep 25 '22 03:09 chasedenecke