scancode-toolkit icon indicating copy to clipboard operation
scancode-toolkit copied to clipboard

Published 20 hours ago • verified publisher icon nexB

Update the minimum system requirements

Open pombredanne opened this issue 3 years ago • 11 comments

https://scancode-toolkit.readthedocs.io/en/latest/getting-started/install.html?highlight=RAM#system-requirements documents the minimum requirements ...

Hardware : ScanCode will run best with a modern X86 processor and at least 4GB of RAM and 500MB of disk. These are bare minimum requirements. Supported operating systems: ScanCode should run on these 64-bit OSes running X86_64 processors: Linux: on recent 64-bit Linux distributions, Mac: on recent 64-bit macOS (10.14 and up), Windows: on Windows 10 and up, FreeBSD,

4GB and 500MB is on the low end and we want a 64 bits processor. We may run OK on Apple ARM M1 too though this is not tested and likely requires the x86 emulation mode.

To do this correctly, the following should be done:

  • [ ] Run tests on multiple machines using the release archive. This can be done using a GH action or azure free. Collect the installed size on disk and get some memory info when running a simple scan.

pombredanne avatar Feb 02 '22 07:02 pombredanne

Testing on M1 here. Wasn't able to build from a git clone. Let me know if I should create a separate issue.

🔌  ⚡️  in src/ 
› git clone https://github.com/nexb/scancode-toolkit
Cloning into 'scancode-toolkit'...
remote: Enumerating objects: 264367, done.
remote: Counting objects: 100% (42671/42671), done.
remote: Compressing objects: 100% (17778/17778), done.
remote: Total 264367 (delta 19952), reused 41722 (delta 19431), pack-reused 221696
Receiving objects: 100% (264367/264367), 537.58 MiB | 20.36 MiB/s, done.
Resolving deltas: 100% (161366/161366), done.
Updating files: 100% (84619/84619), done.

🔌  ⚡️  in src/ 
› cd scancode-toolkit 

🔌  ⚡️  in scancode-toolkit/ on develop
› ./configure && source venv/bin/activate
created virtual environment CPython3.9.9.final.0-64 in 1293ms
  creator CPython3Posix(dest=/Users/jon/src/scancode-toolkit/venv, clear=False, no_vcs_ignore=True, global=False)
  seeder PipInvoke(download=False, pip=embed, setuptools=embed, wheel=embed)
    added seed packages: pip==21.3.1, setuptools==60.2.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
Looking in links: https://thirdparty.aboutcode.org/pypi
Obtaining file:///Users/jon/src/scancode-toolkit
  Checking if build backend supports build_editable ... done
  Preparing metadata (pyproject.toml) ... done
Collecting chardet>=3.0.0
  Downloading https://thirdparty.aboutcode.org/pypi/chardet-4.0.0-py2.py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 2.6 MB/s            
Collecting jsonstreams>=0.5.0
  Downloading https://thirdparty.aboutcode.org/pypi/jsonstreams-0.6.0-py2.py3-none-any.whl (10 kB)
Collecting ftfy>=4.0.0
  Downloading https://thirdparty.aboutcode.org/pypi/ftfy-6.0.3-py3-none-any.whl (41 kB)
     |████████████████████████████████| 41 kB 7.5 MB/s              
Collecting pygments
  Downloading https://thirdparty.aboutcode.org/pypi/Pygments-2.9.0-py3-none-any.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 5.1 MB/s            
Collecting publicsuffix2
  Downloading https://thirdparty.aboutcode.org/pypi/publicsuffix2-2.20191221-py2.py3-none-any.whl (89 kB)
     |████████████████████████████████| 89 kB 21.8 MB/s            
Collecting pymaven-patch>=0.2.8
  Downloading https://thirdparty.aboutcode.org/pypi/pymaven_patch-0.3.0-py2.py3-none-any.whl (23 kB)
Collecting jaraco.functools
  Downloading https://thirdparty.aboutcode.org/pypi/jaraco.functools-3.3.0-py3-none-any.whl (6.8 kB)
Collecting Beautifulsoup4>=4.0.0
  Downloading https://thirdparty.aboutcode.org/pypi/beautifulsoup4-4.9.3-py3-none-any.whl (115 kB)
     |████████████████████████████████| 115 kB 5.3 MB/s             
Collecting bitarray>=0.8.1
  Downloading https://thirdparty.aboutcode.org/pypi/bitarray-2.1.0.tar.gz (71 kB)
     |████████████████████████████████| 71 kB 42.7 MB/s            
  Preparing metadata (setup.py) ... done
Collecting parameter-expansion-patched>=0.2.1b4
  Downloading https://thirdparty.aboutcode.org/pypi/parameter_expansion_patched-0.2.1b4-py3-none-any.whl (12 kB)
Collecting html5lib
  Downloading https://thirdparty.aboutcode.org/pypi/html5lib-1.1-py2.py3-none-any.whl (112 kB)
     |████████████████████████████████| 112 kB 5.3 MB/s            
Collecting boolean.py>=3.5
  Downloading https://thirdparty.aboutcode.org/pypi/boolean.py-3.8-py2.py3-none-any.whl (23 kB)
Collecting license-expression>=21.6.14
  Downloading https://thirdparty.aboutcode.org/pypi/license_expression-21.6.14-py3-none-any.whl (86 kB)
     |████████████████████████████████| 86 kB 22.9 MB/s             
Collecting debian-inspector>=30.0.0
  Downloading https://thirdparty.aboutcode.org/pypi/debian_inspector-30.0.0-py3-none-any.whl (43 kB)
     |████████████████████████████████| 43 kB 9.1 MB/s             
Collecting commoncode>=30.0.0
  Downloading https://thirdparty.aboutcode.org/pypi/commoncode-30.0.0-py3-none-any.whl (82 kB)
     |████████████████████████████████| 82 kB 2.2 MB/s             
Collecting fingerprints>=0.6.0
  Downloading https://thirdparty.aboutcode.org/pypi/fingerprints-1.0.3-py2.py3-none-any.whl (13 kB)
Collecting click!=7.0,>=6.7
  Downloading https://thirdparty.aboutcode.org/pypi/click-8.0.1-py3-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 24.0 MB/s            
Collecting pkginfo2>=30.0.0
  Downloading https://thirdparty.aboutcode.org/pypi/pkginfo2-30.0.0-py3-none-any.whl (25 kB)
Collecting pip-requirements-parser>=31.2.0
  Downloading https://thirdparty.aboutcode.org/pypi/pip_requirements_parser-31.2.0-py3-none-any.whl (33 kB)
Collecting lxml>=4.6.5
  Downloading https://thirdparty.aboutcode.org/pypi/lxml-4.7.1.tar.gz (3.2 MB)
     |████████████████████████████████| 3.2 MB 5.2 MB/s            
  Preparing metadata (setup.py) ... done
Collecting pdfminer.six>=20200101
  Downloading https://thirdparty.aboutcode.org/pypi/pdfminer.six-20201018-py3-none-any.whl (5.6 MB)
     |████████████████████████████████| 5.6 MB 33.8 MB/s            
Collecting requests>=2.7.0
  Downloading https://thirdparty.aboutcode.org/pypi/requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 36.5 MB/s            
Collecting pluggy>=0.12.0
  Downloading https://thirdparty.aboutcode.org/pypi/pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting pygmars>=0.7.0
  Downloading https://thirdparty.aboutcode.org/pypi/pygmars-0.7.0-py3-none-any.whl (28 kB)
Collecting dparse2>=0.6.0
  Downloading https://thirdparty.aboutcode.org/pypi/dparse2-0.6.0-py3-none-any.whl (10 kB)
Collecting plugincode>=21.1.21
  Downloading https://thirdparty.aboutcode.org/pypi/plugincode-21.1.21-py3-none-any.whl (18 kB)
Collecting pefile>=2020.1.1
  Downloading https://thirdparty.aboutcode.org/pypi/pefile-2021.5.24-py3-none-any.whl (62 kB)
     |████████████████████████████████| 62 kB 8.2 MB/s             
Collecting intbitset>=2.3.0
  Downloading https://thirdparty.aboutcode.org/pypi/intbitset-2.4.1.tar.gz (152 kB)
     |████████████████████████████████| 152 kB 5.0 MB/s            
  Preparing metadata (setup.py) ... done
Collecting MarkupSafe>=1.0
  Downloading https://thirdparty.aboutcode.org/pypi/MarkupSafe-2.0.1.tar.gz (18 kB)
  Preparing metadata (setup.py) ... done
Collecting typecode[full]>=21.6.1
  Downloading https://thirdparty.aboutcode.org/pypi/typecode-21.6.1-py3-none-any.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 4.7 MB/s             
Collecting pyahocorasick>=1.4.0
  Downloading https://thirdparty.aboutcode.org/pypi/pyahocorasick-1.4.2.tar.gz (321 kB)
     |████████████████████████████████| 321 kB 4.5 MB/s            
  Preparing metadata (setup.py) ... done
Collecting colorama>=0.3.9
  Downloading https://thirdparty.aboutcode.org/pypi/colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting text-unidecode>=1.0
  Downloading https://thirdparty.aboutcode.org/pypi/text_unidecode-1.3-py2.py3-none-any.whl (78 kB)
     |████████████████████████████████| 78 kB 33.4 MB/s            
Collecting attrs!=20.1.0,>=18.1
  Downloading https://thirdparty.aboutcode.org/pypi/attrs-21.2.0-py2.py3-none-any.whl (53 kB)
     |████████████████████████████████| 53 kB 1.5 MB/s            
Collecting saneyaml>=0.5.2
  Downloading https://thirdparty.aboutcode.org/pypi/saneyaml-0.5.2-py3-none-any.whl (11 kB)
Collecting extractcode[full]>=21.7.23
  Downloading https://thirdparty.aboutcode.org/pypi/extractcode-21.7.23-py3-none-any.whl (54 kB)
     |████████████████████████████████| 54 kB 13.9 MB/s            
Collecting toml>=0.10.0
  Downloading https://thirdparty.aboutcode.org/pypi/toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting spdx-tools>=0.7.0a3
  Downloading https://thirdparty.aboutcode.org/pypi/spdx_tools-0.7.0a3-py3-none-any.whl (227 kB)
     |████████████████████████████████| 227 kB 4.8 MB/s            
Collecting xmltodict>=0.11.0
  Downloading https://thirdparty.aboutcode.org/pypi/xmltodict-0.12.0-py2.py3-none-any.whl (9.2 kB)
Collecting fasteners
  Downloading https://thirdparty.aboutcode.org/pypi/fasteners-0.16-py2.py3-none-any.whl (28 kB)
Collecting packaging>=21.0.0
  Downloading https://thirdparty.aboutcode.org/pypi/packaging-21.3-py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 33.1 MB/s            
Collecting jinja2>=2.7.0
  Downloading https://thirdparty.aboutcode.org/pypi/Jinja2-3.0.1-py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 5.3 MB/s            
Collecting gemfileparser>=0.7.0
  Downloading https://thirdparty.aboutcode.org/pypi/gemfileparser-0.8.0-py2.py3-none-any.whl (19 kB)
Collecting packageurl-python>=0.9.0
  Downloading https://thirdparty.aboutcode.org/pypi/packageurl_python-0.9.4-py2.py3-none-any.whl (23 kB)
Collecting javaproperties>=0.5
  Downloading https://thirdparty.aboutcode.org/pypi/javaproperties-0.8.0-py3-none-any.whl (24 kB)
Collecting urlpy
  Downloading https://thirdparty.aboutcode.org/pypi/urlpy-0.5-py2.py3-none-any.whl (9.1 kB)
Collecting soupsieve>1.2
  Downloading https://thirdparty.aboutcode.org/pypi/soupsieve-2.2.1-py3-none-any.whl (33 kB)
Collecting pyyaml
  Downloading https://thirdparty.aboutcode.org/pypi/PyYAML-5.4.1.tar.gz (175 kB)
     |████████████████████████████████| 175 kB 2.3 MB/s            
  Preparing metadata (pyproject.toml) ... done
Collecting patch>=1.16
  Downloading https://thirdparty.aboutcode.org/pypi/patch-1.16-py3-none-any.whl (12 kB)
INFO: pip is looking at multiple versions of dparse2 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of debian-inspector to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of commoncode to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of colorama to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of click to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of chardet to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of boolean-py to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of bitarray to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of beautifulsoup4 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of scancode-toolkit to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install extractcode[full]==21.7.23 because these package versions have conflicting dependencies.

The conflict is caused by:
    extractcode[full] 21.7.23 depends on extractcode-libarchive>=3.5.1.210525; extra == "full"
    The user requested (constraint) extractcode-libarchive==3.5.1.210531

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

🔌  ⚡️  in scancode-toolkit/ on develop
› python -V
Python 2.7.18

🔌  ⚡️  in scancode-toolkit/ on develop
› python3 -V                             
Python 3.9.9

🔌  ⚡️  in scancode-toolkit/ on develop
› pip -V
zsh: correct 'pip' to 'pp' [nyae]? n
zsh: command not found: pip

🔌  ⚡️  in scancode-toolkit/ on develop
› pip3 -V
pip 21.3.1 from /opt/homebrew/lib/python3.9/site-packages/pip (python 3.9)

🔌  ⚡️  in scancode-toolkit/ on develop
› arch
arm64

jroes avatar Feb 02 '22 21:02 jroes

Also failed with pip install, unfortunately:

🔌  ⚡️  in jon/ 
› pip install --upgrade pip setuptools wheel && pip install scancode-toolkit
Collecting pip
  Downloading pip-22.0.2-py3-none-any.whl (2.1 MB)
     |████████████████████████████████| 2.1 MB 9.1 MB/s 
Collecting setuptools
  Using cached setuptools-60.7.0-py3-none-any.whl (1.0 MB)
Collecting wheel
  Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: pip, setuptools, wheel
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
  Attempting uninstall: setuptools
    Found existing installation: setuptools 49.2.1
    Uninstalling setuptools-49.2.1:
      Successfully uninstalled setuptools-49.2.1
Successfully installed pip-22.0.2 setuptools-60.7.0 wheel-0.37.1
Collecting scancode-toolkit
  Using cached scancode_toolkit-30.1.0-py3-none-any.whl (67.2 MB)
Collecting lxml<5.0.0,>=4.6.3
  Using cached lxml-4.7.1.tar.gz (3.2 MB)
  Preparing metadata (setup.py) ... done
Collecting text-unidecode<2.0,>=1.0
  Using cached text_unidecode-1.3-py2.py3-none-any.whl (78 kB)
Collecting jsonstreams>=0.5.0
  Using cached jsonstreams-0.6.0-py2.py3-none-any.whl (10 kB)
Collecting xmltodict>=0.11.0
  Using cached xmltodict-0.12.0-py2.py3-none-any.whl (9.2 kB)
Collecting importlib-metadata
  Using cached importlib_metadata-4.10.1-py3-none-any.whl (17 kB)
Collecting saneyaml>=0.5.2
  Using cached saneyaml-0.5.2-py3-none-any.whl (11 kB)
Collecting pygmars>=0.7.0
  Using cached pygmars-0.7.0-py3-none-any.whl (28 kB)
Collecting packageurl-python>=0.9.0
  Using cached packageurl_python-0.9.6-py3-none-any.whl (23 kB)
Collecting license-expression>=21.6.14
  Using cached license_expression-21.6.14-py3-none-any.whl (86 kB)
Collecting pefile>=2020.1.1
  Using cached pefile-2021.9.3.tar.gz (72 kB)
  Preparing metadata (setup.py) ... done
Collecting packaging>20
  Using cached packaging-21.3-py3-none-any.whl (40 kB)
Collecting dparse>=0.5.1
  Using cached dparse-0.5.1-py3-none-any.whl (12 kB)
Collecting bitarray<3.0.0,>=0.8.1
  Using cached bitarray-2.3.5.tar.gz (88 kB)
  Preparing metadata (setup.py) ... done
Collecting gemfileparser>=0.7.0
  Using cached gemfileparser-0.8.0-py2.py3-none-any.whl (19 kB)
Collecting html5lib
  Using cached html5lib-1.1-py2.py3-none-any.whl (112 kB)
Collecting jinja2<4.0.0,>=2.7.0
  Using cached Jinja2-3.0.3-py3-none-any.whl (133 kB)
Collecting chardet>=3.0.0
  Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB)
Collecting typecode[full]>=21.6.1
  Using cached typecode-21.6.1-py3-none-any.whl (1.0 MB)
Collecting zipp>=3.0.0
  Using cached zipp-3.7.0-py3-none-any.whl (5.3 kB)
Collecting Beautifulsoup4<5.0.0,>=4.0.0
  Using cached beautifulsoup4-4.10.0-py3-none-any.whl (97 kB)
Collecting pkginfo>=1.7.0
  Using cached pkginfo-1.8.2-py2.py3-none-any.whl (26 kB)
Collecting spdx-tools>=0.7.0a3
  Using cached spdx_tools-0.7.0a3-py3-none-any.whl (227 kB)
Collecting attrs!=20.1.0,>=18.1
  Using cached attrs-21.4.0-py2.py3-none-any.whl (60 kB)
Collecting jaraco.functools
  Using cached jaraco.functools-3.5.0-py3-none-any.whl (7.0 kB)
Collecting MarkupSafe>=1.0
  Downloading MarkupSafe-2.0.1-cp38-cp38-macosx_10_9_universal2.whl (18 kB)
Collecting publicsuffix2
  Using cached publicsuffix2-2.20191221-py2.py3-none-any.whl (89 kB)
Collecting urlpy
  Using cached urlpy-0.5-py2.py3-none-any.whl (9.1 kB)
Collecting pymaven-patch>=0.2.8
  Using cached pymaven_patch-0.3.0-py2.py3-none-any.whl (23 kB)
Collecting commoncode>=30.0.0
  Using cached commoncode-30.0.0-py3-none-any.whl (82 kB)
Collecting click!=7.0,>=6.7
  Using cached click-8.0.3-py3-none-any.whl (97 kB)
Collecting pdfminer.six>=20200101
  Using cached pdfminer.six-20211012-py3-none-any.whl (5.6 MB)
Collecting pygments
  Using cached Pygments-2.11.2-py3-none-any.whl (1.1 MB)
Collecting intbitset<3.0,>=2.3.0
  Using cached intbitset-2.4.1.tar.gz (152 kB)
  Preparing metadata (setup.py) ... done
Collecting debian-inspector>=30.0.0
  Using cached debian_inspector-30.0.0-py3-none-any.whl (43 kB)
Collecting requests<3.0.0,>=2.7.0
  Using cached requests-2.27.1-py2.py3-none-any.whl (63 kB)
Collecting pyahocorasick<1.5,>=1.4.0
  Using cached pyahocorasick-1.4.2.tar.gz (321 kB)
  Preparing metadata (setup.py) ... done
Collecting pluggy<1.0,>=0.12.0
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting boolean.py<4.0,>=3.5
  Using cached boolean.py-3.8-py2.py3-none-any.whl (23 kB)
Collecting extractcode[full]>=21.7.23
  Using cached extractcode-30.0.0-py3-none-any.whl (54 kB)
Collecting parameter-expansion-patched>=0.2.1b4
  Using cached parameter_expansion_patched-0.2.1b4-py3-none-any.whl (12 kB)
Collecting colorama>=0.3.9
  Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting fingerprints>=0.6.0
  Using cached fingerprints-1.0.3-py2.py3-none-any.whl (13 kB)
Collecting plugincode>=21.1.21
  Using cached plugincode-21.1.21-py3-none-any.whl (18 kB)
Collecting fasteners
  Using cached fasteners-0.17.3-py3-none-any.whl (18 kB)
Collecting ftfy>=4.0.0
  Using cached ftfy-6.0.3.tar.gz (64 kB)
  Preparing metadata (setup.py) ... done
Collecting javaproperties>=0.5
  Using cached javaproperties-0.8.1-py3-none-any.whl (23 kB)
Collecting toml>=0.10.0
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting soupsieve>1.2
  Downloading soupsieve-2.3.1-py3-none-any.whl (37 kB)
Collecting pyyaml
  Using cached PyYAML-6.0.tar.gz (124 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting patch>=1.16
  Downloading patch-1.16.zip (12 kB)
  Preparing metadata (setup.py) ... done
Collecting extractcode[full]>=21.7.23
  Downloading extractcode-21.7.23-py3-none-any.whl (54 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.5/54.5 KB 2.1 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of dparse to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of debian-inspector to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of commoncode to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of colorama to determine which version is compatible with other requirements. This could take a while.
Collecting colorama>=0.3.9
  Downloading colorama-0.4.3-py2.py3-none-any.whl (15 kB)
INFO: pip is looking at multiple versions of click to determine which version is compatible with other requirements. This could take a while.
Collecting click!=7.0,>=6.7
  Downloading click-8.0.2-py3-none-any.whl (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.6/97.6 KB 4.1 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of chardet to determine which version is compatible with other requirements. This could take a while.
Collecting chardet>=3.0.0
  Downloading chardet-3.0.4-py2.py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.4/133.4 KB 5.7 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of boolean-py to determine which version is compatible with other requirements. This could take a while.
Collecting boolean.py<4.0,>=3.5
  Downloading boolean.py-3.7-py2.py3-none-any.whl (22 kB)
INFO: pip is looking at multiple versions of bitarray to determine which version is compatible with other requirements. This could take a while.
Collecting bitarray<3.0.0,>=0.8.1
  Downloading bitarray-2.3.4.tar.gz (88 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 88.2/88.2 KB 3.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
INFO: pip is looking at multiple versions of beautifulsoup4 to determine which version is compatible with other requirements. This could take a while.
Collecting Beautifulsoup4<5.0.0,>=4.0.0
  Downloading beautifulsoup4-4.9.3-py3-none-any.whl (115 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.8/115.8 KB 5.0 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
Collecting attrs!=20.1.0,>=18.1
  Downloading attrs-21.3.0-py2.py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.9/61.9 KB 2.6 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of scancode-toolkit to determine which version is compatible with other requirements. This could take a while.
Collecting scancode-toolkit
  Downloading scancode_toolkit-30.0.1-py3-none-any.whl (67.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.2/67.2 MB 51.4 MB/s eta 0:00:00
  Downloading scancode_toolkit-30.0.0-py3-none-any.whl (67.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 67.2/67.2 MB 50.1 MB/s eta 0:00:00
  Downloading scancode_toolkit-21.8.4-py3-none-any.whl (61.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.4/61.4 MB 46.6 MB/s eta 0:00:00
Collecting spdx-tools>=0.6.0
  Downloading spdx_tools-0.6.1-py2.py3-none-any.whl (206 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 206.8/206.8 KB 8.4 MB/s eta 0:00:00
Collecting scancode-toolkit
  Downloading scancode_toolkit-21.7.30-py3-none-any.whl (61.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.4/61.4 MB 46.8 MB/s eta 0:00:00
  Downloading scancode_toolkit-21.6.7-py3-none-any.whl (61.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.2/61.2 MB 43.7 MB/s eta 0:00:00
Collecting nltk!=3.6,<4.0,>=3.2
  Downloading nltk-3.6.7-py3-none-any.whl (1.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 21.4 MB/s eta 0:00:00
Collecting extractcode[full]>=21.6.1
  Downloading extractcode-21.6.2-py3-none-any.whl (54 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.3/54.3 KB 2.0 MB/s eta 0:00:00
  Downloading extractcode-21.6.1-py3-none-any.whl (51 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.9/51.9 KB 1.7 MB/s eta 0:00:00
Collecting scancode-toolkit
  Downloading scancode_toolkit-21.3.31-py3-none-any.whl (55.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.7/55.7 MB 45.2 MB/s eta 0:00:00
Collecting dparse2
  Downloading dparse2-0.6.0-py3-none-any.whl (10 kB)
Collecting jinja2<3.0.0,>=2.7.0
  Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.7/125.7 KB 5.4 MB/s eta 0:00:00
Collecting pycryptodome>=3.4
  Downloading pycryptodome-3.14.0.tar.gz (3.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.4/3.4 MB 41.1 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting ftfy<5.0.0
  Downloading ftfy-4.4.3.tar.gz (50 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.5/50.5 KB 1.6 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting bitarray<1.0.0,>=0.8.1
  Downloading bitarray-0.9.3.tar.gz (38 kB)
  Preparing metadata (setup.py) ... done
Collecting extractcode[full]>=21.2.24
  Downloading extractcode-21.2.24-py3-none-any.whl (46 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 46.0/46.0 KB 1.6 MB/s eta 0:00:00
Collecting scancode-toolkit
  Downloading scancode_toolkit-21.2.25-py3-none-any.whl (52.4 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.4/52.4 MB 40.3 MB/s eta 0:00:00
Collecting debut>=0.9.9
  Downloading debut-0.9.9-py2.py3-none-any.whl (40 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.0/41.0 KB 1.6 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of scancode-toolkit to determine which version is compatible with other requirements. This could take a while.
Collecting scancode-toolkit
  Downloading scancode_toolkit-21.2.9-py3-none-any.whl (50.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 50.1/50.1 MB 45.5 MB/s eta 0:00:00
Collecting normality>=2.0.0
  Downloading normality-2.2.5-py2.py3-none-any.whl (12 kB)
Collecting wcwidth
  Using cached wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting regex>=2021.8.3
  Downloading regex-2022.1.18-cp38-cp38-macosx_11_0_arm64.whl (281 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 281.5/281.5 KB 10.5 MB/s eta 0:00:00
Collecting tqdm
  Downloading tqdm-4.62.3-py2.py3-none-any.whl (76 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 76.2/76.2 KB 3.2 MB/s eta 0:00:00
Collecting joblib
  Downloading joblib-1.1.0-py2.py3-none-any.whl (306 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 307.0/307.0 KB 11.7 MB/s eta 0:00:00
Collecting cryptography
  Downloading cryptography-36.0.1-cp36-abi3-macosx_10_10_universal2.whl (4.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 36.8 MB/s eta 0:00:00
Collecting future
  Downloading future-0.18.2.tar.gz (829 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 829.2/829.2 KB 19.9 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.11-py3-none-any.whl (39 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.10.8-py2.py3-none-any.whl (149 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.3-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.8-py2.py3-none-any.whl (138 kB)
Collecting ply
  Downloading ply-3.11-py2.py3-none-any.whl (49 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 49.6/49.6 KB 2.0 MB/s eta 0:00:00
Collecting rdflib
  Downloading rdflib-6.1.1-py3-none-any.whl (482 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 482.8/482.8 KB 15.9 MB/s eta 0:00:00
Collecting binaryornot
  Using cached binaryornot-0.4.4-py2.py3-none-any.whl (9.0 kB)
Collecting webencodings
  Using cached webencodings-0.5.1-py2.py3-none-any.whl (11 kB)
Collecting more-itertools
  Downloading more_itertools-8.12.0-py3-none-any.whl (54 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.3/54.3 KB 2.1 MB/s eta 0:00:00
Collecting banal>=1.0.1
  Downloading banal-1.0.6-py2.py3-none-any.whl (6.1 kB)
Collecting cffi>=1.12
  Downloading cffi-1.15.0.tar.gz (484 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 484.1/484.1 KB 15.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Collecting pyparsing!=3.0.5,>=2.0.2
  Using cached pyparsing-3.0.7-py3-none-any.whl (98 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.8/site-packages (from rdflib->spdx-tools>=0.6.0->scancode-toolkit) (60.7.0)
Collecting isodate
  Downloading isodate-0.6.1-py2.py3-none-any.whl (41 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.7/41.7 KB 1.5 MB/s eta 0:00:00
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Building wheels for collected packages: bitarray, ftfy, intbitset, lxml, pefile, pyahocorasick, pycryptodome, future, pyyaml, cffi
  Building wheel for bitarray (setup.py) ... done
  Created wheel for bitarray: filename=bitarray-0.9.3-cp38-cp38-macosx_10_14_arm64.whl size=61342 sha256=42ce7c290ebde30425dc027786602298546ec8c541e024f336cabefc8f2c35c2
  Stored in directory: /Users/jon/Library/Caches/pip/wheels/ed/c4/d1/73abb0f702f16fc38a3d2e30a305c0cdce6132e7efd04865f1
  Building wheel for ftfy (setup.py) ... done
  Created wheel for ftfy: filename=ftfy-4.4.3-py3-none-any.whl size=41082 sha256=43ba19eabb20e7e379e607d2fdfd1342b4c3fbfc3550d0e65b914d9b2a2d7e57
  Stored in directory: /Users/jon/Library/Caches/pip/wheels/97/ea/83/0ea1632b87aa8a3ecdc2c3914d7dc786ed2ff37eb9e9242f3a
  Building wheel for intbitset (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      running bdist_wheel
      running build
      running build_py
      running egg_info
      writing intbitset/intbitset.egg-info/PKG-INFO
      writing dependency_links to intbitset/intbitset.egg-info/dependency_links.txt
      writing requirements to intbitset/intbitset.egg-info/requires.txt
      writing top-level names to intbitset/intbitset.egg-info/top_level.txt
      reading manifest file 'intbitset/intbitset.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching '*.css' under directory 'docs/_themes'
      warning: no files found matching '*.css_t' under directory 'docs/_themes'
      warning: no files found matching '*.conf' under directory 'docs/_themes'
      warning: no files found matching '*.html' under directory 'docs/_themes'
      warning: no files found matching 'COPYING' under directory 'docs/_themes'
      warning: no files found matching 'README' under directory 'docs/_themes'
      warning: no files found matching '*.html' under directory 'docs/_templates'
      adding license file 'LICENSE'
      adding license file 'README.rst'
      adding license file 'CHANGES.rst'
      adding license file 'RELEASE-NOTES.rst'
      running build_ext
      building 'intbitset' extension
      clang: error: the clang compiler does not support '-march=core2'
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for intbitset
  Running setup.py clean for intbitset
  Building wheel for lxml (setup.py) ... done
  Created wheel for lxml: filename=lxml-4.7.1-cp38-cp38-macosx_10_14_arm64.whl size=3003854 sha256=cef0fa35314b78b97de6373247bd60d86ff051c669ed95d4b25c746c7f2c861b
  Stored in directory: /Users/jon/Library/Caches/pip/wheels/c7/d8/ac/9c62a8804498b75b652f0cfbe020c93ae3f7265a9bbb40cee9
  Building wheel for pefile (setup.py) ... done
  Created wheel for pefile: filename=pefile-2021.9.3-py3-none-any.whl size=68846 sha256=7a47321437c867eae0853c6d87678cee712ed077be2829f32d34a5912864bbfa
  Stored in directory: /Users/jon/Library/Caches/pip/wheels/ee/a0/e1/ff2dafe5ae846f536acf0a4cc4a8b8d4ccb3044ab87f3ef174
  Building wheel for pyahocorasick (setup.py) ... done
  Created wheel for pyahocorasick: filename=pyahocorasick-1.4.2-cp38-cp38-macosx_10_14_arm64.whl size=57725 sha256=9a7a8333d046929d5bd8918f971282d73ee7b618c00ffc83c5f109627f9daeca
  Stored in directory: /Users/jon/Library/Caches/pip/wheels/74/bc/b8/e5f739a84005620cfe66d3fcb8bb182e309d6056bc6700b60e
  Building wheel for pycryptodome (setup.py) ... done
  Created wheel for pycryptodome: filename=pycryptodome-3.14.0-cp35-abi3-macosx_10_14_arm64.whl size=2301585 sha256=aec5a963d421915ea46c50ab21b7bca03ca6455975719509d31449598d36e424
  Stored in directory: /Users/jon/Library/Caches/pip/wheels/b5/8f/73/2c6eacf228ae4e0233b3e5a5f5e6ae9617867ffb1b880e0a0a
  Building wheel for future (setup.py) ... done
  Created wheel for future: filename=future-0.18.2-py3-none-any.whl size=491070 sha256=5170b199366504227a8b4e6fb2b58fc101c2f051db6c94ec373e7dbae51c87df
  Stored in directory: /Users/jon/Library/Caches/pip/wheels/8e/70/28/3d6ccd6e315f65f245da085482a2e1c7d14b90b30f239e2cf4
  Building wheel for pyyaml (pyproject.toml) ... done
  Created wheel for pyyaml: filename=PyYAML-6.0-cp38-cp38-macosx_10_14_arm64.whl size=45339 sha256=4690ea7d8cee0f23c81adac9a80ba5fa71e4ebba9924cceee62fe1fe9b6c764e
  Stored in directory: /Users/jon/Library/Caches/pip/wheels/52/84/66/50912fd7bf1639a31758e40bd4312602e104a8eca1e0da9645
  Building wheel for cffi (setup.py) ... done
  Created wheel for cffi: filename=cffi-1.15.0-cp38-cp38-macosx_10_14_arm64.whl size=258929 sha256=84a92d44b78bfa73001a504d76dddea614e68b6c846d6827358ec097e3307b87
  Stored in directory: /Users/jon/Library/Caches/pip/wheels/a4/cb/1a/277a076c4434fadc0707a37ba48587dc2ac6397d517c3b9de7
Successfully built bitarray ftfy lxml pefile pyahocorasick pycryptodome future pyyaml cffi
Failed to build intbitset
Installing collected packages: webencodings, wcwidth, text-unidecode, regex, pyahocorasick, publicsuffix2, ply, pkginfo, gemfileparser, certifi, boolean.py, bitarray, banal, xmltodict, urlpy, urllib3, tqdm, toml, soupsieve, six, pyyaml, pyparsing, pygments, pycryptodome, pycparser, pluggy, packageurl-python, more-itertools, MarkupSafe, lxml, license-expression, joblib, javaproperties, idna, future, fasteners, colorama, click, charset-normalizer, chardet, attrs, saneyaml, requests, pefile, packaging, normality, nltk, jsonstreams, jinja2, jaraco.functools, isodate, intbitset, html5lib, debut, cffi, binaryornot, Beautifulsoup4, rdflib, pymaven-patch, ftfy, fingerprints, dparse2, cryptography, commoncode, spdx-tools, plugincode, pdfminer.six, typecode, scancode-toolkit
  Running setup.py install for intbitset ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for intbitset did not run successfully.
  │ exit code: 1
  ╰─> [27 lines of output]
      running install
      /Users/jon/venv/lib/python3.8/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      running egg_info
      writing intbitset/intbitset.egg-info/PKG-INFO
      writing dependency_links to intbitset/intbitset.egg-info/dependency_links.txt
      writing requirements to intbitset/intbitset.egg-info/requires.txt
      writing top-level names to intbitset/intbitset.egg-info/top_level.txt
      reading manifest file 'intbitset/intbitset.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching '*.css' under directory 'docs/_themes'
      warning: no files found matching '*.css_t' under directory 'docs/_themes'
      warning: no files found matching '*.conf' under directory 'docs/_themes'
      warning: no files found matching '*.html' under directory 'docs/_themes'
      warning: no files found matching 'COPYING' under directory 'docs/_themes'
      warning: no files found matching 'README' under directory 'docs/_themes'
      warning: no files found matching '*.html' under directory 'docs/_templates'
      adding license file 'LICENSE'
      adding license file 'README.rst'
      adding license file 'CHANGES.rst'
      adding license file 'RELEASE-NOTES.rst'
      running build_ext
      building 'intbitset' extension
      clang: error: the clang compiler does not support '-march=core2'
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> intbitset

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
(venv)

jroes avatar Feb 02 '22 21:02 jroes

@jroes Thank you ++ this helps!

I am working on a fix for this.

See also these:

  • https://github.com/nexB/scancode-toolkit/issues/2836
  • https://github.com/nexB/skeleton/issues/53
  • and the many issues listed https://github.com/nexB/scancode-toolkit/issues?q=is%3Aopen+is%3Aissue+label%3A"installation+and+packaging"+label%3A"os+support"

pombredanne avatar Feb 03 '22 08:02 pombredanne

We may run OK on Apple ARM M1 too though this is not tested and likely requires the x86 emulation mode.

For me, running ScanCode on my M1 Mac fails using x86_64 bash and python3.8:

~/bin/scancode-toolkit-30.1.0
❯ arch -x86_64 /opt/local/bin/bash -c "/opt/local/bin/python3.8 -c 'import os; print(os.uname())'"
posix.uname_result(sysname='Darwin', nodename='Marcels-MacBook-Pro.local', release='21.3.0', version='Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_ARM64_T6000', machine='x86_64')

~/bin/scancode-toolkit-30.1.0
❯ arch -x86_64 /opt/local/bin/bash -c "./scancode --help"
* Configuring ScanCode for first use...
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/marcel/bin/scancode-toolkit-30.1.0/etc/thirdparty/virtualenv.pyz/__main__.py", line 4, in <module>
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/zipfile.py", line 6, in <module>
    import binascii
ImportError: dynamic module does not define module export function (PyInit_binascii)

MarcelBochtler avatar Feb 18 '22 22:02 MarcelBochtler

I'm also trying to install on m1 mac, but I get the same error as https://github.com/nexB/scancode-toolkit/issues/2835#issuecomment-1028381190. (m1, python 3.9)

soimkim avatar Mar 07 '22 07:03 soimkim

@soimkim @MarcelBochtler Thank you for the reports. I now officially hate Apple and ARM. :angel:

pombredanne avatar Mar 07 '22 09:03 pombredanne

Downloaded tarball fails on my m1:

Successfully installed Beautifulsoup4-4.9.3 MarkupSafe-2.0.1 attrs-21.2.0 banal-1.0.6 binaryornot-0.4.4 bitarray-2.1.0 boolean.py-3.8 certifi-2021.5.30 cffi-1.14.5 chardet-4.0.0 click-8.0.1 colorama-0.4.4 commoncode-30.0.0 cryptography-3.4.7 debian-inspector-30.0.0 dparse-0.5.1 extractcode-21.7.23 extractcode-7z-16.5.210531 extractcode-libarchive-3.5.1.210531 fasteners-0.16 fingerprints-1.0.3 ftfy-6.0.3 future-0.18.2 gemfileparser-0.8.0 html5lib-1.1 idna-2.10 intbitset-2.4.1 isodate-0.6.0 jaraco.functools-3.3.0 javaproperties-0.8.0 jinja2-3.0.1 jsonstreams-0.6.0 license-expression-21.6.14 lxml-4.6.3 more-itertools-8.8.0 normality-2.2.2 packageurl-python-0.9.4 packaging-20.9 parameter-expansion-patched-0.2.1b4 patch-1.16 pdfminer.six-20201018 pefile-2021.5.24 pkginfo-1.7.0 pluggy-0.13.1 plugincode-21.1.21 ply-3.11 publicsuffix2-2.20191221 pyahocorasick-1.4.2 pycparser-2.20 pygmars-0.7.0 pygments-2.9.0 pymaven-patch-0.3.0 pyparsing-2.4.7 pyyaml-5.4.1 rdflib-5.0.0 requests-2.25.1 saneyaml-0.5.2 scancode-toolkit-30.1.0 six-1.16.0 sortedcontainers-2.4.0 soupsieve-2.2.1 spdx-tools-0.7.0a3 text-unidecode-1.3 toml-0.10.2 typecode-21.6.1 typecode-libmagic-5.39.210531 urllib3-1.26.5 urlpy-0.5 wcwidth-0.2.5 webencodings-0.5.1 xmltodict-0.12.0
Traceback (most recent call last):
  File "/Users/gus/tools/scancode-toolkit-30.1.0/bin/scancode", line 33, in <module>
    sys.exit(load_entry_point('scancode-toolkit', 'console_scripts', 'scancode')())
  File "/Users/gus/tools/scancode-toolkit-30.1.0/bin/scancode", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/gus/tools/scancode-toolkit-30.1.0/src/scancode/cli.py", line 42, in <module>
    from commoncode.resource import Codebase
  File "/Users/gus/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/commoncode/resource.py", line 27, in <module>
    from intbitset import intbitset
ImportError: dlopen(/Users/gus/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/intbitset.cpython-39-darwin.so, 0x0002): tried: '/Users/gus/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/intbitset.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64e')), '/usr/lib/intbitset.cpython-39-darwin.so' (no such file)

scancode-toolkit-30.1.0_py39-macos.tar.xz

NS2-MacBook-Pro:tools gus$ python3 --version
Python 3.9.13
NS2-MacBook-Pro:tools gus$ which python3
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3
NS2-MacBook-Pro:tools gus$ file `which python3`
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64:Mach-O 64-bit executable arm64]
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3 (for architecture x86_64):	Mach-O 64-bit executable x86_64
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3 (for architecture arm64):	Mach-O 64-bit executable arm64
NS2-MacBook-Pro:tools gus$ file /Users/gus/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/intbitset.cpython-39-darwin.so
/Users/gus/tools/scancode-toolkit-30.1.0/lib/python3.9/site-packages/intbitset.cpython-39-darwin.so: Mach-O 64-bit bundle x86_64

nsoft avatar Jul 07 '22 23:07 nsoft

@nsoft Thanks. The latest 31.* release candidate tarball should work on macOS with an M1 chip. The trick is that it requires to run in x86 emulation mode (aka. "Rosetta")... The installation/setup script in the release tarball handles this. See https://github.com/nexB/scancode-toolkit/blob/746c33683225efe16cf6630753b51ef7697574bb/scancode#L123

Alternatively you can force this mode on older versions by running first arch -x86_64 /bin/bash and then running configure or scancode proper!

See also #2938 and in particular this comment: https://github.com/nexB/scancode-toolkit/issues/2938#issuecomment-1154747132

pombredanne avatar Jul 08 '22 11:07 pombredanne

@soimkim @MarcelBochtler @jroes I would be grateful if you can try and report if the issue is still there with the latest 31.x release candidate or the procedure explained above https://github.com/nexB/scancode-toolkit/issues/2835#issuecomment-1178867194

pombredanne avatar Jul 08 '22 11:07 pombredanne

@pombredanne , In the environment set with pyenv on Mac M1, the following error occurs when pip install scancode-toolkit.

Collecting bitarray<3.0.0,>=0.8.1
  Downloading bitarray-2.6.0.tar.gz (102 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 102.8/102.8 kB 3.1 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [1 lines of output]
      ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

However, in the environment set with conda, it is installed without error. On Mac M1, installing and using scancode-toolkit with conda solves the problem. 😃

soimkim avatar Jul 26 '22 00:07 soimkim

@soimkim Thank you ++ for the report! For a pip install scancode-toolkit on Apple M1 you would need to ensure that you run in rosetta X86 emulation mode for now using either arch or starting a terminal in X86 mode.

pombredanne avatar Jul 27 '22 15:07 pombredanne

The doc is updated in the https://github.com/nexB/scancode-toolkit/pull/3053 PR. There are also extensive release tests on various OS

pombredanne avatar Aug 17 '22 13:08 pombredanne

The tests are also running on release archives on various mac, windows and linux configurations. I am running Apple M1 tests by hand on a borrowed machine. Closing this for now! Thanks you all.

pombredanne avatar Aug 17 '22 13:08 pombredanne