frontera
frontera copied to clipboard
scrapinghub
pip install Failed with exit status 2
I can't install the library with pip. System: Windows 10 Python: 3.6.5 pip: 19.0.3
`Collecting cityhash>=0.1.7 (from frontera) Using cached https://files.pythonhosted.org/packages/25/67/65ad3755997c066766750525fdf3045609f5ee892fa5b037e7fc066505a4/cityhash-0.2.3.post9.tar.gz Requirement already satisfied: w3lib>=1.15.0 in c:\users\leond\anaconda3\lib\site-packages (from frontera) (1.19.0) Requirement already satisfied: six>=1.8.0 in c:\users\leond\anaconda3\lib\site-packages (from frontera) (1.11.0) Building wheels for collected packages: cityhash Building wheel for cityhash (setup.py) ... \ Subprocess output does not appear to be encoded as cp1252 error Complete output from command c:\users\leond\anaconda3\python.exe -u -c "import setuptools, tokenize;file='C:\Users\leond\AppData\Local\Temp\pip-install-y5d4iow2\cityhash\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d C:\Users\leond\AppData\Local\Temp\pip-wheel-iio6q8jj --python-tag cp36: running bdist_wheel running build running build_ext skipping 'src\cityhash.cpp' Cython extension (up-to-date) building 'cityhash' extension creating build creating build\temp.win-amd64-3.6 creating build\temp.win-amd64-3.6\Release creating build\temp.win-amd64-3.6\Release\src D:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.20.27508\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Iinclude -Ic:\users\leond\anaconda3\include -Ic:\users\leond\anaconda3\include "-ID:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.20.27508\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" /EHsc /Tpsrc/city.cc /Fobuild\temp.win-amd64-3.6\Release\src/city.obj -O3 -Wno-unused-value -Wno-unused-function cl : Befehlszeile error D8021 : Ung\x81ltiges numerisches Argument /Wno-unused-value. error: command 'D:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.20.27508\bin\HostX86\x64\cl.exe' failed with exit status 2
Failed building wheel for cityhash Running setup.py clean for cityhash Failed to build cityhash Installing collected packages: cityhash, frontera Running setup.py install for cityhash ... \ Subprocess output does not appear to be encoded as cp1252 error Complete output from command c:\users\leond\anaconda3\python.exe -u -c "import setuptools, tokenize;file='C:\Users\leond\AppData\Local\Temp\pip-install-y5d4iow2\cityhash\setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record C:\Users\leond\AppData\Local\Temp\pip-record-8o88hmtl\install-record.txt --single-version-externally-managed --compile: running install running build running build_ext skipping 'src\cityhash.cpp' Cython extension (up-to-date) building 'cityhash' extension creating build creating build\temp.win-amd64-3.6 creating build\temp.win-amd64-3.6\Release creating build\temp.win-amd64-3.6\Release\src D:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.20.27508\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Iinclude -Ic:\users\leond\anaconda3\include -Ic:\users\leond\anaconda3\include "-ID:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.20.27508\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\cppwinrt" /EHsc /Tpsrc/city.cc /Fobuild\temp.win-amd64-3.6\Release\src/city.obj -O3 -Wno-unused-value -Wno-unused-function cl : Befehlszeile error D8021 : Ung\x81ltiges numerisches Argument /Wno-unused-value. error: command 'D:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.20.27508\bin\HostX86\x64\cl.exe' failed with exit status 2`
The problems seems to be cityhash, but trying any version between 0.1.7-0.2.3 seems to create the same problem.
Yes, cityhash can't compile on Windows. Honestly, we (Scrapinghub) never tried Frontera on Windows, so likely you will run into previously undiscovered issues. Therefore, I'm suggesting to switch to Linux.
Other option to solve this is to use CRC32 instead of cityhash here https://github.com/scrapinghub/frontera/blob/da87cbdcd4b3577f292bca77571e90e8b3a3140a/frontera/contrib/backends/partitioners.py This is the only place where cityhash is used.
