FastChat
FastChat copied to clipboard
Published
20 hours ago •
lm-sys
lm-sys
Failed to install sentencepiece
trafficstars
Hello folks,
I'm strugling to install fastchat, it seems to fail to install sentencepiece. Any help appreciated!
C:\chat\FastChat>pip list
Package Version
---------- -------
pip 23.0.1
setuptools 65.5.0
tokenizers 0.13.3
wcwidth 0.2.6
C:\chat\FastChat>python --version
Python 3.11.3
This is the output when installing:
C:\chat\FastChat>pip3 install -e .
Obtaining file:///C:/chat/FastChat
Installing build dependencies ... done
Checking if build backend supports build_editable ... done
Getting requirements to build editable ... done
Installing backend dependencies ... done
Preparing editable metadata (pyproject.toml) ... done
Collecting transformers@ git+https://github.com/huggingface/transformers.git
Cloning https://github.com/huggingface/transformers.git to c:\users\msn\appdata\local\temp\pip-install-rcqm1i7z\transformers_cd16d0a0811b4ee1b1d70cc4eb7a391d
Running command git clone --filter=blob:none --quiet https://github.com/huggingface/transformers.git 'C:\Users\msn\AppData\Local\Temp\pip-install-rcqm1i7z\transformers_cd16d0a0811b4ee1b1d70cc4eb7a391d'
Resolved https://github.com/huggingface/transformers.git to commit 656e869a4523f6a0ce90b3aacbb05cc8fb5794bb
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting accelerate
Using cached accelerate-0.18.0-py3-none-any.whl (215 kB)
Collecting fastapi
Using cached fastapi-0.95.0-py3-none-any.whl (57 kB)
Collecting gradio==3.23
Using cached gradio-3.23.0-py3-none-any.whl (15.8 MB)
Collecting markdown2[all]
Using cached markdown2-2.4.8-py2.py3-none-any.whl (38 kB)
Collecting numpy
Using cached numpy-1.24.2-cp311-cp311-win_amd64.whl (14.8 MB)
Collecting requests
Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Collecting sentencepiece
Using cached sentencepiece-0.1.97.tar.gz (524 kB)
Preparing metadata (setup.py) ... done
Requirement already satisfied: tokenizers>=0.12.1 in c:\program files\python311\lib\site-packages (from fschat==0.1.9) (0.13.3)
Collecting torch
Using cached torch-2.0.0-cp311-cp311-win_amd64.whl (172.3 MB)
Collecting uvicorn
Using cached uvicorn-0.21.1-py3-none-any.whl (57 kB)
Collecting wandb
Using cached wandb-0.14.2-py3-none-any.whl (2.0 MB)
Collecting prompt-toolkit>=3.0.0
Downloading prompt_toolkit-3.0.38-py3-none-any.whl (385 kB)
---------------------------------------- 385.8/385.8 kB 1.2 MB/s eta 0:00:00
Collecting rich>=10.0.0
Using cached rich-13.3.3-py3-none-any.whl (238 kB)
Collecting aiofiles
Using cached aiofiles-23.1.0-py3-none-any.whl (14 kB)
Collecting aiohttp
Using cached aiohttp-3.8.4-cp311-cp311-win_amd64.whl (317 kB)
Collecting altair>=4.2.0
Using cached altair-4.2.2-py3-none-any.whl (813 kB)
Collecting ffmpy
Using cached ffmpy-0.3.0.tar.gz (4.8 kB)
Preparing metadata (setup.py) ... done
Collecting fsspec
Using cached fsspec-2023.4.0-py3-none-any.whl (153 kB)
Collecting httpx
Using cached httpx-0.23.3-py3-none-any.whl (71 kB)
Collecting huggingface-hub
Using cached huggingface_hub-0.13.4-py3-none-any.whl (200 kB)
Collecting jinja2
Using cached Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting markdown-it-py[linkify]>=2.0.0
Using cached markdown_it_py-2.2.0-py3-none-any.whl (84 kB)
Collecting markupsafe
Using cached MarkupSafe-2.1.2-cp311-cp311-win_amd64.whl (16 kB)
Collecting matplotlib
Using cached matplotlib-3.7.1-cp311-cp311-win_amd64.whl (7.6 MB)
Collecting mdit-py-plugins<=0.3.3
Using cached mdit_py_plugins-0.3.3-py3-none-any.whl (50 kB)
Collecting orjson
Using cached orjson-3.8.9-cp311-none-win_amd64.whl (202 kB)
Collecting pandas
Using cached pandas-2.0.0-cp311-cp311-win_amd64.whl (11.2 MB)
Collecting pillow
Using cached Pillow-9.5.0-cp311-cp311-win_amd64.whl (2.5 MB)
Collecting pydantic
Using cached pydantic-1.10.7-cp311-cp311-win_amd64.whl (2.1 MB)
Collecting pydub
Using cached pydub-0.25.1-py2.py3-none-any.whl (32 kB)
Collecting python-multipart
Using cached python_multipart-0.0.6-py3-none-any.whl (45 kB)
Collecting pyyaml
Using cached PyYAML-6.0-cp311-cp311-win_amd64.whl (143 kB)
Collecting semantic-version
Using cached semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
Collecting typing-extensions
Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Collecting websockets>=10.0
Using cached websockets-11.0.1-cp311-cp311-win_amd64.whl (124 kB)
Collecting wcwidth
Downloading wcwidth-0.2.6-py2.py3-none-any.whl (29 kB)
Collecting pygments<3.0.0,>=2.13.0
Using cached Pygments-2.14.0-py3-none-any.whl (1.1 MB)
Collecting packaging>=20.0
Using cached packaging-23.0-py3-none-any.whl (42 kB)
Collecting psutil
Using cached psutil-5.9.4-cp36-abi3-win_amd64.whl (252 kB)
Collecting filelock
Using cached filelock-3.11.0-py3-none-any.whl (10.0 kB)
Collecting sympy
Using cached sympy-1.11.1-py3-none-any.whl (6.5 MB)
Collecting networkx
Using cached networkx-3.1-py3-none-any.whl (2.1 MB)
Collecting starlette<0.27.0,>=0.26.1
Using cached starlette-0.26.1-py3-none-any.whl (66 kB)
Collecting wavedrom
Using cached wavedrom-2.0.3.post3.tar.gz (137 kB)
Preparing metadata (setup.py) ... done
Collecting charset-normalizer<4,>=2
Using cached charset_normalizer-3.1.0-cp311-cp311-win_amd64.whl (96 kB)
Collecting idna<4,>=2.5
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<1.27,>=1.21.1
Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
Collecting certifi>=2017.4.17
Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting regex!=2019.12.17
Using cached regex-2023.3.23-cp311-cp311-win_amd64.whl (267 kB)
Collecting tqdm>=4.27
Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
Collecting click>=7.0
Using cached click-8.1.3-py3-none-any.whl (96 kB)
Collecting h11>=0.8
Using cached h11-0.14.0-py3-none-any.whl (58 kB)
Collecting GitPython!=3.1.29,>=1.0.0
Using cached GitPython-3.1.31-py3-none-any.whl (184 kB)
Collecting sentry-sdk>=1.0.0
Using cached sentry_sdk-1.19.1-py2.py3-none-any.whl (199 kB)
Collecting docker-pycreds>=0.4.0
Using cached docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)
Collecting pathtools
Using cached pathtools-0.1.2.tar.gz (11 kB)
Preparing metadata (setup.py) ... done
Collecting setproctitle
Using cached setproctitle-1.3.2-cp311-cp311-win_amd64.whl (11 kB)
Requirement already satisfied: setuptools in c:\program files\python311\lib\site-packages (from wandb->fschat==0.1.9) (65.5.0)
Collecting appdirs>=1.4.3
Using cached appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
Collecting protobuf!=4.21.0,<5,>=3.19.0
Using cached protobuf-4.22.1-cp310-abi3-win_amd64.whl (420 kB)
Collecting entrypoints
Using cached entrypoints-0.4-py3-none-any.whl (5.3 kB)
Collecting jsonschema>=3.0
Using cached jsonschema-4.17.3-py3-none-any.whl (90 kB)
Collecting toolz
Using cached toolz-0.12.0-py3-none-any.whl (55 kB)
Collecting colorama
Using cached colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Collecting six>=1.4.0
Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting gitdb<5,>=4.0.1
Using cached gitdb-4.0.10-py3-none-any.whl (62 kB)
Collecting mdurl~=0.1
Using cached mdurl-0.1.2-py3-none-any.whl (10.0 kB)
Collecting linkify-it-py<3,>=1
Using cached linkify_it_py-2.0.0-py3-none-any.whl (19 kB)
Collecting python-dateutil>=2.8.2
Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting pytz>=2020.1
Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
Collecting tzdata>=2022.1
Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)
Collecting anyio<5,>=3.4.0
Using cached anyio-3.6.2-py3-none-any.whl (80 kB)
Collecting attrs>=17.3.0
Using cached attrs-22.2.0-py3-none-any.whl (60 kB)
Collecting multidict<7.0,>=4.5
Using cached multidict-6.0.4-cp311-cp311-win_amd64.whl (28 kB)
Collecting async-timeout<5.0,>=4.0.0a3
Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting yarl<2.0,>=1.0
Using cached yarl-1.8.2-cp311-cp311-win_amd64.whl (55 kB)
Collecting frozenlist>=1.1.1
Using cached frozenlist-1.3.3-cp311-cp311-win_amd64.whl (32 kB)
Collecting aiosignal>=1.1.2
Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
Collecting httpcore<0.17.0,>=0.15.0
Using cached httpcore-0.16.3-py3-none-any.whl (69 kB)
Collecting rfc3986[idna2008]<2,>=1.3
Using cached rfc3986-1.5.0-py2.py3-none-any.whl (31 kB)
Collecting sniffio
Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
Collecting contourpy>=1.0.1
Using cached contourpy-1.0.7-cp311-cp311-win_amd64.whl (162 kB)
Collecting cycler>=0.10
Using cached cycler-0.11.0-py3-none-any.whl (6.4 kB)
Collecting fonttools>=4.22.0
Using cached fonttools-4.39.3-py3-none-any.whl (1.0 MB)
Collecting kiwisolver>=1.0.1
Using cached kiwisolver-1.4.4-cp311-cp311-win_amd64.whl (55 kB)
Collecting pyparsing>=2.3.1
Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting mpmath>=0.19
Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)
Collecting svgwrite
Using cached svgwrite-1.4.3-py3-none-any.whl (67 kB)
Collecting smmap<6,>=3.0.1
Using cached smmap-5.0.0-py3-none-any.whl (24 kB)
Collecting pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0
Using cached pyrsistent-0.19.3-cp311-cp311-win_amd64.whl (62 kB)
Collecting uc-micro-py
Using cached uc_micro_py-1.0.1-py3-none-any.whl (6.2 kB)
Building wheels for collected packages: fschat, transformers
Building editable for fschat (pyproject.toml) ... done
Created wheel for fschat: filename=fschat-0.1.9-0.editable-py3-none-any.whl size=11439 sha256=d6b27e424f08488b3a12e7474070e01dc31820a0f4e71cba08defad7f66fa151
Stored in directory: C:\Users\msn\AppData\Local\Temp\pip-ephem-wheel-cache-s7ffwqci\wheels\d4\1a\90\7b409cecf8f604c048bbcdbb8ba7edea8635772646b05e33fa
Building wheel for transformers (pyproject.toml) ... done
Created wheel for transformers: filename=transformers-4.28.0.dev0-py3-none-any.whl size=6895878 sha256=74a7d2e9ec1d2e045fd74ce2fe8aa7a77540837b3452a21c9a6db66b65c3fe78
Stored in directory: C:\Users\msn\AppData\Local\Temp\pip-ephem-wheel-cache-s7ffwqci\wheels\32\4b\78\f195c684dd3a9ed21f3b39fe8f85b48df7918581b6437be143
Successfully built fschat transformers
Installing collected packages: wcwidth, sentencepiece, rfc3986, pytz, pydub, pathtools, mpmath, ffmpy, appdirs, websockets, urllib3, uc-micro-py, tzdata, typing-extensions, toolz, sympy, svgwrite, sniffio, smmap, six, setproctitle, semantic-version, regex, pyyaml, python-multipart, pyrsistent, pyparsing, pygments, psutil, protobuf, prompt-toolkit, pillow, packaging, orjson, numpy, networkx, multidict, mdurl, markupsafe, markdown2, kiwisolver, idna, h11, fsspec, frozenlist, fonttools, filelock, entrypoints, cycler, colorama, charset-normalizer, certifi, attrs, async-timeout, aiofiles, yarl, wavedrom, tqdm, sentry-sdk, requests, python-dateutil, pydantic, markdown-it-py, linkify-it-py, jsonschema, jinja2, gitdb, docker-pycreds, contourpy, click, anyio, aiosignal, uvicorn, torch, starlette, rich, pandas, mdit-py-plugins, matplotlib, huggingface-hub, httpcore, GitPython, aiohttp, wandb, transformers, httpx, fastapi, altair, accelerate, gradio, fschat
DEPRECATION: sentencepiece is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
Running setup.py install for sentencepiece ... error
error: subprocess-exited-with-error
× Running setup.py install for sentencepiece did not run successfully.
│ exit code: 1
╰─> [23 lines of output]
running install
C:\Program Files\Python311\Lib\site-packages\setuptools\command\install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-311
creating build\lib.win-amd64-cpython-311\sentencepiece
copying src\sentencepiece/__init__.py -> build\lib.win-amd64-cpython-311\sentencepiece
copying src\sentencepiece/_version.py -> build\lib.win-amd64-cpython-311\sentencepiece
copying src\sentencepiece/sentencepiece_model_pb2.py -> build\lib.win-amd64-cpython-311\sentencepiece
copying src\sentencepiece/sentencepiece_pb2.py -> build\lib.win-amd64-cpython-311\sentencepiece
running build_ext
building 'sentencepiece._sentencepiece' extension
creating build\temp.win-amd64-cpython-311
creating build\temp.win-amd64-cpython-311\Release
creating build\temp.win-amd64-cpython-311\Release\src
creating build\temp.win-amd64-cpython-311\Release\src\sentencepiece
"C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\bin\HostX86\x64\cl.exe" /c /nologo /O2 /W3 /GL /DNDEBUG /MD "-IC:\Program Files\Python311\include" "-IC:\Program Files\Python311\Include" "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Tools\MSVC\14.35.32215\include" "-IC:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\VS\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt" "-IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" /EHsc /Tpsrc/sentencepiece/sentencepiece_wrap.cxx /Fobuild\temp.win-amd64-cpython-311\Release\src/sentencepiece/sentencepiece_wrap.obj /std:c++17 /MT /I..\build\root\include
cl : Command line warning D9025 : overriding '/MD' with '/MT'
sentencepiece_wrap.cxx
src/sentencepiece/sentencepiece_wrap.cxx(2822): fatal error C1083: Cannot open include file: 'sentencepiece_processor.h': No such file or directory
error: command 'C:\\Program Files\\Microsoft Visual Studio\\2022\\Community\\VC\\Tools\\MSVC\\14.35.32215\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> sentencepiece
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
C:\chat\FastChat>
I had to downgrade to python 3.9.13 via pyenv couldn't get it to work on 3.11
there is a sentencepiece wheel for python 3.10.
I was able to build sentencepiece for python 3.11 but then ran into other issues when serving the model later. So, 3.10 may be the less troublesome way to go.
Seems it is because the Python version is too high. Please try to downgrade. Meanwhile, we should specificy the right Python versions in our toml.
