GLM-130B
GLM-130B copied to clipboard
Published
20 hours ago •
THUDM
THUDM
Addressing the Installation Bug in Building Sentencepiece
Hi team,
I am sharing a log related to an issue with the package for sentencepiece. The package appears to fail to build when CMake is not installed first. Therefore, when setting up a new conda environment or installing packages from scratch, please make sure to install CMake before attempting to install sentencepiece.
I hope this log is helpful in resolving any issues related to the installation of sentencepiece.
Here is the log when I install it from a completely new conda environment:
Building wheel for sentencepiece (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [80 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-311
creating build/lib.linux-x86_64-cpython-311/sentencepiece
copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
copying src/sentencepiece/_version.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
running build_ext
/bin/sh: 1: pkg-config: not found
Cloning into 'sentencepiece'...
Note: switching to '58f256cf6f01bb86e6fa634a5cc560de5bd1667d'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
./build_bundled.sh: 19: cmake: not found
./build_bundled.sh: 20: cmake: not found
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-install-ainsg3z2/sentencepiece_3bc691100aba437286ca44d76e09c437/setup.py", line 136, in <module>
setup(
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/__init__.py", line 108, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/dist.py", line 1221, in run_command
super().run_command(command)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 343, in run
self.run_command("build")
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/dist.py", line 1221, in run_command
super().run_command(command)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
self.run_command(cmd_name)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/dist.py", line 1221, in run_command
super().run_command(command)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 84, in run
_build_ext.run(self)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
self.build_extensions()
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
self._build_extensions_serial()
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
self.build_extension(ext)
File "/tmp/pip-install-ainsg3z2/sentencepiece_3bc691100aba437286ca44d76e09c437/setup.py", line 89, in build_extension
subprocess.check_call(['./build_bundled.sh', __version__])
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./build_bundled.sh', '0.1.97']' returned non-zero exit status 127.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for sentencepiece
Running setup.py clean for sentencepiece
Building wheel for anykeystore (setup.py) ... done
Created wheel for anykeystore: filename=anykeystore-0.2-py3-none-any.whl size=16814 sha256=7ccd544780f0b3ec4e0af12e17ac5d03a6249f681c679d6ee99ef616270f4482
Stored in directory: /home/xuyijie/.cache/pip/wheels/1a/69/29/bb8eca19356ee2b9c96124c679c64ad107880b1e5160627efb
Building wheel for pbkdf2 (setup.py) ... done
Created wheel for pbkdf2: filename=pbkdf2-1.3-py3-none-any.whl size=5081 sha256=72d4ba81ce4cee11b7a8fbaecbba732e54c167252d627cf876a6b493af305ebe
Stored in directory: /home/xuyijie/.cache/pip/wheels/69/9e/91/97e44b66d0a6a2ca5ed0fd4e1c2c082d504d926d46be0e9204
Building wheel for lit (setup.py) ... done
Created wheel for lit: filename=lit-16.0.0-py3-none-any.whl size=93586 sha256=faab572d58df366907496a09b089efecac2f5fdba2cf12b74ae2dd8c28d5bf85
Stored in directory: /home/xuyijie/.cache/pip/wheels/32/7b/f9/056928d438450844917f9ee12eebcb6c77065c77ff3bf40dc6
Successfully built apex velruse cryptacular deepspeed anykeystore pbkdf2 lit
Failed to build sentencepiece
Installing collected packages: translationstring, tokenizers, sentencepiece, pytz, py-cpuinfo, pbkdf2, ninja, mpmath, lit, hjson, dataclass_wizard, cpm_kernels, cmake, anykeystore, zope.interface, zope.deprecation, xxhash, webob, venusian, urllib3, typing-extensions, tqdm, sympy, six, regex, pyyaml, psutil, protobuf, plaster, pillow, PasteDeploy, packaging, oauthlib, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, numpy, networkx, multidict, MarkupSafe, idna, hupper, greenlet, fsspec, frozenlist, filelock, dill, defusedxml, cryptacular, charset-normalizer, certifi, attrs, async-timeout, yarl, wtforms, transaction, tensorboardX, SQLAlchemy, scipy, requests, python3-openid, python-dateutil, pydantic, pyarrow, plaster-pastedeploy, nvidia-cusolver-cu11, nvidia-cudnn-cu11, multiprocess, jinja2, aiosignal, zope.sqlalchemy, wtforms-recaptcha, responses, requests-oauthlib, repoze.sendmail, pyramid, pandas, huggingface-hub, aiohttp, velruse, transformers, pyramid_mailer, datasets, apex, triton, torch, torchvision, deepspeed, SwissArmyTransformer, icetk
Running setup.py install for sentencepiece ... error
error: subprocess-exited-with-error
× Running setup.py install for sentencepiece did not run successfully.
│ exit code: 1
╰─> [69 lines of output]
running install
/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-311
creating build/lib.linux-x86_64-cpython-311/sentencepiece
copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
copying src/sentencepiece/_version.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-cpython-311/sentencepiece
running build_ext
/bin/sh: 1: pkg-config: not found
fatal: destination path 'sentencepiece' already exists and is not an empty directory.
fatal: destination path 'sentencepiece' already exists and is not an empty directory.
./build_bundled.sh: 19: cmake: not found
./build_bundled.sh: 20: cmake: not found
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/tmp/pip-install-ainsg3z2/sentencepiece_3bc691100aba437286ca44d76e09c437/setup.py", line 136, in <module>
setup(
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/__init__.py", line 108, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/dist.py", line 1221, in run_command
super().run_command(command)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/command/install.py", line 68, in run
return orig.install.run(self)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/install.py", line 697, in run
self.run_command('build')
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/dist.py", line 1221, in run_command
super().run_command(command)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
self.run_command(cmd_name)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/dist.py", line 1221, in run_command
super().run_command(command)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 84, in run
_build_ext.run(self)
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
self.build_extensions()
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 467, in build_extensions
self._build_extensions_serial()
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 493, in _build_extensions_serial
self.build_extension(ext)
File "/tmp/pip-install-ainsg3z2/sentencepiece_3bc691100aba437286ca44d76e09c437/setup.py", line 89, in build_extension
subprocess.check_call(['./build_bundled.sh', __version__])
File "/home/data/xuyijie/anaconda3/envs/glm/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['./build_bundled.sh', '0.1.97']' returned non-zero exit status 127.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> sentencepiece
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
After running conda install cmake and rerun pip install -r requirements.txt, it is resolved.
