sentencepiece icon indicating copy to clipboard operation
sentencepiece copied to clipboard

[Question] Could not install via pip or build from source code as a model of python

Open Mikachu2333 opened this issue 11 months ago • 2 comments

What I met

  1. Could not install this package via pip or build from source code
  2. I want to use the model HanLP in python and it need this

SYS infos

  1. Win11 24h2 Pro (26100.2605, Official iso, amd x86_64, Lenovo ThinkBook 14s yoga)
  2. Msys2 installed
  3. Python 3.12 installed
  4. MSVC toolchain (Visual Studio) installed
    • .NET desktop development
    • Desktop development with C++
    • Windows application development
  5. Cmake and make installed
  6. GNU target installed (mingw-w64-ucrt-x86_64)
$PATH$

%SYSTEMROOT%\System32\OpenSSH; %SYSTEMROOT%\System32\WindowsPowerShell\v1.0; %SystemRoot%; %SystemRoot%\System32\Wbem; %SystemRoot%\system32; C:\Program Files\Eclipse Adoptium\jdk-21.0.4.7-hotspot\bin; C:\Program Files\Graphviz\bin; C:\Program Files\OpenSSL-Win64\bin; C:\Program Files\PowerShell\7; C:\Program Files\dotnet; C:\msys64; C:\msys64\usr\bin; C:\msys64\ucrt64\bin; C:\msys64\ucrt\include; D:\Common_Process; D:\Git\cmd; D:\Microsoft VS Code\bin; D:\Windows Kits\10\Windows Performance Toolkit\

MSYS2 softwares

base 2022.06-1 base-devel 2024.11-1 bash 5.2.037-1 bash-completion 2.16.0-1 binutils 2.43.1-1 bison 3.8.2-5 brotli 1.1.0-1 bsdtar 3.7.7-1 bzip2 1.0.8-4 ca-certificates 20240203-2 cmake 3.31.3-1 coreutils 8.32-5 cppdap 1.58.0a-3 curl 8.11.1-1 dash 0.5.12-1 db 6.2.32-5 diffstat 1.67-1 diffutils 3.10-1 dos2unix 7.5.2-1 file 5.46-1 filesystem 2024.12.18-1 findutils 4.10.0-2 flex 2.6.4-3 gawk 5.3.1-1 gcc-libs 13.3.0-1 gdbm 1.24-1 getent 2.18.90-5 gettext 0.22.4-1 git 2.47.1-1 gmp 6.3.0-1 gnupg 2.4.7-1 grep 1~3.0-6 gzip 1.13-1 heimdal 7.8.0-5 heimdal-libs 7.8.0-5 inetutils 2.5-2 info 7.1.1-1 isl 0.27-1 jsoncpp 1.9.6-2 less 668-1 libarchive 3.7.7-1 libargp 20241207-1 libasprintf 0.22.4-1 libassuan 2.5.7-1 libbz2 1.0.8-4 libcbor 0.11.0-1 libcurl 8.11.1-1 libdb 6.2.32-5 libedit 20240808_3.1-1 libexpat 2.6.4-1 libffi 3.4.6-1 libfido2 1.15.0-1 libgcrypt 1.11.0-1 libgdbm 1.24-1 libgettextpo 0.22.4-1 libgnutls 3.8.8-1 libgpg-error 1.51-1 libhogweed 3.10.1-1 libiconv 1.18-1 libidn2 2.3.7-1 libintl 0.22.4-1 libksba 1.6.7-1 liblz4 1.10.0-1 liblzma 5.6.3-1 libnettle 3.10.1-1 libnghttp2 1.64.0-1 libnpth 1.8-1 libopenssl 3.4.0-1 libp11-kit 0.25.5-2 libpcre 8.45-4 libpcre2_8 10.44-1 libpsl 0.21.5-2 libreadline 8.2.013-1 librhash 1.4.4-1 libsqlite 3.46.1-1 libssh2 1.11.1-1 libtasn1 4.19.0-1 libunistring 1.2-1 libutil-linux 2.35.2-4 libuv 1.49.2-1 libxcrypt 4.4.37-1 libzstd 1.5.6-1 m4 1.4.19-2 make 4.4.1-2 mingw-w64-cross-common-binutils 2.43.1-2 mingw-w64-cross-mingw64-binutils 2.43.1-2 mingw-w64-cross-mingw64-crt 12.0.0.r250.gc6bf4bdf6-2 mingw-w64-cross-mingw64-gcc 14.2.0-3 mingw-w64-cross-mingw64-headers 12.0.0.r250.gc6bf4bdf6-1 mingw-w64-cross-mingw64-windows-default-manifest 6.4-5 mingw-w64-cross-mingw64-winpthreads 12.0.0.r250.gc6bf4bdf6-1 mingw-w64-ucrt-x86_64-binutils 2.43.1-1 mingw-w64-ucrt-x86_64-bzip2 1.0.8-3 mingw-w64-ucrt-x86_64-crt-git 12.0.0.r459.g63f3f2846-1 mingw-w64-ucrt-x86_64-expat 2.6.4-1 mingw-w64-ucrt-x86_64-gcc 14.2.0-2 mingw-w64-ucrt-x86_64-gcc-libgfortran 14.2.0-2 mingw-w64-ucrt-x86_64-gcc-libs 14.2.0-2 mingw-w64-ucrt-x86_64-gdb 15.2-2 mingw-w64-ucrt-x86_64-gdb-multiarch 15.2-2 mingw-w64-ucrt-x86_64-gettext-runtime 0.23.1-1 mingw-w64-ucrt-x86_64-gmp 6.3.0-2 mingw-w64-ucrt-x86_64-gperf 3.1-2 mingw-w64-ucrt-x86_64-headers-git 12.0.0.r459.g63f3f2846-3 mingw-w64-ucrt-x86_64-isl 0.27-1 mingw-w64-ucrt-x86_64-libffi 3.4.6-1 mingw-w64-ucrt-x86_64-libiconv 1.18-1 mingw-w64-ucrt-x86_64-libmangle-git 12.0.0.r459.g63f3f2846-1 mingw-w64-ucrt-x86_64-libsystre 1.0.1-6 mingw-w64-ucrt-x86_64-libtre 0.9.0-1 mingw-w64-ucrt-x86_64-libwinpthread-git 12.0.0.r459.g63f3f2846-1 mingw-w64-ucrt-x86_64-make 4.4.1-3 mingw-w64-ucrt-x86_64-mpc 1.3.1-2 mingw-w64-ucrt-x86_64-mpdecimal 4.0.0-1 mingw-w64-ucrt-x86_64-mpfr 4.2.1-2 mingw-w64-ucrt-x86_64-ncurses 6.5.20241228-2 mingw-w64-ucrt-x86_64-openblas 0.3.28-2 mingw-w64-ucrt-x86_64-opencc 1.1.9-1 mingw-w64-ucrt-x86_64-openssl 3.4.0-1 mingw-w64-ucrt-x86_64-pkgconf 1~2.3.0-1 mingw-w64-ucrt-x86_64-python 3.12.8-1 mingw-w64-ucrt-x86_64-python-argcomplete 3.5.3-1 mingw-w64-ucrt-x86_64-python-arrow 1.3.0-3 mingw-w64-ucrt-x86_64-python-babel 2.16.0-2 mingw-w64-ucrt-x86_64-python-certifi 2024.8.30-2 mingw-w64-ucrt-x86_64-python-charset-normalizer 3.4.1-1 mingw-w64-ucrt-x86_64-python-click 8.1.7-2 mingw-w64-ucrt-x86_64-python-colorama 0.4.6-3 mingw-w64-ucrt-x86_64-python-dateutil 2.9.0-2 mingw-w64-ucrt-x86_64-python-distlib 0.3.8-3 mingw-w64-ucrt-x86_64-python-docutils 0.20.1-3 mingw-w64-ucrt-x86_64-python-idna 3.8-2 mingw-w64-ucrt-x86_64-python-imagesize 1.4.1-3 mingw-w64-ucrt-x86_64-python-jinja 3.1.5-1 mingw-w64-ucrt-x86_64-python-markupsafe 2.1.5-3 mingw-w64-ucrt-x86_64-python-numpy 2.2.0-1 mingw-w64-ucrt-x86_64-python-packaging 24.2-1 mingw-w64-ucrt-x86_64-python-pandas 2.2.2-4 mingw-w64-ucrt-x86_64-python-pip 24.3.1-1 mingw-w64-ucrt-x86_64-python-pipx 1.7.1-2 mingw-w64-ucrt-x86_64-python-platformdirs 4.3.6-2 mingw-w64-ucrt-x86_64-python-pygments 2.18.0-2 mingw-w64-ucrt-x86_64-python-pytz 2024.2-2 mingw-w64-ucrt-x86_64-python-requests 2.32.3-2 mingw-w64-ucrt-x86_64-python-setuptools 75.6.0-1 mingw-w64-ucrt-x86_64-python-six 1.16.0-6 mingw-w64-ucrt-x86_64-python-snowballstemmer 2.2.0-4 mingw-w64-ucrt-x86_64-python-sphinx 8.1.3-2 mingw-w64-ucrt-x86_64-python-sphinx-alabaster-theme 1.0.0-2 mingw-w64-ucrt-x86_64-python-sphinxcontrib-applehelp 2.0.0-2 mingw-w64-ucrt-x86_64-python-sphinxcontrib-devhelp 2.0.0-2 mingw-w64-ucrt-x86_64-python-sphinxcontrib-htmlhelp 2.1.0-2 mingw-w64-ucrt-x86_64-python-sphinxcontrib-jsmath 1.0.1-5 mingw-w64-ucrt-x86_64-python-sphinxcontrib-moderncmakedomain 3.29.0-2 mingw-w64-ucrt-x86_64-python-sphinxcontrib-qthelp 2.0.0-2 mingw-w64-ucrt-x86_64-python-sphinxcontrib-serializinghtml 2.0.0-2 mingw-w64-ucrt-x86_64-python-types-python-dateutil 2.9.0.20241003-1 mingw-w64-ucrt-x86_64-python-tzdata 2024.2-2 mingw-w64-ucrt-x86_64-python-urllib3 1.26.20-2 mingw-w64-ucrt-x86_64-python-userpath 1.9.2-2 mingw-w64-ucrt-x86_64-readline 8.2.013-1 mingw-w64-ucrt-x86_64-sqlite3 3.47.2-1 mingw-w64-ucrt-x86_64-tcl 8.6.13-1 mingw-w64-ucrt-x86_64-termcap 1.3.1-7 mingw-w64-ucrt-x86_64-tk 8.6.13-1 mingw-w64-ucrt-x86_64-tools-git 12.0.0.r459.g63f3f2846-1 mingw-w64-ucrt-x86_64-tzdata 2024b-1 mingw-w64-ucrt-x86_64-windows-default-manifest 6.4-4 mingw-w64-ucrt-x86_64-winpthreads-git 12.0.0.r459.g63f3f2846-1 mingw-w64-ucrt-x86_64-winstorecompat-git 12.0.0.r459.g63f3f2846-1 mingw-w64-ucrt-x86_64-xxhash 0.8.2-2 mingw-w64-ucrt-x86_64-xz 5.6.3-3 mingw-w64-ucrt-x86_64-zlib 1.3.1-1 mingw-w64-ucrt-x86_64-zstd 1.5.6-2 mingw-w64-x86_64-binutils 2.43.1-1 mingw-w64-x86_64-bzip2 1.0.8-3 mingw-w64-x86_64-crt-git 12.0.0.r459.g63f3f2846-1 mingw-w64-x86_64-expat 2.6.4-1 mingw-w64-x86_64-gcc 14.2.0-2 mingw-w64-x86_64-gcc-libs 14.2.0-2 mingw-w64-x86_64-gcc-rust 14.2.0-2 mingw-w64-x86_64-gettext-runtime 0.23.1-1 mingw-w64-x86_64-gmp 6.3.0-2 mingw-w64-x86_64-headers-git 12.0.0.r459.g63f3f2846-3 mingw-w64-x86_64-isl 0.27-1 mingw-w64-x86_64-libffi 3.4.6-1 mingw-w64-x86_64-libiconv 1.18-1 mingw-w64-x86_64-libsystre 1.0.1-6 mingw-w64-x86_64-libtre 0.9.0-1 mingw-w64-x86_64-libwinpthread-git 12.0.0.r459.g63f3f2846-1 mingw-w64-x86_64-mpc 1.3.1-2 mingw-w64-x86_64-mpdecimal 4.0.0-1 mingw-w64-x86_64-mpfr 4.2.1-2 mingw-w64-x86_64-ncurses 6.5.20241228-2 mingw-w64-x86_64-openssl 3.4.0-1 mingw-w64-x86_64-python 3.12.8-1 mingw-w64-x86_64-python-regex 2024.11.6-1 mingw-w64-x86_64-readline 8.2.013-1 mingw-w64-x86_64-sqlite3 3.47.2-1 mingw-w64-x86_64-tcl 8.6.13-1 mingw-w64-x86_64-termcap 1.3.1-7 mingw-w64-x86_64-tk 8.6.13-1 mingw-w64-x86_64-tzdata 2024b-1 mingw-w64-x86_64-windows-default-manifest 6.4-4 mingw-w64-x86_64-winpthreads-git 12.0.0.r459.g63f3f2846-1 mingw-w64-x86_64-xz 5.6.3-3 mingw-w64-x86_64-zlib 1.3.1-1 mingw-w64-x86_64-zstd 1.5.6-2 mintty 1~3.7.7-1 mpc 1.3.1-1 mpdecimal 4.0.0-1 mpfr 4.2.1-1 msys2-keyring 1~20241007-1 msys2-launcher 1.5-3 msys2-runtime 3.5.4-8 nano 8.2-1 nasm 2.16.03-1 ncurses 6.5.20240831-2 nettle 3.10.1-1 openssh 9.9p1-1 openssl 3.4.0-1 p11-kit 0.25.5-2 pacman 6.1.0-10 pacman-contrib 1.10.6-1 pacman-mirrors 20241217-1 patch 2.7.6-2 perl 5.38.2-2 perl-Authen-SASL 2.1700-1 perl-Clone 0.47-1 perl-Convert-BinHex 1.125-2 perl-Encode-Locale 1.05-2 perl-Error 0.17029-2 perl-File-Listing 6.16-1 perl-HTML-Parser 3.83-1 perl-HTML-Tagset 3.24-1 perl-HTTP-Cookies 6.11-1 perl-HTTP-Daemon 6.16-1 perl-HTTP-Date 6.06-1 perl-HTTP-Message 7.00-1 perl-HTTP-Negotiate 6.01-3 perl-IO-HTML 1.004-2 perl-IO-Socket-SSL 2.089-1 perl-IO-Stringy 2.113-2 perl-LWP-MediaTypes 6.04-2 perl-MIME-tools 5.515-1 perl-MailTools 2.22-1 perl-Net-HTTP 6.23-1 perl-Net-SMTP-SSL 1.04-2 perl-Net-SSLeay 1.94-1 perl-TermReadKey 2.38-6 perl-TimeDate 2.33-2 perl-Try-Tiny 0.32-1 perl-URI 5.31-1 perl-WWW-RobotRules 6.02-3 perl-http-cookiejar 0.014-1 perl-libwww 6.77-1 pinentry 1.3.1-1 pkgconf 2.3.0-1 python 3.12.8-1 python-setuptools 75.2.0-1 rebase 4.5.0-4 sed 4.9-1 tar 1.35-2 texinfo 7.1.1-1 texinfo-tex 7.1.1-1 time 1.9-3 tzcode 2024a-1 util-linux 2.35.2-4 wget 1.25.0-1 which 2.21-4 xz 5.6.3-1 zlib 1.3.1-1 zstd 1.5.6-1

Python list

Package Version alabaster 1.0.0 argcomplete 3.5.3 arrow 1.3.0 babel 2.16.0 certifi 2024.8.30 charset-normalizer 3.4.1 click 8.1.7 cnlunar 0.2.0 colorama 0.4.6 distlib 0.3.8 docutils 0.20.1 hanlp_common 0.0.22 hanlp_restful 0.0.24 idna 3.8 imagesize 1.4.1 Jinja2 3.1.5 MarkupSafe 2.1.5 numpy 2.2.0 packaging 24.2 pandas 2.2.2 phrasetree 0.0.9 pip 24.3.1 pipx 1.7.1 platformdirs 4.3.6 Pygments 2.18.0 python-dateutil 2.9.0 pytz 2024.2 requests 2.32.3 setuptools 75.6.0 six 1.16.0 snowballstemmer 2.2.0 Sphinx 8.1.3 sphinxcontrib-applehelp 2.0.0 sphinxcontrib-devhelp 2.0.0 sphinxcontrib-htmlhelp 2.1.0 sphinxcontrib-jsmath 1.0.1 sphinxcontrib-moderncmakedomain 3.29.0 sphinxcontrib-qthelp 2.0.0 sphinxcontrib-serializinghtml 2.0.0 types-python-dateutil 2.9.0.20241003 tzdata 2024.2 urllib3 1.26.20 userpath 1.9.2 wheel 0.45.1

Error info

pip

Collecting sentencepiece==0.2.0
  Using cached https://mirrors.ustc.edu.cn/pypi/packages/c9/d2/b9c7ca067c26d8ff085d252c89b5f69609ca93fb85a00ede95f4857865d4/sentencepiece-0.2.0.tar.gz (2.6 MB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      CMake Deprecation Warning at CMakeLists.txt:15 (cmake_minimum_required):
        Compatibility with CMake < 3.10 will be removed from a future version of
        CMake.

        Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
        to tell CMake that the project requires at least <min> but has been updated
        to work with policies introduced by <max> or earlier.


      -- VERSION: 0.2.0
      CMake Error at CMakeLists.txt:23 (project):
        Generator

          Unix Makefiles

        does not support platform specification, but platform

          x64

        was specified.


      CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
      CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
      -- Configuring incomplete, errors occurred!
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:/msys64/tmp/pip-install-4gp49ze2/sentencepiece_bc98403eeed743409f481dd760c49335/setup.py", line 128, in <module>
          subprocess.check_call([
        File "C:/msys64/ucrt64/lib/python3.12/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['cmake', 'sentencepiece', '-A', 'x64', '-B', 'build', '-DSPM_ENABLE_SHARED=OFF', '-DCMAKE_INSTALL_PREFIX=build\\root']' returned non-zero exit status 1.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

build from source (MSVC)

PS D:\Temp\vcpkg> ./vcpkg install sentencepiece
Computing installation plan...
The following packages will be built and installed:
  * abseil:[email protected]
  * protobuf:[email protected]
    sentencepiece:[email protected]#1
  * utf8-range:[email protected]
  * vcpkg-cmake:x64-windows@2024-04-23
  * vcpkg-cmake-config:x64-windows@2024-05-23
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet x64-windows...
error: while detecting compiler information:
The log file content at "D:\Temp\vcpkg\buildtrees\detect_compiler\stdout-x64-windows.log" is:
CMake Error at D:/Temp/vcpkg/scripts/ports.cmake:143 (message):
  Cannot find port:

    Directory does not exist: D;/Temp/vcpkg/scripts/detect_compiler


error: vcpkg was unable to detect the active compiler's information. See above for the CMake failure output.

build from source (GNU)

CMake Deprecation Warning at CMakeLists.txt:15 (cmake_minimum_required):
  Compatibility with CMake < 3.10 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
  to tell CMake that the project requires at least <min> but has been updated
  to work with policies introduced by <max> or earlier.


-- VERSION: 0.2.1
-- The CXX compiler identification is unknown
CMake Error at CMakeLists.txt:23 (project):
  The CMAKE_CXX_COMPILER:

    C

  is not a full path and was not found in the PATH.

  Tell CMake where to find the compiler by setting either the environment
  variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
  to the compiler, or to the compiler name if it is in the PATH.


-- Configuring incomplete, errors occurred!
make: *** No targets specified and no makefile found.  Stop.

What I've tried

  1. I guess this is due to the inability to find the specified compiler, so I specified the gcc.exe and g++.exe of the GNU target (UCRT64) as the via through environment variables, but it still reported this error.
  2. I've also tried cl.exe but nothing happened
  3. I explore these issues: #927 #950 #987 #1060 #1070 #990 #1025 #
  4. I tried to use cmake .. -DCMAKE_CXX_COMPILER=C:/msys2/usr/bin/g++.exe -DCMAKE_C_COMPILER=C:/msys2/usr/bin/gcc.exe to fix error but nothing happened

What I want

Install this project as python model successfully

Mikachu2333 avatar Jan 08 '25 09:01 Mikachu2333

I installed successfully on WSL(debian) but still failed on Windows

Mikachu2333 avatar Jan 08 '25 09:01 Mikachu2333

i'm trying to install sentencepiece on mac and getting the issue with wheel building.

dsnehitha avatar Feb 05 '25 22:02 dsnehitha

Closed by #1120

crusaderky avatar Jun 27 '25 13:06 crusaderky