ComfyUI sentencepiece is DEAD and needs to be removed as a dependency

Expected Behavior

Install ComfyUI on an updated (modern/2025) linux environment.

Actual Behavior

ComfyUI installation fails due to a broken dependency on sentencepiece, which is dead and no longer under development.

Steps to Reproduce

Install ComfyUI on a 2025 or late 2024 linux distribution. Installation fails because python 3.13 is the default and often the only option.

Debug Logs

Building wheels for collected packages: sentencepiece
  Building wheel for sentencepiece (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for sentencepiece (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [147 lines of output]
      /tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py:289: UserWarning: Unknown distribution option: 'test_suite'
        warnings.warn(msg)
      /tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
      !!
      
              ********************************************************************************
              Please consider removing the following classifiers in favor of a SPDX license expression:
      
              License :: OSI Approved :: Apache Software License
      
              See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
              ********************************************************************************
      
      !!
        self._finalize_license_expression()
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-313/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-cpython-313/sentencepiece
      copying src/sentencepiece/_version.py -> build/lib.linux-x86_64-cpython-313/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-cpython-313/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-cpython-313/sentencepiece
      running build_ext
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      Package 'sentencepiece' not found
      CMake Deprecation Warning at CMakeLists.txt:15 (cmake_minimum_required):
        Compatibility with CMake < 3.10 will be removed from a future version of
        CMake.
      
        Update the VERSION argument <min> value.  Or, use the <min>...<max> syntax
        to tell CMake that the project requires at least <min> but has been updated
        to work with policies introduced by <max> or earlier.
      
      
      -- VERSION: 0.2.0
      -- The C compiler identification is GNU 15.0.1
      -- The CXX compiler identification is unknown
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/sbin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      CMake Error at CMakeLists.txt:23 (project):
        No CMAKE_CXX_COMPILER could be found.
      
        Tell CMake where to find the compiler by setting either the environment
        variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
        to the compiler, or to the compiler name if it is in the PATH.
      
      
      -- Configuring incomplete, errors occurred!
      gmake: Makefile: No such file or directory
      gmake: *** No rule to make target 'Makefile'.  Stop.
      Traceback (most recent call last):
        File "/var/home/ricperry1/ComfyUI/venv313/lib64/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
          ~~~~^^
        File "/var/home/ricperry1/ComfyUI/venv313/lib64/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ~~~~^^^^^^^^^^^^^^^^^^^^^^^^
        File "/var/home/ricperry1/ComfyUI/venv313/lib64/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                              metadata_directory)
                                              ^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 435, in build_wheel
          return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 423, in _build
          return self._build_with_temp_dir(
                 ~~~~~~~~~~~~~~~~~~~~~~~~~^
              cmd,
              ^^^^
          ...<3 lines>...
              self._arbitrary_args(config_settings),
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          )
          ^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 404, in _build_with_temp_dir
          self.run_setup()
          ~~~~~~~~~~~~~~^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 512, in run_setup
          super().run_setup(setup_script=setup_script)
          ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/build_meta.py", line 317, in run_setup
          exec(code, locals())
          ~~~~^^^^^^^^^^^^^^^^
        File "<string>", line 169, in <module>
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/__init__.py", line 117, in setup
          return distutils.core.setup(**attrs)
                 ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/core.py", line 186, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/core.py", line 202, in run_commands
          dist.run_commands()
          ~~~~~~~~~~~~~~~~~^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1002, in run_commands
          self.run_command(cmd)
          ~~~~~~~~~~~~~~~~^^^^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
          ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
          ~~~~~~~~~~~^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/command/bdist_wheel.py", line 370, in run
          self.run_command("build")
          ~~~~~~~~~~~~~~~~^^^^^^^^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
          ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
          ~~~~~~~~~~~^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/command/build.py", line 135, in run
          self.run_command(cmd_name)
          ~~~~~~~~~~~~~~~~^^^^^^^^^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/cmd.py", line 357, in run_command
          self.distribution.run_command(command)
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/dist.py", line 1104, in run_command
          super().run_command(command)
          ~~~~~~~~~~~~~~~~~~~^^^^^^^^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py", line 1021, in run_command
          cmd_obj.run()
          ~~~~~~~~~~~^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/command/build_ext.py", line 99, in run
          _build_ext.run(self)
          ~~~~~~~~~~~~~~^^^^^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 368, in run
          self.build_extensions()
          ~~~~~~~~~~~~~~~~~~~~~^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 484, in build_extensions
          self._build_extensions_serial()
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
        File "/tmp/pip-build-env-xx81toe5/overlay/lib/python3.13/site-packages/setuptools/_distutils/command/build_ext.py", line 510, in _build_extensions_serial
          self.build_extension(ext)
          ~~~~~~~~~~~~~~~~~~~~^^^^^
        File "<string>", line 87, in build_extension
        File "/usr/lib64/python3.13/subprocess.py", line 421, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['./build_bundled.sh', '0.2.0']' returned non-zero exit status 2.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for sentencepiece
Failed to build sentencepiece

[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: pip install --upgrade pip
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (sentencepiece)

Other

As the title says, sentencepiece is no longer being updated, and is falling behind. ComfyUI devs need to find an alternative and replace it in the dependencies. Recommend using tokenizers instead. The issue was closed (https://github.com/comfyanonymous/ComfyUI/issues/6169) but should not have been. This is a real issue, and the devs need to sort it out.

🔷 Hugging Face tokenizers Benefits:

Performance: Extremely fast due to Rust backend with Python bindings.

Flexibility: Offers a wide range of tokenization algorithms (BPE, WordPiece, Unigram).

Customizability: Fine-grained control over preprocessing, postprocessing, normalization, etc.

Integration: Native support in Hugging Face Transformers ecosystem.

Python 3.13 Support: Actively maintained and compatible with latest Python versions.

Drawbacks:

Complexity: API is powerful but may be overwhelming for simple use cases.

Lesser Standardization: Not as widely used in traditional NLP pipelines as SentencePiece.

Binary Files: Requires compilation or pre-built wheels (minor concern, but relevant in limited environments).

🔷 Google sentencepiece Benefits:

Language-Agnostic: Designed to work on raw text without language-specific rules (especially useful for multilingual applications).

Simplicity: Easy CLI and Python API for training and encoding/decoding.

Proven Track Record: Used in production at Google, supports models like T5, mT5, and XLNet.

Unigram Model: Provides robust subword segmentation based on a probabilistic model.

Drawbacks:

Performance: Slower than tokenizers, especially in large-scale or real-time scenarios.

Python 3.13 Incompatibility: As of now, it does not officially support Python 3.13.

Less Customizable: Harder to fine-tune preprocessing or add custom behavior compared to tokenizers.

Apr 23 '25 01:04 ricperry

The huggingface libraries actually depend on sentencepiece so they are not a valid alternative.

Sentencepiece compiles fine on python 3.13, if they don't have official wheels by the time we start shipping ComfyUI with python 3.13 we will compile them ourselves.

Apr 23 '25 03:04 comfyanonymous

Hi,

Similar issue If i use cmake 4.0.0. Works fine with cmake 3.31.5.

Apr 29 '25 15:04 Inglebard

Sentencepiece is also failing on GCC 15. I was able to manually install from source with a few modifications though

May 10 '25 19:05 gptlang

completely broken on arch

May 17 '25 13:05 jpVm5jYYRE1VIKL

completely broken on arch

export CMAKE_POLICY_VERSION_MINIMUM=3.5
export CXXFLAGS="-include cstdint"

via: https://stackoverflow.com/questions/79534856/cannot-build-cmake-project-because-compatibility-with-cmake-3-5-has-been-remo https://github.com/google/sentencepiece/issues/1108

May 17 '25 23:05 0x4E69676874466F78

completely broken on arch
export CMAKE_POLICY_VERSION_MINIMUM=3.5
export CXXFLAGS="-include cstdint"
via: https://stackoverflow.com/questions/79534856/cannot-build-cmake-project-because-compatibility-with-cmake-3-5-has-been-remo google/sentencepiece#1108

Can confirm this worked.

Is there a way for the devs to prioritize a fix to this?

May 18 '25 03:05 lupusvir

In your log the build failed because you don't have a compiler

CMake Error at CMakeLists.txt:23 (project): No CMAKE_CXX_COMPILER could be found. Tell CMake where to find the compiler by setting either the environment variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path to the compiler, or to the compiler name if it is in the PATH.

May 29 '25 21:05 Rivers47

For posterity for anyone else who runs into this issue:

# Install cmake $ sudo apt install cmake

# Re-run the installer command that failed, or run the following to install sentencepiece manually $ pip install sentencepiece

Jun 25 '25 04:06 dugopark

For anyone encountering this issue on Fedora 42, what fixed it for me was following this guide on setting up multiple Python versions.

https://linuxconfig.org/how-to-switch-between-python-versions-on-fedora-linux

The guide talks about installing 3.9 along side 3.12, but the concept is the same. Fedora 42 ships with 3.13 which sentencepiece is incompatible with. So you need to install 3.12 along side 3.13.

And the important part here, is once you've done this and set 3.12 as the default for the python command, when following the ComfyUI Linux installation tutorial make sure you don't use the python3 -m venv comfy-env command, but instead use python -m venv comfy-env.

If you do a python3 --version you'll still get 3.13. But python --version will return 3.12, your alternate installed version. Install works fine with Python 3.12, but fails on installing sentencepiece on 3.13.

Jun 28 '25 05:06 canuckbrian

For posterity for anyone else who runs into this issue:

Install cmake $ sudo apt install cmake

Re-run the installer command that failed, or run the following to install sentencepiece manually $ pip install sentencepiece

This does not help - after doing the "sudo apt install cmake", the "pip install sentencepiece" still fails with the same error as before.

But the ComfyUI that is integrated in SwarmUI works successful for me, so I just use that. How is SwarmUI making it work?

Jul 07 '25 02:07 driqeks

For anyone encountering this issue on Fedora 42, what fixed it for me was following this guide on setting up multiple Python versions.

https://linuxconfig.org/how-to-switch-between-python-versions-on-fedora-linux

The guide talks about installing 3.9 along side 3.12, but the concept is the same. Fedora 42 ships with 3.13 which sentencepiece is incompatible with. So you need to install 3.12 along side 3.13.

And the important part here, is once you've done this and set 3.12 as the default for the python command, when following the ComfyUI Linux installation tutorial make sure you don't use the python3 -m venv comfy-env command, but instead use python -m venv comfy-env.

If you do a python3 --version you'll still get 3.13. But python --version will return 3.12, your alternate installed version. Install works fine with Python 3.12, but fails on installing sentencepiece on 3.13.

Jus watned to say that this is the proper fix for Fedora 42 as of 7/7/2025.

Jul 07 '25 19:07 soten355

For Arch Linux users, Python is always at the latest version (i.e., 3.13 or newer), so you'll need to downgrade to 3.11. This can be done without contaminating the system by using pyenv.

System Info

$ python --version
Python 3.13.5
$ cat /proc/version 
Linux version 6.15.6-arch1-1 (linux@archlinux) (gcc (GCC) 15.1.1 20250425, GNU ld (GNU Binutils) 2.44.0) #1 SMP PREEMPT_DYNAMIC Thu, 10 Jul 2025 17:10:18 +0000

ComfyUI Installation

Install ComfyUI on Arch Linux using pyenv, pip, and a virtual environment as follows:

$ pacman -Syu pyenv
$ pyenv --version
pyenv 2.6.4
$ pyenv install 3.11.9
$ cd $HOME/your/project/path
$ git clone https://github.com/comfyanonymous/ComfyUI.git cui
$ cd cui
$ eval "$(pyenv init -)"
$ pyenv local 3.11.9
$ python --version
Python 3.11.9
$ pip install -r requirements.txt
$ python main.py

Browse to http://127.0.0.1:8188/.

Continue

Copy (or rename) extra_model_paths.yaml.example to extra_model_paths.yaml.

Update that configuration file with fully qualified paths to where you want to put the model files. For example:

comfyui:
    checkpoints: /mnt/models/checkpoints/
    vae: /mnt/models/vae/
    text_encoders: /mnt/models/text_encoders/
    diffusion_models: /mnt/models/diffusion_models/

Download https://huggingface.co/Comfy-Org/stable-diffusion-v1-5-archive/blob/main/v1-5-pruned-emaonly-fp16.safetensors into the path to models/checkpoints/
From the ComfyUI Examples, download:
- https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/blob/main/split_files/text_encoders/umt5_xxl_fp16.safetensors into the path to models/text_encoders/
- https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/blob/main/split_files/vae/wan_2.1_vae.safetensors into the path to models/vae/
- https://huggingface.co/Comfy-Org/Wan_2.1_ComfyUI_repackaged/blob/main/split_files/diffusion_models/wan2.1_t2v_1.3B_fp16.safetensors into the path to models/diffusion_models/
Restart ComfyUI.

Jul 12 '25 01:07 DarkTyger

Progress on this. A better solution than what I posted previously. Instead of installing Python 3.12 along side Python 3.13.

There are two types of errors that can occur when trying to install the requirements for ComfyUI. Both have the same solution.

Install of sentencepiece fails because "cmake" is not found
Install of sentencepiece fails because the version of "cmake" is too new

If you already have cmake installed uninstall it using PIP.

Install an older version of cmake using PIP. I found this version works:

pip install cmake==3.27

After installing this and then re-installing the requirements, sentencepiece is built and install properly.

Source: https://github.com/google/sentencepiece/issues/932

Jul 24 '25 17:07 canuckbrian

im running on fedora 42, python 3.13.5 and ran then pip install cmake==3.27

but i still get the error

  × Building wheel for sentencepiece (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [127 lines of output]
      /tmp/pip-build-env-y8szvfsr/overlay/lib/python3.13/site-packages/setuptools/_distutils/dist.py:289: UserWarning: Unknown distribution option: 'test_suite'
        warnings.warn(msg)
      /tmp/pip-build-env-y8szvfsr/overlay/lib/python3.13/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
      !!
      
              ********************************************************************************
              Please consider removing the following classifiers in favor of a SPDX license expression:
      
              License :: OSI Approved :: Apache Software License
      
              See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
              ********************************************************************************
      
      !!
        self._finalize_license_expression()
      running bdist_wheel
      running build
      running build_py
      creating build/lib.linux-x86_64-cpython-313/sentencepiece
      copying src/sentencepiece/__init__.py -> build/lib.linux-x86_64-cpython-313/sentencepiece
      copying src/sentencepiece/_version.py -> build/lib.linux-x86_64-cpython-313/sentencepiece
      copying src/sentencepiece/sentencepiece_model_pb2.py -> build/lib.linux-x86_64-cpython-313/sentencepiece
      copying src/sentencepiece/sentencepiece_pb2.py -> build/lib.linux-x86_64-cpython-313/sentencepiece
      running build_ext
      Package sentencepiece was not found in the pkg-config search path.
      Perhaps you should add the directory containing `sentencepiece.pc'
      to the PKG_CONFIG_PATH environment variable
      Package 'sentencepiece' not found
      Traceback (most recent call last):
        File "/home/ramonskie/workspace/ComfyUI/comfy-env/bin/cmake", line 5, in <module>
          from cmake import cmake
      ModuleNotFoundError: No module named 'cmake'
      Traceback (most recent call last):
        File "/home/ramonskie/workspace/ComfyUI/comfy-env/bin/cmake", line 5, in <module>
          from cmake import cmake
      ModuleNotFoundError: No module named 'cmake'

any other suggestions @canuckbrian ??

Jul 29 '25 19:07 ramonskie

If you already have cmake installed uninstall it using PIP.

Install an older version of cmake using PIP. I found this version works:

pip install cmake==3.27

After installing this and then re-installing the requirements, sentencepiece is built and install properly.

I have tried that now. Enabled venv, uninstalled cmake, installed cmake==3.27, installed requirements.txt. Does not work unfortunately. Still results in the error "ModuleNotFoundError: No module named 'cmake'":

Jul 29 '25 23:07 driqeks

okay i solved it for fedora: STEPS:

sudo dnf install python3.12.x86_64
python3.12 -m venv comfy-env 
source comfy-env/bin/activate
pip install comfy-cli
comfy install

Jul 31 '25 10:07 ramonskie

okay i solved it for fedora: STEPS:

sudo dnf install python3.12.x86_64
python3.12 -m venv comfy-env 
source comfy-env/bin/activate
pip install comfy-cli
comfy install

Same for me, this also worked for me in fedora 😁

Aug 05 '25 16:08 ZorroDKK

Jumping on here because this thread solved my issue, if you're a Mac user looking to do this, you want Pip3.12, and can specify the correct version with: pip3.12 install -r requirements.txt

Aug 10 '25 10:08 notamedicaldoctor