tortoise-tts Failed building wheel for tokenizers

Hi, I got an error after pip install. I don't know where the problem comes from!

I installed pyproject.toml and rust but I still have this error

ERROR :

                  error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
                  [end of output]
          note: This error originates from a subprocess, and is likely not a problem with pip.
          ERROR: Failed building wheel for tokenizers
  Successfully built tortoise-tts
  Failed to build tokenizers
  ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

Feb 15 '24 22:02 ks-kasra

I am getting a similar error:

error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-312-darwin.so'` failed with code 101
      [end of output]
        note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Successfully built tortoise-tts
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

It originates from here:

      error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
         --> tokenizers-lib/src/models/bpe/trainer.rs:526:47
          |
      522 |                     let w = &words[*i] as *const _ as *mut _;
          |                             -------------------------------- casting happend here
      ...
      526 |                         let word: &mut Word = &mut (*w);
          |                                               ^^^^^^^^^
          |
          = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
          = note: `#[deny(invalid_reference_casting)]` on by default
      
      warning: `tokenizers` (lib) generated 3 warnings
      error: could not compile `tokenizers` (lib) due to 1 previous error; 3 warnings emitted

Feb 19 '24 14:02 valkanesko

I got the same problem on an Apple Silicon Mac with rustc 1.76.0 installed with HomeBrew.

I found this page which linked to the follow issue in tokenizers, but that was closed as working and it mentioned rust 1.75: https://github.com/huggingface/tokenizers/issues/1359

A work around is to set the RUSTFLAGS environment variable to "-A invalid_reference_casting" before running pip install.

export RUSTFLAGS="-A invalid_reference_casting"

After I did that, I could install successfully.

Looking at that page though it appears the actual fix is to upgrade to a newer version of tokenizer.