tortoise-tts icon indicating copy to clipboard operation
tortoise-tts copied to clipboard

Published 20 hours ago • verified publisher icon neonbjb

Issue with tokenizers

Open PatiB87 opened this issue 4 months ago • 2 comments

Hey, on Apple Silicon M2 Max Mac with Sonoms 14.4 I get this error:

Running rustc --crate-name tokenizers --edition=2018 tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="cached-path"' --cfg 'feature="clap"' --cfg 'feature="cli"' --cfg 'feature="default"' --cfg 'feature="dirs"' --cfg 'feature="esaxx_fast"' --cfg 'feature="http"' --cfg 'feature="indicatif"' --cfg 'feature="onig"' --cfg 'feature="progressbar"' --cfg 'feature="reqwest"' -C metadata=a838c54d75003fe3 -C extra-filename=-a838c54d75003fe3 --out-dir /private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps -L dependency=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps --extern aho_corasick=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libaho_corasick-cf7f722ec57c198c.rmeta --extern cached_path=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libcached_path-a143f921f9453d11.rmeta --extern clap=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libclap-d33a0ce1bf9623e1.rmeta --extern derive_builder=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libderive_builder-2b414fe86f3f7dae.rmeta --extern dirs=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libdirs-40dac3a47aae9a63.rmeta --extern esaxx_rs=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libesaxx_rs-34939aa2deba81ff.rmeta --extern getrandom=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libgetrandom-d27a1b9442658553.rmeta --extern indicatif=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libindicatif-c0504f481b9fa0be.rmeta --extern itertools=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libitertools-90031d8d651a7dea.rmeta --extern lazy_static=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/liblazy_static-d31bb14af56b19a1.rmeta --extern log=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/liblog-9190404a24feee53.rmeta --extern macro_rules_attribute=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libmacro_rules_attribute-04b2257e6a22bdf7.rmeta --extern monostate=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libmonostate-b60e123f4c54ab51.rmeta --extern onig=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libonig-4308de740a029f74.rmeta --extern paste=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libpaste-0d5ab52f5414cf39.dylib --extern rand=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/librand-f8add6882e4a6275.rmeta --extern rayon=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/librayon-cf05149d401ef4ba.rmeta --extern rayon_cond=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/librayon_cond-c33184e37c11d202.rmeta --extern regex=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libregex-afa2ad7903fb991a.rmeta --extern regex_syntax=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libregex_syntax-5547d7a028bf4216.rmeta --extern reqwest=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libreqwest-c185a395a99ee36b.rmeta --extern serde=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libserde-773632c5ddd7bea9.rmeta --extern serde_json=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libserde_json-1dd05001fee61fc2.rmeta --extern spm_precompiled=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libspm_precompiled-d9e9ebfcc69aabea.rmeta --extern thiserror=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libthiserror-bb5c7bf2954fac7d.rmeta --extern unicode_normalization_alignments=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libunicode_normalization_alignments-b092663a307c74ed.rmeta --extern unicode_segmentation=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libunicode_segmentation-d1383c684a17ba8d.rmeta --extern unicode_categories=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libunicode_categories-4cf6f47b5f7bd286.rmeta -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/bzip2-sys-afef6ee6e2a804f2/out/lib -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/zstd-sys-5955d8c7b8dca691/out -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/esaxx-rs-f21eeb3477fea3ce/out -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/onig_sys-32c35aba3220b1d3/outwarning: variable does not need to be mutable --> tokenizers-lib/src/models/unigram/model.rs:265:21 | 265 | let mut target_node = &mut best_path_ends_at[key_pos]; | ----^^^^^^^^^^^ | | | help: remove thismut| = note:#[warn(unused_mut)]` on by default

  warning: variable does not need to be mutable
     --> tokenizers-lib/src/models/unigram/model.rs:282:21
      |
  282 |                 let mut target_node = &mut best_path_ends_at[starts_at + mblen];
      |                     ----^^^^^^^^^^^
      |                     |
      |                     help: remove this `mut`
  
  warning: variable does not need to be mutable
     --> tokenizers-lib/src/pre_tokenizers/byte_level.rs:200:59
      |
  200 |     encoding.process_tokens_with_offsets_mut(|(i, (token, mut offsets))| {
      |                                                           ----^^^^^^^
      |                                                           |
      |                                                           help: remove this `mut`
  
  error: casting `&T` to `&mut T` is undefined behavior, even if the reference is unused, consider instead using an `UnsafeCell`
     --> tokenizers-lib/src/models/bpe/trainer.rs:526:47
      |
  522 |                     let w = &words[*i] as *const _ as *mut _;
      |                             -------------------------------- casting happend here
  ...
  526 |                         let word: &mut Word = &mut (*w);
      |                                               ^^^^^^^^^
      |
      = note: for more information, visit <https://doc.rust-lang.org/book/ch15-05-interior-mutability.html>
      = note: `#[deny(invalid_reference_casting)]` on by default
  
  warning: `tokenizers` (lib) generated 3 warnings
  error: could not compile `tokenizers` (lib) due to 1 previous error; 3 warnings emitted
  
  Caused by:
    process didn't exit successfully: `rustc --crate-name tokenizers --edition=2018 tokenizers-lib/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg 'feature="cached-path"' --cfg 'feature="clap"' --cfg 'feature="cli"' --cfg 'feature="default"' --cfg 'feature="dirs"' --cfg 'feature="esaxx_fast"' --cfg 'feature="http"' --cfg 'feature="indicatif"' --cfg 'feature="onig"' --cfg 'feature="progressbar"' --cfg 'feature="reqwest"' -C metadata=a838c54d75003fe3 -C extra-filename=-a838c54d75003fe3 --out-dir /private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps -L dependency=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps --extern aho_corasick=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libaho_corasick-cf7f722ec57c198c.rmeta --extern cached_path=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libcached_path-a143f921f9453d11.rmeta --extern clap=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libclap-d33a0ce1bf9623e1.rmeta --extern derive_builder=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libderive_builder-2b414fe86f3f7dae.rmeta --extern dirs=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libdirs-40dac3a47aae9a63.rmeta --extern esaxx_rs=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libesaxx_rs-34939aa2deba81ff.rmeta --extern getrandom=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libgetrandom-d27a1b9442658553.rmeta --extern indicatif=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libindicatif-c0504f481b9fa0be.rmeta --extern itertools=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libitertools-90031d8d651a7dea.rmeta --extern lazy_static=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/liblazy_static-d31bb14af56b19a1.rmeta --extern log=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/liblog-9190404a24feee53.rmeta --extern macro_rules_attribute=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libmacro_rules_attribute-04b2257e6a22bdf7.rmeta --extern monostate=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libmonostate-b60e123f4c54ab51.rmeta --extern onig=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libonig-4308de740a029f74.rmeta --extern paste=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libpaste-0d5ab52f5414cf39.dylib --extern rand=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/librand-f8add6882e4a6275.rmeta --extern rayon=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/librayon-cf05149d401ef4ba.rmeta --extern rayon_cond=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/librayon_cond-c33184e37c11d202.rmeta --extern regex=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libregex-afa2ad7903fb991a.rmeta --extern regex_syntax=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libregex_syntax-5547d7a028bf4216.rmeta --extern reqwest=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libreqwest-c185a395a99ee36b.rmeta --extern serde=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libserde-773632c5ddd7bea9.rmeta --extern serde_json=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libserde_json-1dd05001fee61fc2.rmeta --extern spm_precompiled=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libspm_precompiled-d9e9ebfcc69aabea.rmeta --extern thiserror=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libthiserror-bb5c7bf2954fac7d.rmeta --extern unicode_normalization_alignments=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libunicode_normalization_alignments-b092663a307c74ed.rmeta --extern unicode_segmentation=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libunicode_segmentation-d1383c684a17ba8d.rmeta --extern unicode_categories=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/deps/libunicode_categories-4cf6f47b5f7bd286.rmeta -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/bzip2-sys-afef6ee6e2a804f2/out/lib -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/zstd-sys-5955d8c7b8dca691/out -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/esaxx-rs-f21eeb3477fea3ce/out -L native=/private/var/folders/3z/hy5w4qn91919mtyn2fsjbx2r0000gp/T/pip-install-r6u5q10i/tokenizers_3248066542ee420a80ff412d926077d4/target/release/build/onig_sys-32c35aba3220b1d3/out` (exit status: 1)
  error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-312-darwin.so'` failed with code 101`

=> I cannot seem to fix this. Any help?

PatiB87 avatar Mar 16 '24 11:03 PatiB87

I found the solution mentioned here, same issue: https://github.com/neonbjb/tortoise-tts/issues/728#issuecomment-1962438240

The steps to install:

python3 -m venv .venv
source .venv/bin/activate
pip install numba inflect psutil
pip install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cpu
pip install transformers
git clone https://github.com/neonbjb/tortoise-tts.git
cd tortoise-tts
export RUSTFLAGS="-A invalid_reference_casting"
pip install .

neoneye avatar Mar 16 '24 19:03 neoneye

I had a similar problem under Windows 11. I had the latest versions of Rust and cargo installed, latest transformers and tokenizers but no luck. I tried the above instructions but they did not work. I found that using set in a virtual environment wasn't working. I had to use:

$env:RUSTFLAGS="-A invalid_reference_casting"

robarbms avatar Apr 01 '24 21:04 robarbms