[BUG] Failed to build tokenizers

[dongqqcom]

Describe the bug Failed to build tokenizers when executing the command "pip install -e .". ERROR: Failed building wheel for tokenizers Failed to build tokenizers ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects

To Reproduce Steps to reproduce the behavior:

1. pip install -e .

Expected behavior Installing the dependencies without errors.

Additional context System: macos Python: 3.6

[dongqqcom]

The same error occurs in both macos and Linux.

[zqwerty]

It seems that the newest version of tokenizer need rust installed. you can install the version we used tokenizers==0.8.0. Or install rust.

[dongqqcom]

你好，我已收到你的邮件，会尽快给你回复。