lightllm icon indicating copy to clipboard operation
lightllm copied to clipboard

[BUG] Slow Tokenizer Message is printing when the Fast Tokenizer may be in use

Open david-vectorflow opened this issue 1 year ago • 1 comments

Issue description: For Llava 1.5 13b, if you run it with the --tokenizer_mode "auto" flag set, it still prints a message that the slow tokenizer is being used. Llava has an image processor and a text tokenizer. It is not possible (to my knowledge) to set a fast image processor, but you can set a fast image tokenizer from the sentence piece library. Setting the flag above should create that fast tokenizer. However the code still prints this message Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead. If you look at the code here since the LlavaTokenizer's text tokenizer is a field, it should actually be: if not isinstance(tokenizer, PreTrainedTokenizerFast) or (isinstance(tokenizer, LlavaTokenizer) and not isinstance(tokenizer.tokenizer, PreTrainedTokenizerFast)):

Steps to reproduce:

Please list the steps to reproduce the issue, such as:

subprocess.Popen([
        "python", "-m", "lightllm.server.api_server", 
        "--host", "0.0.0.0",
        "--port", "8080",
        "--tp", "1",
        "--max_total_token_num", "20000",
        "--trust_remote_code",
        "--enable_multimodal",
        "--cache_capacity", "1000",
        "--model_dir", MODEL_DIR, 
        "--tokenizer_mode", "auto",
        "--max_req_total_len", "6000"])

Expected behavior: No message should print. OR a message should print showing which fast tokenizer is in use.

Error logging: INFO 05-07 21:41:31 [tokenizer.py:76] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.

david-vectorflow avatar May 07 '24 23:05 david-vectorflow

@david-vectorflow thanks, we will fix this。

hiworldwzj avatar May 10 '24 01:05 hiworldwzj