Mingkuan Zhao

Have you solved your problem? I have the same confusion. Thank you.

Thank you, I just found that the tokenizer.json file is defined like this:（https://huggingface.co/allenai/OLMoE-1B-7B-0125-Instruct/raw/main/tokenizer.json） { "id": 50279, "content": "|||IP_ADDRESS|||", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "special": true }, {...

Thank you, your solution is very good. I think I found the real reason for the problem: in the tokenizer.json of version 09.24 (https://huggingface.co/allenai/OLMoE-1B-7B-0924- Instruct/blob/main/tokenizer.json), it is defined as follows:...