Cannot correctly recognize <im_patch>

[m2408gj]

trafficstars

When running codes on jetson, I found that the tokenizer (tokenizer = AutoTokenizer.from_pretrained(args.model_path, use_fast=False)) cannot correctly convert the LLAVA_DEFAULT_IMAGE_PATCH_TOKEN, i.e., <im_patch>, into the index in the vocabulary. That is, the tokenizer cannot recognize the LLAVA_DEFAULT_IMAGE_PATCH_TOKEN as a special token.

The environment is built following the instructions in the AWQ project (https://github.com/mit-han-lab/llm-awq), while the version of transformers is 4.36.2.

Did I do something wrong?

[Efficient-Large-Language-Model]

Could you try VILA1.5 and let us know if there is still issue?

[Lyken17]

Close due to inactivity. Feel free to reopen if the issue still exsits.