Auto convert without tokenizer.json to prevent performance downgrade?

[leiwen83]

As mentioned in #20 , lightllm performance would downgrade a lot if without tokenizer.json. So for those model without this file, shall it be reasonable to add some auto conversion process in the server start to workaround this case?

Thx

[llehtahw]

Hi @leiwen83, Were you able to use the fast tokenizer in LightLLM? If yes, would you like to open a pull request ? 😄

[super-buster]

You can save fast tokenizer in advane, refer to fix slow tokenizer

Then auto load fast tokenizer if you want. I have modified in my code