SharpToken icon indicating copy to clipboard operation
SharpToken copied to clipboard

Published 20 hours ago • verified publisher icon dmitry-brazhenko

SharpToken.GptEncoding.GetEncoding(o200k_base) takes minutes

Open vovanb opened this issue 1 year ago • 1 comments

Hi I am using your library in MAUI android. For some reason method SharpToken.GptEncoding.GetEncoding(o200k_base) first time takes minutes to respond What can be issue?

vovanb avatar Jul 25 '24 22:07 vovanb

Hey!

Sounds interesting. Could you share some more details please

  1. Text length
  2. android ver
  3. device

Probably could you share apk to check it? My guess is that it takes significant time just to read tokenizer dict, but not sure about it.

dmitry-brazhenko avatar Jul 26 '24 03:07 dmitry-brazhenko