cwleungar

Token loss for llama.tokenize() with mixed Chinese/English text

5

When calling llama.tokenize() from llama_cpp_dart on a mixed Chinese/English string, the returned token count is significantly smaller than the token count produced by llama-cpp-python using the same GGUF model and...