llama-cpp-python icon indicating copy to clipboard operation
llama-cpp-python copied to clipboard

Published 20 hours ago • verified publisher icon abetlen

Enable detokenizing special tokens

Open benniekiss opened this issue 7 months ago • 1 comments

I noticed that it was not possible to detokenize special tokens, such as the EOS token, when using Phi-3(.1). This PR makes sure that the special flag can be passed to the detokenize() method of both the LlamaTokenizer and the LlamaHFTokenizer

I also noticed that prev_tokens was not being used in the detokenize method of LlamaTokenizer, so this PR also adds that functionality based on LlamaHFTokenizer

benniekiss avatar Jul 13 '24 14:07 benniekiss