Vincent Haines

Special tokens not showing up correctly when tokenized.

1

I tried adding some special tokens to the vocab of a pretrained model. Made a PR for minor code fix. When I try and encode strings these new tokens are...

Fix bug in changing size of tokenizer with special tokens. This function returns none instead of the data object that its updating. This cause it to break.

Add support for qwen 2.5 chat template