llama.cpp icon indicating copy to clipboard operation
llama.cpp copied to clipboard

add support of codeqwen due to tokenizer

Open JustinLin610 opened this issue 1 year ago • 1 comments

we just released codeqwen1.5, see blog and model for more info.

since for codeqwen1.5, we use a different tokenizer, which is based on sentencepiece, we need to make some changes for the convert-hf-to-gguf.py for the adaptation.

JustinLin610 avatar Apr 16 '24 16:04 JustinLin610

@ggerganov feel free to take a look at this small code change 😃

JustinLin610 avatar Apr 16 '24 16:04 JustinLin610