Philpax
Philpax
This is really well done! Awesome work, and sorry about not looking at it earlier! Honestly, it's so thorough I'm wondering if I should encourage you to submit it to...
> Can you write one sentence that describes the purpose of token-scoring? Is this correct? > > > Depending on the GGML version, the token may also include a 32-bit...
You need to retrieve the submodules - try `git submodule update --init`.
https://huggingface.co/docs/transformers/model_doc/gpt_bigcode https://huggingface.co/HuggingFaceH4/starchat-alpha/blob/main/config.json Looks to be a GPT-2 variant.
This is good to go, but will produce garbage output after a few tokens without this fix to `ggml` https://github.com/ggerganov/ggml/pull/145/files#diff-b4a500ab2765c31526c5541f3e51e21e46990b87d9774cac6f3089db315bdc5b We'll have to wait for https://github.com/ggerganov/ggml/pull/145 to be merged.
Sounds like this is all good to go! @danforbes Does this include all of the changes from the ggml PR?
You'll probably have to convert it using that PR's Python: https://github.com/ggerganov/ggml/blob/b43281ab20a7d175a46af0c50ab3563509c59ed5/examples/mpt/convert-h5-to-ggml.py
MLC is very cool, but I'm not sure how much effort it would be to integrate. We'll have to implement support for custom backends first (#31).
Our current tokenizer is built around scores. Perhaps we should use a simpler tokenizer for the models where it's known no score is present for the tokens?
Yeah maybe, see #35