Austin
Austin
We do things on Linux very differently. We usually don't download random stuff and we're very picky about our pipelines. Usually, the downloader will handle anything for you, winetricks components...
Since wine runs apps in a contianer, it's usually better to let the authoring software hande it for you to avoid breakage, conflicts, mangled setups, etcetera. It's also the same...
I had to do a fresh install, so I wanted to try out renaming `version.dll` and `bink2w64.dll`. Renaming it did not work for me at all. I made multiple attempts...
It's always the tokenizer. The tokenizers are always a mess. Special tokens apply to the [instruct tuned model](https://github.com/meta-llama/llama3/blob/main/README.md#instruction-tuned-models). The [ChatFormat](https://github.com/meta-llama/llama3/blob/main/llama/tokenizer.py#L202) class in the source code shows how they implemented it....
Okay, it's in there. ```python # BOS / EOS token IDs self.bos_id: int = self.special_tokens[""] self.eos_id: int = self.special_tokens[""] self.pad_id: int = -1 self.stop_tokens = { self.special_tokens[""], self.special_tokens[""], } ```...
In the original `convert.py`, before the refactoring, `BpeVocab` would scan the first line if could and assume it was a plaintext format. Now it's assuming the huggingface BPE format instead...
@pcuenca Thanks! I really appreciate your input and feedback. I'll check it out when I have some time. @ryao That's the Huggingface model created with the `transformers` and `tokenizers` frameworks....
What makes the `convert.py` script so valuable and useful is that it doesn't load the full model into memory and it is not supposed to depend on any other library...
> Aren't the weights the same? @ryao The answer to that question is that it's nuanced. The issue stems from the file formats and tokenizers (aka vocabularies). The `consolidated.00.pth` is...
@pcuenca Yeah, that makes me glad I'm waiting. Patience usually wins out. I can implement it later on once I have some more time. I did the vocab factory for...