tan90º
> Extremely hacky, but I managed to work around this by passing my own byte_decoder as part of the tokenizer. > > e.g. for Llama 3, you can brute force...
> Extremely hacky, but I managed to work around this by passing my own byte_decoder as part of the tokenizer. > > e.g. for Llama 3, you can brute force...
yes, and I prefer transformers because they support a lot of models. It would be better if it could support TurboMind. I always use TurboMind to run LLM evaluation, because...