itsdotscience
Results
2
comments of
itsdotscience
> AMD Might be able to help with regard to both AMD AVX512 and the no improvement over AVX2 on some systems. Lets start with AMD because I've been eager...
It seems there was a change recently that pins bpe to vocab.json . From the HF docs it looks like any compatible PretrainedTokenizer transformers supports could be represented by tokenizer.json...