Pascal Voitot
Results
3
issues of
Pascal Voitot
Encoding provides 2 functions to pad and truncate the current encoding. But the pad/truncate functions return `()` so you can't chain calls on the Encoding structure such as: ``` encoding...
I'm training tokenizers but I need to manipulate the generated tokens sometimes. In current API, there is no way to access unknown tokens (and others) which are hidden in Models...
enhancement