Avi Jain

icon location New York icon location CS Grad from BITS

Results 1 comments of Avi Jain

Difference in behavior between fast tokenizers and normal tokenizers regarding unicode characters in strings

Oh sure `AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=True)`