Binwei Yao
Results
3
comments of
Binwei Yao
For HF, we use ``` from transformers import GPT2Tokenizer tokenizer = GPT2Tokenizer.from_pretrained('gpt2') text = "xxx" start = time.time() encoded_input = tokenizer.encode(truncated_text) end = time.time() ``` For tiktoken, we just initialize...
Have you solved the problem? I also met the same issue.
I think it's because of the version of `datasets` package. I just upgraded datasets and it works for me.