Sagor Sarker

icon indicating a website link https://sagorbrur.github.io icon indicating an email [email protected]

icon company @hishab-nlp icon location Dhaka, Bangladesh icon location An enthusiastic NLP/AI/ML practitioner.

Results 1 issues of Sagor Sarker

Tiktoken educational BPE trainer takes long time to train with vocab size 30k

2 comment

Hi, I am trying to train Tiktoken on a custom dataset (size 15 GB) with 30k vocab size. It seems it will take a long time to finish. 1 vocab...