Brett Kuprel comments

Repositories
Issues
Comments

Results 22 comments of


                                            Brett Kuprel

Train BasicTokenizer on GPU with PyTorch, 100x speedup

All of the tests pass

Train BasicTokenizer on GPU with PyTorch, 100x speedup

Thanks for the feedback! I made the diff more surgical. Now the only added files are: - minbpe/basic_torch.py - Contains `merge_torch` and `BasicTorchTokenizer`, overrides the `train` and `encode` methods of...