minbpe icon indicating copy to clipboard operation
minbpe copied to clipboard

Published 20 hours ago • verified publisher icon karpathy
→

Metadata

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

Results 61 minbpe issues
Sort by recently updated
recently updated
newest added

updating stats across merge to reduce computation

Instead of computing stats from scratch for every merge, we can calculate it once and update it during `merge`. This results in reduced computation as we update the stats dictionary...

imdaredevil avatar imdaredevil

← Metadata

7.7k
Stars
646
Forks
76
Watchers

Owner

verified publisher icon karpathy

Metadata

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

Back