minbpe
minbpe copied to clipboard
updating stats across merge to reduce computation
Instead of computing stats from scratch for every merge, we can calculate it once and update it during merge
. This results in reduced computation as we update the stats dictionary only for tokens that are affected by the merge.