python-bpe icon indicating copy to clipboard operation
python-bpe copied to clipboard

It amazes me that this implementation of BPE doesn't require LOOP. How to make it correct?

Open lsy641 opened this issue 3 years ago • 0 comments

If a pair is the best in frequency, their combination should reduce the count of both single tokens respectively, which may result in the single tokens' difficulty in keeping high ranks in frequency sorting. But your implementation seems to count all 1-gram, 2-gram .. max-gram in one shot. Is it correct?

lsy641 avatar Oct 11 '22 03:10 lsy641