python-bpe
python-bpe copied to clipboard
It amazes me that this implementation of BPE doesn't require LOOP. How to make it correct?
If a pair is the best in frequency, their combination should reduce the count of both single tokens respectively, which may result in the single tokens' difficulty in keeping high ranks in frequency sorting. But your implementation seems to count all 1-gram, 2-gram .. max-gram in one shot. Is it correct?