fastBPE icon indicating copy to clipboard operation
fastBPE copied to clipboard

Published 20 hours ago • verified publisher icon glample

limitVocab

Open geert56 opened this issue 3 years ago • 0 comments

Why is the limitVocab function necessary? And is it correctly implemented? I see that the query string might have a kTokenDelim attached but then the lookup in vocab makes no sense. What is going on here?

geert56 avatar Oct 22 '21 20:10 geert56