stanford_alpaca icon indicating copy to clipboard operation
stanford_alpaca copied to clipboard

Published 20 hours ago • verified publisher icon tatsu-lab

Why one token corresponds to multiple token ids

Open FinalFlowers opened this issue 2 years ago • 1 comments

f4ce54cf-7ef4-4895-b7e0-9b09df84f711

FinalFlowers avatar Apr 09 '23 05:04 FinalFlowers

I think one token corresponds to "yes" and the other one is "yes". The special character "" represents space in the sentencepiece tokenizer.

ArvinZhuang avatar Apr 11 '23 05:04 ArvinZhuang