llama2.c icon indicating copy to clipboard operation
llama2.c copied to clipboard

Published 20 hours ago • verified publisher icon karpathy

tok512.model adding a token at start.

Open vish-pr opened this issue 2 years ago • 0 comments

To reproduce: sp_mode.Encode('R') -> [410, 461] sp_model.Encode(sp_model.Decode(461)) -> [410, 461]

I am not able to understand this behaviour, not related to bos, eos, unk

vish-pr avatar Oct 17 '23 14:10 vish-pr