machinelearning icon indicating copy to clipboard operation
machinelearning copied to clipboard
verified publisher icon dotnet

text tokenizer EncodeToTokens repeat

Open syp10000 opened this issue 2 months ago • 2 comments

input text："如果此时在位检测传感器的橙色LED灯熄灭" Encodes input text to a list of EncodedTokens. offset value repeat

Image

syp10000 avatar Oct 11 '25 01:10 syp10000