Kins

icon company Nowhere icon location ZhouHang icon location Little yellow flower of the story

Results 1 issues of Kins

Update tokenizer.py

The tokenize function can return index now. It would be helpful for developing [https://huggingface.co/docs/tokenizers/api/pre-tokenizers](pre-tokenizer) for Java.