javalang icon indicating copy to clipboard operation
javalang copied to clipboard
verified publisher icon c2nes

Update tokenizer.py

Open mingchen-li opened this issue 2 years ago • 0 comments

The tokenize function can return index now. It would be helpful for developing https://huggingface.co/docs/tokenizers/api/pre-tokenizers for Java.

mingchen-li avatar Sep 12 '23 05:09 mingchen-li