transformers icon indicating copy to clipboard operation
transformers copied to clipboard

Published 20 hours ago • verified publisher icon huggingface

FastTokenizer for LLaMa

Open theblackcat102 opened this issue 2 years ago • 3 comments

Feature request

FastTokenizer support for LLaMa sentencepiece tokenizer.

Motivation

The offset_mapping is only available in FastTokenizer, it would be useful if there's support for this.

Your contribution

I have tried using existing sentencepiece based model as replacement. However hf conversation code means we are missing the byte fallback support

The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers

Which means out of vocabulary tokens are simply mapped to instead of using the byte mapping inside the vocab.

theblackcat102 avatar Mar 12 '23 15:03 theblackcat102

Let's maybe wait for the LLaMa PR to be merged first?

sgugger avatar Mar 13 '23 15:03 sgugger

it is fix on tokenizers

https://github.com/huggingface/tokenizers/pull/1183

dongs0104 avatar Mar 17 '23 15:03 dongs0104

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

github-actions[bot] avatar Apr 30 '23 15:04 github-actions[bot]