Ita Zaporozhets comments

Results 37 comments of


                                            Ita Zaporozhets

Load a pretrainedfast tokenizer if fast=true and tokenizer.json exists

@ArthurZucker this feature requires that `PreTrainedTokenizerFast` is an option for a given model in this specific dictionary: https://github.com/huggingface/transformers/pull/29969/files#r1792386118 So if we create a private testing model on the hub without...

Fix convert_tekken_tokenizer

run-slow: ministral3, mistral3

Fix convert_tekken_tokenizer

Hey! Thanks for the PR, can you please share a short reproducer of the problem (you mentioned in chat templates)? perhaps we'll need to add a test !

[WIP] Add Tokenizer for MyT5 Model

Thanks for applying the feedback! Looks good, please update with changes from main and address the minor CI failures (+ ruff reformat)! Otherwise LGTM 🤗

[WIP] Add Tokenizer for MyT5 Model

There might be a couple of minor conflicts to resolve!

Fix BLT training_ci overfit test

`use_cache=False` I believe since we have the BltPatcher which requires the full sequence to know how to patch / group tokens edit (more info): The `BltPatcher` will group the raw...

Add `add_language_codes` helper to `NllbTokenizer` for new language codes

Hey :) Thanks for the PR. It's great that support for new language codes can be supported. I think we can leave this feature to be covered by the existing...