spacy-transformers icon indicating copy to clipboard operation
spacy-transformers copied to clipboard

Serialize tokenizer use_fast setting

Open adrianeboyd opened this issue 3 years ago • 0 comments

Preserve tokenizer.is_fast as use_fast in HFShim in order to be able to restore the same slow/fast tokenizer settings.

This will be necessary when fast tokenizers switch to offset alignments from wordpiece string alignments.

adrianeboyd avatar Jul 26 '22 13:07 adrianeboyd