spacy-transformers
spacy-transformers copied to clipboard
Serialize tokenizer use_fast setting
Preserve tokenizer.is_fast as use_fast in HFShim in order to be able to restore the same slow/fast tokenizer settings.
This will be necessary when fast tokenizers switch to offset alignments from wordpiece string alignments.