Guozhen Li

University of California, Davis [ECS 265, Distributed Database Systems](https://msadoghi.github.io/ecs265-fall2018.html) [Mohammad Sadoghi](https://msadoghi.github.io/) Fall 2018 Included the Raft paper as class presentation material.

DebertaV2TokenizerFast and XLMRobertaTokenizerFast has overlapping offsets/CharSpans which leads to char_to_token() pointing to unexpected token

1

Noticed this when working with `transformers.DebertaV2TokenizerFast` and `XLMRobertaTokenizerFast`. ## My `transformers` and `tokenizers` versions ```python import transformers, tokenizers print(f'{transformers.__version__=}') print(f'{tokenizers.__version__=}') ``` ``` transformers.__version__='4.51.3' tokenizers.__version__='0.21.1' ``` ## Repro in Python ```python...