markjr-cisco

Tried `examples/example.py` with a tokenizer derived from a `dict[str, int]`: ```python from tokenizers import Tokenizer from tokenizers.models import WordLevel tokenizer = WordLevel(Tokenizer(str_to_int_dict)) tokenizer.eos_token_id = '\n' ``` Stack trace: ```shell Traceback...

### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. -...