Tokenizer clean_up_tokenization_spaces

[Nickwiz]

FYI

Changes in transformers tokenizer gives deprecation warning.

/xxx/dltranslate/lib/python3.12/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: clean_up_tokenization_spaces was not set. It will be set to True by default. This behavior will be depracted in transformers v4.45, and will be then set to False by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884

Something like this can be used:

mt = dlt.TranslationModel(
        tokenizer_options = {
            "clean_up_tokenization_spaces": True # Add this
        }
)

Have not found any difference in result by using True vs False, but then again I just started looking at this project.