Add tokenizer to language modelling Trainer examples

Open lewtun opened this issue 2 years ago • 1 comments

trafficstars

What does this PR do?

Propagates a fix for the MLM and CLM examples first reported in the course in https://github.com/huggingface/course/pull/432

tl;dr we were missing the tokenizer in the Trainer, so the resulting models on the Hub couldn't be loaded.

Feel free to tag members/contributors who may be interested in your PR.

Dec 28 '22 06:12 lewtun

Check out this pull request on

See visual diffs & provide feedback on Jupyter Notebooks.

Powered by ReviewNB

Dec 28 '22 06:12 review-notebook-app[bot]