transformers

transformers copied to clipboard

This PR is a rewrite of the TF examples, including several modern methods. I'm focusing on updating everything to use modern methods like prepare_tf_dataset and the evaluate library as well as adding features and functionality I missed when I first ported them, since transformers TF support was much shakier when these were first written (and I didn't know the library as well).

Just a draft for now, will ping reviewers when it's ready!

TO DO:

• [x] Draft rewrite for all scripts
• [x] Test run all scripts
• [x] Make sure we're handling batch sizes correctly in multi-GPU/TPU scopes
• [x] Make sure we're correctly using AdamW + LR decay everywhere
• [x] Make sure we're using evaluate instead of load_metric
• [x] Add metadata for push_to_hub
• [x] Add explanatory comments for things like KerasMetricCallback, jit_compile and PushToHubCallback where appropriate
• [x] Replace all the old ad-hoc data loading code with prepare_tf_dataset
• [x] Add explanatory links to the docs whenever we use prepare_tf_dataset
• [x] Add example tests
• [x] Make sure there's no case where we pass optimizer=None to compile()
• [ ] Final manual testing
• [x] ~Add HF metrics like MaskedAccuracy?~

The documentation is not available anymore as the PR was closed or merged.

This is now ready for review @sgugger @gante! I'm tracking down a couple of remaining bugs in the tests and doing some final manual checks, but almost everything should be finished by now.

I realize it's a very large PR, but you can see from the checklist above what the main changes are.

@sgugger Tests are now enabled in config.yml and everything still looks green!

That might because your new job did not run ;-) You need to add at the end here for the one at each commit and there for the nigthly one ;-)

@sgugger tests are now actually passing! I had to skip one - it fails because of a known issue with shape inference on small datasets in to_tf_dataset. There is a PR to fix that at https://github.com/huggingface/datasets/pull/4763 , we just need to wait for that to be merged before we can re-enable the test!