Use tiny models for the tests

[glerzing]

🚀 The feature, motivation, and pitch

Using tiny models for the tests may speed up the tests by a factor of 2 or 3, while still effectively verifying the functionalities.

Some examples of tiny models worth considering:

• google/t5-efficient-tiny (Seq2Seq)
• sshleifer/tiny-gpt2 (has perhaps some numerical instabilities, but it should not matter for most tests)
• EleutherAI/pythia-70m (not so tiny, but still smaller than EleutherAI/pythia-160m that is currently used)
• roneneldan/TinyStories-1M (particularly small)