UMT5 & ByT5 Support

[Respaired]

Feature request

Adding support to ByT5 & UMT5, two popular variants of the T5 Seq2Seq models, would be great.

Motivation

ByT5 is an essential model that outperforms all other variants of T5 for grammar correction and most importantly the Grapheme-to-Phoneme conversion which is the core part of most Text-to-Speech models, I cannot emphasis enough how import latency is in this field.

As for UMT5, it's the most recent variant of T5 and it seem to be the SOTA when it comes to this architecture. unfortunately the latency is a bit high using when we use these models, especially since their smallest models are 300M which is still quite large.

Your contribution

I'm afraind not, It's a bit beyond my current skills.