Could this tool apply for encoder-decoder model, like Flan-T5?

[henryxiao1997]

It's great tool to show how to build a chatgpt-like model based on some foundation models. I wonder could it support the encoder-decoder model, like Flan-T5? Could we just directly load those models and run these scripts on them? Thanks!

[ruihan0495]

Hard to say. We changed the model to our pretrained GPT, it kept crash during training. I guess there are a lot of things to tune in ds_utils.py to make it actually work for different models.

[henryxiao1997]

what's the difference between your model and opt-x? Architecture?

[ruihan0495]

Our Model is a GPT model.

[BrunoGomesCoelho]

There is a example of Flan-T5 in the repo: training/deepseed-flan-t5-summarization.ipynb