add pretraining code and multi modal support?

[leo-young]

Hi, thanks for the excellent work, I found it very useful to learn the llm sft by the code. So do you plan to add the pretraining support, a easy and efficient preprocess pipline and pretrain medium-sized llm? 7B? and by far, multi modal llm is very popular, like llava, do you plan to add this kind of support?

[kartikayk]

Thanks for opening this issue @leo-young! Glad torchtune has been helpful for you.

We are working on adding multimodal support and have a couple of RFCs that you can read on this: #1224 and #1283. The goal is to land a modern version of Flamingo followed by llava.

You can read more here for continued pre-training support in torchtune and @RdoubleA can share more pointers in case you're interested.

[joecummings]

Closing b/c no activity in awhile and we have added multimodal support.