[FEATURE]: Any plan to support train_dreambooth_colossalai with train_text_encoder?

[vonchenplus]

Describe the feature

I found that the train_dreambooth_colossalai script you provide no longer supports train_text_encoder parameter, but it's gives much better results especially on faces, so I would like to ask if you have plans to support this parameter?

Thanks again!

[blx0102]

+1

[mingqizhang]

Have you solve this problem?

[vonchenplus]

No, ColossalAI's optimizer cannot support multiple network parameters.

[feifeibear]

You are right. The GeminiDPP can only deal with parameters from a single model, rather than parameters from multiple models. But, you can support train_text_encoder parameter with some code modifications.

[feifeibear]

We are considering making Gemini more flexible, but we have no such plan in the next two weeks.

[haofanwang]

+1. It would be much more flexible if we can train multi-modules, which is common when finetuning stable diffusion.

[haofanwang]

@feifeibear any update for this?

[hdjsjyl]

hi @feifeibear , thanks for your great job! Would you explain a little bit about which part I should change for supporting text_encoder parameters? For my thinking, I put all models in a single model class, make unet and text encoder tainable together and fix other models. The single model will be the input of GeminiDDP class. Is it correct? Any help would be appreciated! Thank you so much!

[feifeibear]

We have a very intensive discussion on this. Can could please report the latest conclusion @1SAA , @ver217 .

[shileims]

Hi @feifeibear , Thanks for your reply.

Hi @1SAA and @ver217 , would you share your latest conclusion here? I am looking forward to your reply.

Thank you so much!

[shileims]

Hi @feifeibear , @1SAA ， and @ver217 , Any advice would be very appreciated since I am stuck here for a long time. Really looking forward to your reply. Pls feel free to update your solution here, I can validate the idea. Thank you so much!

[shileims]

Hi @feifeibear , @1SAA , @ver217 , Any chance to get your discussion here? It will be very helpful. Looking forward to your reply. Thank you so much!

[shileims]

Hi @feifeibear , @1SAA , @ver217 , Could you spend a little bit of time describing your discussion here? Because it is beneficial for many users and in many cases. Thank you so much!

[shileims]

Hi @feifeibear , @1SAA , @ver217 , Hi authors of Colossalai, would you provide some useful suggestions for training text encoder and unet together? Thanks

[shileims]

Hi @feifeibear , @1SAA , @ver217 , Any updates for this question? I am still waiting for your answers. Any answer will be appreciated! Thanks

[shileims]

Hi @feifeibear , @1SAA , and @ver217 , Would you share your brainstorm here? It will be very helpful. Thanks

[ver217]

@Fazziekey Could you answer this question?

[JThh]

#3146 should be a related issue. And we should solve these together.