diffusers
diffusers copied to clipboard
huggingface
Community Integration: Making AIGC cheaper, faster, and more efficient.
Is your feature request related to a problem? Please describe. AIGC has recently risen to be one of the hottest topics in AI. Unfortunately, large hardware requirements and training costs are still a severe impediment to the rapid growth of the AIGC industry. The Stable Diffusion v1 version of the model requires 150,000 A100 GPU Hours for a single training session.
Describe the solution you'd like We are happy to share a fantastic solution where the costs of training AIGC models such as stable diffusion can be 7 times cheaper!
Colossal-AI release a complete open-source Stable Diffusion pretraining and fine-tuning solution with the pretraining cost reduced by 6.5 times, and the hardware cost of fine-tuning by 7 times. An RTX 2070/3050 PC is good enough to accomplish the fine-tuning task flow, allowing AIGC models such as Stable Diffusion to be available to a wider community.
Open-source code：https://github.com/hpcaitech/ColossalAI/tree/main/examples/images/diffusion
Additional context More details can be found on the blog. We believe the democratization of AIGC models is also very helpful for Hugging Face members. We would appreciate it if we could build the integration with you to benefit both of our users and we are willing to provide help you need in this cooperation for free.
Thank you very much.
Best regards, Yongbin Li, HPC-AI Tech
Hey @binmakeswell,
This sounds great - I think we'd be very interested in having a better integration here! Also cc @patil-suraj @thomwolf
This is very cool @binmakeswell ! Would be very happy to integrate diffusers in colossalAI examples!
Thanks for your enthusiastic response and we look forward to working together!
How do you expect it to be carried out? Take our collaboration with Lightning for example, we communicated in slack for details, provided documentation and eventually did a joint release.
Would love to see a simple integration between diffusers and ColossalAI!
Here's a collection of related issues I found/opened on various StableDiffusion / Dreambooth related projects about this:
- https://github.com/hpcaitech/ColossalAI/issues/1935
- https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/4606
- https://github.com/d8ahazard/sd_dreambooth_extension/issues/174
- https://github.com/TheLastBen/fast-stable-diffusion/issues/493
- https://github.com/ShivamShrirao/diffusers/issues/129
It would also be really interesting to hear/see benchmarks for how ColossalAI is compares and contrasts with the various other speed/memory/etc optimisations out there (eg. xformers, AITemplate, deepspeed, deepspeed-MII, bitsandbytes, kernl, Triton, etc); whether it is a replacement for them, or whether it can be used alongside them for even greater gains. A few related issues off the top of my head:
- https://github.com/huggingface/diffusers/issues/1094
- https://github.com/huggingface/diffusers/pull/532#issuecomment-1301753138
- Mentions AITemplate, DeepSpeed, DeepSpeed-MII, Triton
- https://github.com/huggingface/diffusers/pull/532#issuecomment-1303759989
- More about DeepSpeed-MII
- https://github.com/huggingface/diffusers/pull/532#issuecomment-1309786571
- AITemplate + xformers
Thanks for your enthusiastic response and we look forward to working together!
How do you expect it to be carried out? Take our collaboration with Lightning for example, we communicated in slack for details, provided documentation and eventually did a joint release.
Hi, @patrickvonplaten and @patil-suraj , thank you for the enthusiastic response.
Do you have any requests or suggestions? Hopefully, we can discuss the details soon and complete this integration. Thanks.
Hey @binmakeswell,
Should we jump on a call to discuss? :-) I think we've reached out via email
Hey @binmakeswell,
Should we jump on a call to discuss? :-) I think we've reached out via email
Sure. @patrickvonplaten I just emailed([email protected]) you our available hours. Looking forward to our meeting and cooperation :-)
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
Any update @binmakeswell ? It would be very great to fasten training process.
I found train_dreambooth.py and train_dreambooth_colossalai.py that are based on diffusers.
Any update @binmakeswell ? It would be very great to fasten training process.
I found train_dreambooth.py and train_dreambooth_colossalai.py that are based on diffusers.
Thank you for your interest and contribution to colossalai, we have merged into the hugging face diffuser library. https://github.com/huggingface/diffusers/tree/main/examples/research_projects/colossalai
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
It's merged to diffusers :-) See:
https://github.com/huggingface/diffusers/tree/main/examples/research_projects/colossalai
It's merged to
diffusers:-) See: https://github.com/huggingface/diffusers/tree/main/examples/research_projects/colossalai
so does that mean that integration with Automatic1111 is around the corner?
@Sniper199999 Last I paid attention, automatic1111 isn't based on diffusers at all, so unless that has changed since I last looked, short answer is "no, probably not"
Actually we'd really like to integrate diffusers into A1111, but it seems quite complicated. Note sure if anybody has a clear plan of action. Also related: https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/2577
Should we try to create a module plug-in?