stable-dreamfusion
stable-dreamfusion copied to clipboard
ashawkey
Implementing DALLE2 for 2D Image Generation
Hi all!
I saw that LAION-AI has an open source DALLE2 model hosted on huggingface at: https://huggingface.co/laion/DALLE2-PyTorch The model is pretrained and ready for use. I am going to test it out compared to StableDiffusion, but from what I've seen online, it seems that the DALLE2 architecture and output does seem to be more advanced than StableDiffusion. LAION-AI created a Github repo with inference scripts to use with the model found here: https://github.com/LAION-AI/dalle2-laion Could this be implemented in DreamFusion and thus produce better 3D models?
I implemented Laionide (GLIDE), it does seem to solve the Janus problem in the squirrel case
Cool! I'm super new to ML still, so what exactly is Laionide (GLIDE)? I saw the Github Repo they have but still not super sure what it is. How does it compare to DALLE2?
@neverix could you please provide your implementation in a forked repo if possible Would be really really helpfull
Cool! I'm super new to ML still, so what exactly is Laionide (GLIDE)? I saw the Github Repo they have but still not super sure what it is. How does it compare to DALLE2?
It's a smaller model trained on LAION by afiaka87. Pretty stable
@neverix great work on adding the laionide(GLIDE). Do you have some sample outputs where the Janus problem seems to be gone as you mentioned. It will be really helpful for me. Thanks
I just checked, there actually is a subtle Janus problem. Going to try dalle2-pytorch instead.
https://user-images.githubusercontent.com/46641404/208136968-340478c0-5932-4296-9c87-db7889e1f211.mp4
@neverix Hi, have you implement the dreamfusion using the dalle2 model? Could you please provide your implementation in the forked repo if possible. thanks a lot.
Yes, for now I'm trying https://github.com/kakaobrain/karlo, this is in the same repo
Yes, for now I'm trying https://github.com/kakaobrain/karlo, this is in the same repo
Thanks a lot. Great work!
Yes, for now I'm trying https://github.com/kakaobrain/karlo, this is in the same repo
How are the results with implementing Karlo into Dreamfusion? Is it producing better results than Stable-Diffusion and DALLE2? Thanks for giving the link to your repo. What additional command line arguments do I need to include to use main.py with Karlo?
The flag is --guidance karlo, it technically works but results are even worse that GLIDE. I'm trying to figure out why the images are so incoherent.
Closing this issue since it's old, and @ashawkey made tons of progress since then.
