chaiNNer
chaiNNer copied to clipboard
joeyballentine
Internal Stable Diffusion
- Proof of concept and request for comment
- Using a fork of ComfyUI that's packaged as a library (https://github.com/adodge/ComfyLib)
- Currently you need to manually install from git, but I'll put it in PyPi if this looks viable
- Need to work on the Navi input/output types and the renderer components
- Would be nice to cache checkpoints between runs. loading the model can take 5-10 seconds
- Needs a higher-level node that does something like the exsting external "Text to Image"
- There are lots of other nodes in ComfyUI that could be ported over
Did some work on the library. Better VRAM memory management options. Rearranged the node inputs to have the image and model as the first two inputs.
Stuff to do before I mark this ready to review:
- [x] more specific input/output types and frontend stuff to, e.g., show the resolution of latent images or version of models
- [x] make monolithic "text to image" and "image to image" nodes
- [x] make sure this works for v2.x stable diffusion models, or make sure it emits a useful error
- [x] put the library on PyPi and link it in the dependency manager
- [x] Automatically round down sizes to the nearest valid values, like we do for the external SD
Note: this will now need to be updated to the new package node importing & function-based node format. I can do this myself if you'd prefer
Note: this will now need to be updated to the new package node importing & function-based node format. I can do this myself if you'd prefer
That would be very nice, if it's not too much trouble. Sorry this is taking so long for me to get back to. Hoping for this week.
Sorry, been working on other things and haven't gotten a chance to make that change. If you still want me to do it i can do it today
I think it shouldn't take me too long. Will be a good chance to familiarize myself with the new stuff
Rebase done. Haven't addressed any of the other feedback, but it works again now. I like the new system!
@joeyballentine , I came across your tool and discovered this PR while looking for similar functionality. Is the PR still a Work in Progress or is it stale ?
@MElhagaly at this point, this PR is stale as adodge never finished it and I don't have the time to redo the work that went into it to fix it.
If you want node-based SD, just use comfyui
@joeyballentine I was considering picking up this PR and finish it. At least, if the functionality is deemed interesting and relevant to the community.
Thoughts?
To be honest, I don't think having SD functionality in chaiNNer is even worth it. I don't have the time to keep up with the constantly changing landscape and constant new features popping up in that space. Even if this PR were to be finished, it would be massively out of date and wouldn't support most of the things people have come to expect in an SD application in the time since this PR was first made. It would take a significant amount of effort to implement all that new stuff on top of this (not to mention you'd need to update ComfyLib first).
Maybe some day once chaiNNer supports extensions/third party nodes, someone can make and maintain their own SD extension independently. That would probably be the most ideal scenario.
So, if you were to finish this out, that would be great, but it would be sorta a maintenance hell for us unless you're willing to stick with it and keep it up to date for us. And that would include maintaining some version of ComfyLib (if you were to finish out that project as well). If you're willing to do that and can commit to at least keeping it up to date every once in a while, then I say go for it (and I would very much appreciate it).
I don't remember exactly where this left off, but I think one of the biggest blockers were all of the unnecessary dependencies ComfyLib was pulling in. In order for this to be considered, it would need to significantly reduce that.
Thank you for your detailed reply. I will take a look at this PR, and try and get it working. I see what you mean with ComfyLib having a huge amount of dependencies. My workflow will be to get everything up and running using @adodge's work. Then see what can eliminated.
I already noticed that Dependency management has been moved to the backend since the start of this PR. So I will try and evaluate what other changes have been pushes as well. If you have a chance, you could save my sometime, if you can summarize other major changes that have taken place since.
In the meantime, I will get started and keep you updated.
I think dependency definitions being moved to the backend is the only major thing that changed since then, but I could be forgetting things. After a quick glance at the code changes for this, that seems like the only thing thats super out of date (in terms of chaiNNer code that is)
@joeyballentine quick update: I have managed to get this PR to work on the latest version of ChaiNNer. It works as expected. I haven't implemented the dependencies part yet, mainly to figure out how many dependency installs are required. As you stated, it is a lot of (unnecessary) dependencies.
Next steps for me are:
- Implement dependency installation
- Fork and Extend ComfyLib to be more of a wrapper/interface API around ComfyUI. Ideally we would want ChaiNNer to simply be extended with ComfyUI Chainner without the tight coupling. What do you think?
I am willing to dedicate some time to get the base up and running as well as maintain it in the long run.
You already mentioned a future functionality where chaiNNer would allow third party packages / nodes. We can implement this for now and once that functionality is implemented we can extract the package from chaiNNer and host it in a separate repo.
Any thoughts?
@MElhagaly that sounds good to me. Thanks for the update and I appreciate you being willing to work on this 🙏