Acly
Synchronizing Krita AI Diffusion with ComfyUI-Blender
Hello,
I am the developer and maintainer of ComfyUI-Blender, a Blender add-on that serves as an interface to execute ComfyUI workflows from Blender.
Some users of my add-on have expressed the need to synchronize workflows executed from Krita AI Diffusion to the ComfyUI-Blender add-on. Essentially retrieving generation results triggered from Krita, into Blender (see issue: https://github.com/alexisrolland/ComfyUI-Blender/issues/35).
The workflow they describe is as follow:
- The user creates a workflow using the Krita AI Diffusion custom nodes and the ComfyUI-Blender custom nodes
- The user edits his image from Krita and triggers the workflow execution on the ComfyUI server
- The workflow executes and generates outputs such as images or 3D models
- The user would like the outputs to be automatically retrieved in the ComfyUI-Blender add-on, even though the execution was triggered from Krita
This workflow is not possible at the moment because the ComfyUI-Blender add-on establishes the websocket connection with the ComfyUI server, for a specific client Id. This client Id is a UUID that is defined in the settings of the add-on. WebSocket messages for workflows executed from the Krita AI Diffusion will not be received through this connection since they likely have a different client Id.
I'd like to know if you would be keen to make some evolution to Krita AI Diffusion in order to support the requested feature? I think this could be as simple as exposing the Krita AI Diffusion client Id in its settings, so that the user could edit it and use the same client Id in Krita / Blender.
What do you think?
Hi!
From what it looks like your Blender plugin works in a similar way to custom graphs in Krita. How are they implemented?
The Krita plugin uses "Parameter" nodes as placeholders and substitutes them in the prompt workflow before sending it to ComfyUI backend. This requires to trigger the workflow from Krita. Does the Blender plugin use a similar mechanism or something else?
I think a better way to use ComfyUI and multiple applications together would be for the parameter and other input nodes to "pull" the data fromt the respective applications. For example:
- a workflow contains some Krita parameters, or an image source, which the user can configure from within Krita
- a workflow contains some Blender parameters, or image or 3D source, which the user can configure from within Blender
- the workflow can be triggered from Krita/Blender/ComfyWebUI and will request parameters/images/models from the applications
This could be very flexible and powerful, but it's quite a lot of work. I'm just throwing it out in case you are interested.
Sharing the client ID with the Blender plugin is probably very easy. It would allow the Blender plugin to receive outputs of custom workflows triggered in Krita. But if the workflow also contains Blender parameters/inputs it probably won't work?
Hi!
From what it looks like your Blender plugin works in a similar way to custom graphs in Krita. How are they implemented?
The Krita plugin uses "Parameter" nodes as placeholders and substitutes them in the prompt workflow before sending it to ComfyUI backend. This requires to trigger the workflow from Krita. Does the Blender plugin use a similar mechanism or something else?
You are correct, ComfyUI-Blender works in a similar way to your custom graphs:
- The ComfyUI-Blender custom nodes include input and output nodes of different data types that users can use in their workflows (they seem equivalent to your parameter nodes). They are documented here: ComfyUI Custom Nodes
- When a workflow is imported in the ComfyUI-Blender add-on, the add-on automatically composes the UI according to the input nodes it detects.
- When the workflow is executed from the ComfyUI-Blender add-on, the add-on retrieves the outputs according to the ComfyUI-Blender output nodes it detects (can be images or 3D models).
Most of the input and output nodes are actually just wrappers for the native ComfyUI nodes, to which they add additional widget parameters used for configuring the input controls in Blender (like min, max, step, etc.). See input nodes here.
Sharing the client ID with the Blender plugin is probably very easy. It would allow the Blender plugin to receive outputs of custom workflows triggered in Krita. But if the workflow also contains Blender parameters/inputs it probably won't work?
The envisioned usage is as follow:
- Users can create a workflow using the Krita parameter nodes, they do not even need to add the ComfyUI-Blender nodes
- In Krita, users can copy the client ID
- In Blender, users can edit the client ID to use the client ID copied from Krita, then connect the add-on to the ComfyUI server
- In Krita, users can trigger the workflow, the workflow will execute on the ComfyUI server
- In Blender, the add-on listen to the server and will retrieve any message/output for the Krita Client ID, regardless of the output nodes it comes from (I don't have the output node class info in the websocket messages anyway)
So I think this would just take a minor change on your side to expose the client ID, and I would have to do some work on the Blender add-on side to process messages when receiving them. There might be a better way to make this more robust but we could start simple, see how it works and iterate :)
Yea I think this will work for very basic scenario. I just wanted to point out that it's probably a bad approach for everything that is a little more complex (like mixing krita/blender nodes). The client ID purpose is to be unique per client after all.
Anyway, ID can now be copied from connection menu:
Fantastic! I can work with this.
What I am thinking for now is:
- Add an explicit Krita Client Id in the ComfyUI-Blender settings
- When set, it will listen to the ComfyUI server in a dedicated thread
- It will retrieve and display outputs in a dedicated "Krita" panel in Blender, to not mix all outputs coming from Blender vs Krita
I'll keep you posted on this.
Thanks for the prompt update!
Unfortunately, after further testing, it seems ComfyUI server does not allow to have multiple connections for the same client Id. I tried as follow:
- Fetched the client Id from ComfyUI web app
- Open WebSocket connection in Blender for the client Id
- Run workflow from ComfyUI web app
==> I do see websocket messages coming in Blender, but the ComfyUI web lost the connection and fails to retrieve the results
- I refreshed the ComfyUI web app
- Run workflow from it
- No websocket message come in Blender
==> Which makes me think the WebSocket connection from the web app was recreated, and the WebSocket connection in Blender got closed
I'm not sure whether we would get the same behavior with krita, but it's likely to be the case. I can test once you release your new version. If this does not work I'm out of idea...
[edit] Actually one idea could be to simply poll the outputs endpoint of ComfyUI API from Blender, but it will return everything which I am not sure is desirable
ComfyUI keeps websocket connections in a dict which maps client-id -> socket, so there can only be one connection per ID. Which makes sense to allow reconnects.
Actually one idea could be to simply poll the outputs endpoint of ComfyUI API from Blender, but it will return everything which I am not sure is desirable
Krita plugin doesn't write outputs to disk so they won't appear there.
Maybe you can make a custom node "Send Image To" which allows user to enter a client ID in the workflow where the image should go?