feat: Add ComfyUI tool for Stable Diffusion

[QunBB]

Checklist:

[!IMPORTANT]
Please review the checklist below before submitting your pull request.

• [ ] Please open an issue before creating a PR or link to an existing issue
• [x] I have performed a self-review of my own code
• [x] I have commented my code, particularly in hard-to-understand areas
• [ ] I ran dev/reformat(backend) and cd web && npx lint-staged(frontend) to appease the lint gods

Description

Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. Close issue syntax: Fixes #<issue number>, see documentation for more details.

Fixes

Type of Change

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update, included: Dify Document
• [ ] Improvement, including but not limited to code refactoring, performance optimization, and UI/UX improvement
• [ ] Dependency upgrade

Testing Instructions

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• [ ] Test A
• [ ] Test B

[QunBB]

@crazywoola Done. Check it again please.

[WeepsDanky]

Hi @QunBB . When I was testing this tools, an error shows up: Failed to get models, Please input model. My comfyui is started on http://127.0.0.1:8188/ with a blank canvas. How to resolve this error? I have my models loaded in /models/checkpoints.

[QunBB]

@crazywoola @WeepsDanky The model name should be also exposed when verifying tool's credentials. Some modified code was after verifying credentials, so i miss the test about the changed code about validate_models. I will fix it later.

[QunBB]

@crazywoola @WeepsDanky I have fixed it, please check it again.

[QunBB]

@crazywoola But i am troubled by Ruff check, should i reformat the files?

[WeepsDanky]

@QunBB Great thanks, it is working now.

[WeepsDanky]

@QunBB I noticed the tool currently can only use a pre-defined workflow txt2img.json. Can you please be more specific about the name and description in yaml? We need to make sure other users understand this tool can only use this workflow.

For example: name: txt2img workflow, description: a pre-defined comfyui workflow that can use one model and up to 3 loras to generate images. Does not support newer models such as stable diffusion 3 that requires a triple clip loader.

[QunBB]

@WeepsDanky Hi, i have change the name and description in yaml. In addition, i add the support for SD3 and FLUX. You could try them like the examples:

• SD3
• flux-fp8

[wisepmlin]

Mac comfloxy Failed to get models, [Errno 111] Connection refused

[QunBB]

@wisepmlin This seems to be a network issue. I have tried to install Mac comflowy and it worked too.

[Dongnc1017]

https://docs.dify.ai/tutorials/tool-configuration/comfyui 的链接无法访问 1、请问我想使用flux 的模型，Flux Dev和Flux Schnell，都支持吗？是否下载过来放在ComfyUl/models/unet/ 就行了。 2、但是又说支持包含文本编码器/clip的SD1.5、SDXL、SD3和FLUX，但不支持需要clip加载器的模型。 这句话是什么意思？ 比如t5xxl fp16.safetensors 或t5xxl fp8 e4m3fn.safetensors:？ 能否出一个详细的教程

[QunBB]

@Dongnc1017 The comments above have already mentioned it. You should download those models which contain text encoders, then put them into ComfyUl/models/checkpoints/ like official comfyui examples:

• SD3
• flux-fp8

Flux Dev and Flux Schnell are both supported.

[laipz8200]

Hi @QunBB, would you be interested in updating the documentation for this tool? Or perhaps removing the inaccessible links from the configuration?

[QunBB]

@laipz8200 I'm intersted in updating it when i'm free later. I would replace the current link with the official ComfyUI website first, and will add it again when i finish the documentation in dify.

[hjlarry]

Hi @QunBB I think current ComfyUI tool can be better by reduce the user input and selection, just give the user a prompt_text input.

The comfyUi workflow is similar to dify's workflow, support a variety of nodes. Current implementation seems define specific steps, the user can only select each params of these steps, seems inflexible.

A more reasonable user case is: the user edit the workflow in the comfy UI, and then export the json of this workflow which is simliar to dify's DSL, then paste it to this tool to get the image in dify. Now they combine the dify and comfyUI's workflow.

This is the export API button:

This is a simple workflow just to change an ICON's style, but it has 40 nodes, current solution can't do this:

what do you guys think about this ?

[QunBB]

@hjlarry Sure, we could add it, then it will support any image generation workflows in ComfyUI. But i don't think LLM could correctly generate the prompt text via agent, so it may only be used in dify's workflow. By the way, i think the current tool could be continue to be used for beginners or for simplicity like the pattern of stable diffusion tool. And it could be both used in dify's agent and workflow.

[hjlarry]

@QunBB Sure, we can add a new tool and keep both of them. I think the new tool can config the workflow's json and which node is the text node, to make LLM just gen text prompts then used to agent app.

[QunBB]

@hjlarry It is a good idea.