awesome-opensource icon indicating copy to clipboard operation
awesome-opensource copied to clipboard

Published 20 hours ago • verified publisher icon gitroomhq

Feature Request: Add ComfyUI as a Node for Advanced Image/Video Generation

Open thunder-velvet opened this issue 6 months ago • 1 comments

Feature Request: ComfyUI Integration

Description

We should add ComfyUI as a new node type in the Agent Flow system. ComfyUI is a powerful, node-based UI for Stable Diffusion that offers advanced control over the image generation process with custom workflows.

Why It's Needed

  • Advanced Image Generation: ComfyUI supports complex workflows with fine-grained control over the generation process
  • Complementary to Existing Solutions: While we already have image generation providers like Replicate, OpenAI, and Fal.ai, ComfyUI would allow for more customized and complex image workflows
  • Community Support: ComfyUI has a large community with many custom nodes and extensions that our users could benefit from
  • Local Generation Option: Could provide an option for users to run generation locally if desired

Proposed Implementation

  1. Add a new provider in packages/backend/generations/providers/comfyui.provider.ts
  2. Create a new node type in packages/shared/agents/agent.flow.ts for ComfyUI
  3. Implement the agent in packages/backend/agents/providers/generate-image/comfyui.agent.ts
  4. Add the UI component in apps/frontend/src/components/agents/nodes/generate-image/comfyui.node.tsx
  5. Update the node list in apps/frontend/src/components/agents/nodes/node.list.ts

API Integration

ComfyUI can be interfaced with via its HTTP API:

  • Running workflows
  • Uploading/downloading images
  • Interfacing with the queue system

UI Configuration Options

  • API endpoint configuration
  • Workflow JSON selection/customization
  • Input parameters mapping
  • Output handling

Questions to Consider

  • Should we support self-hosted ComfyUI instances, or should we integrate with a hosted API?
  • How detailed should the workflow configuration UI be?
  • Should we provide pre-built workflows or allow full customization?

Related Work

This would expand our existing provider ecosystem:

  • Currently supporting Replicate, OpenAI, Fal.ai, and RunwayML
  • ComfyUI would provide a different approach to image generation with node-based workflows

References

thunder-velvet avatar Apr 10 '25 05:04 thunder-velvet