ComfyUI
ComfyUI copied to clipboard
[Feature Request] Deterministic custom nodes loading
Feature Idea
Scenario
I had a project that load more than 20+ custom nodes, it runs fine in local (one development A10G cloud linux instance).
But, when I tried to deploy it to another cloud A10G linux instance, it causes "Segmentation Fault" on ComfyUI. I found the custom nodes loading log are NOT deterministic for different instances which make the troubleshoot process difficult as I cannot reproduce the loading orders.
Then, I tried to deploy again to a new A10G linux instance, it works again. Once it works, it has no issues afterwards.
https://github.com/comfyanonymous/ComfyUI/blob/f2aaa0a4754abf4155bf3672279073149589b5ee/nodes.py#L2041-L2044
Example
Example logs in instance 1 (failure)
Trying to load custom node /opt/program/ComfyUI/custom_nodes/comfyui-reactor-node
#033[0;33m[ReActor]#033[0m - #033[38;5;173mSTATUS#033[0m - #033[0;32mRunning v0.5.1-b2 in ComfyUI#033[0m
Torch version: 2.5.0+cu121
Trying to load custom node /path/ComfyUI/custom_nodes/Comfy_KepListStuff
Trying to load custom node /path/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite
Trying to load custom node /path/ComfyUI/custom_nodes/ComfyUI-KJNodes
Total VRAM 22732 MB, total RAM 63719 MB
pytorch version: 2.5.0+cu121
Set vram state to: NORMAL_VRAM
Disabling smart memory management
Device: cuda:0 NVIDIA A10G : cudaMallocAsync
Trying to load custom node /path/ComfyUI/custom_nodes/ComfyUI-Merlin
start_comfy.sh: line 18: 72 Segmentation fault python3 -u ${COMFYUI_PATH}/main.py --listen 0.0.0.0 --port $1 --disable-smart-memory --cuda-device ${gpuIndex} --input-directory ${COMFYUI_INPUT_DIR} --output-directory ${output_dir} --temp-directory ${temp_dir} --verbose
Example logs in instance 2 (success)
Trying to load custom node /path/ComfyUI/custom_nodes/ComfyUI-Merlin
Trying to load custom node /path/ComfyUI/custom_nodes/was-node-suite-comfyui
Trying to load custom node /path/ComfyUI/custom_nodes/ComfyUI-VideoHelperSuite
Trying to load custom node /path/ComfyUI/custom_nodes/comfyui-reactor-node
#033[0;33m[ReActor]#033[0m - #033[38;5;173mSTATUS#033[0m - #033[0;32mRunning v0.5.1-b2 in ComfyUI#033[0m
Torch version: 2.5.0+cu121
Trying to load custom node /path/ComfyUI/custom_nodes/ComfyUI-MimicMotionWrapper
Trying to load custom node /path/ComfyUI/custom_nodes/ComfyUI-LivePortraitKJ
Setting output directory to: /path/ComfyUI/output/0
Setting input directory to: /path/ComfyUI/input
Starting server
To see the GUI go to: http://0.0.0.0:7860
Existing Solutions
I am not very familiar to the ComfyUI architecture, but adding a sorted function will make the custom nodes loading deterministic for different OS and instances.
For example, the init_external_custom_nodes
- node_paths = folder_paths.get_folder_paths("custom_nodes")
+ node_paths = sorted(folder_paths.get_folder_paths("custom_nodes"))
node_import_times = []
for custom_node_path in node_paths:
- possible_modules = os.listdir(os.path.realpath(custom_node_path))
+ possible_modules = sorted(os.listdir(os.path.realpath(custom_node_path)))
Other
No response