ComfyUI
ComfyUI copied to clipboard
Published
2 weeks ago
•
comfyanonymous
comfyanonymous
nunchaku OOM after update to new ComfyUI
Custom Node Testing
- [ ] I have tried disabling custom nodes and the issue persists (see how to disable custom nodes if you need help)
Expected Behavior
It used to work fine. But after I update comfyui to latest and also update custom nodes. it always shows OOM
Actual Behavior
A Nunchaku workflow which it works perfectly but now it shows OOM
Steps to Reproduce
just run the workflow and it errors
Debug Logs
# ComfyUI Error Report
## Error Details
- **Node ID:** 23
- **Node Type:** NunchakuQwenImageDiTLoader
- **Exception Type:** torch.OutOfMemoryError
- **Exception Message:** Allocation on device
This error means you ran out of memory on your GPU.
TIPS: If the workflow worked before you might have accidentally set the batch_size to a large number.
## Stack Trace
File "/home/george/ComfyUI/app/execution.py", line 515, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/execution.py", line 329, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/execution.py", line 303, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "/home/george/ComfyUI/app/execution.py", line 291, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/nodes/models/qwenimage.py", line 216, in load_model
model = load_diffusion_model_state_dict(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/nodes/models/qwenimage.py", line 98, in load_diffusion_model_state_dict
model = model_config.get_model(new_sd, "", load_device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/model_configs/qwenimage.py", line 38, in get_model
out = model_base.NunchakuQwenImage(self, device=device, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/model_base/qwenimage.py", line 42, in __init__
super(QwenImage, self).__init__(
File "/home/george/ComfyUI/app/comfy/model_base.py", line 141, in __init__
self.diffusion_model = unet_model(**unet_config, device=device, operations=operations)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/models/qwenimage.py", line 595, in __init__
NunchakuQwenImageTransformerBlock(
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/models/qwenimage.py", line 389, in __init__
self.txt_mlp = NunchakuFeedForward(dim=dim, dim_out=dim, dtype=dtype, device=device, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/models/qwenimage.py", line 119, in __init__
NunchakuGELU(dim, inner_dim, approximate="tanh", bias=bias, dtype=dtype, device=device, **kwargs)
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/models/qwenimage.py", line 70, in __init__
self.proj = SVDQW4A4Linear(dim_in, dim_out, bias=bias, torch_dtype=dtype, device=device, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/venv/lib/python3.12/site-packages/nunchaku/models/linear.py", line 97, in __init__
torch.empty(out_features, in_features // 2, dtype=torch.int8, device=device), requires_grad=False
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
## System Information
- **ComfyUI Version:** 0.3.76
- **Arguments:** main.py --preview-method auto --listen 0.0.0.0 --port 8188
- **OS:** linux
- **Python Version:** 3.12.3 (main, Jun 18 2025, 17:59:45) [GCC 13.3.0]
- **Embedded Python:** false
- **PyTorch Version:** 2.9.1+cu128
## Devices
- **Name:** cuda:0 NVIDIA GeForce RTX 3080 : cudaMallocAsync
- **Type:** cuda
- **VRAM Total:** 21037187072
- **VRAM Free:** 19956039680
- **Torch VRAM Total:** 100663296
- **Torch VRAM Free:** 91095040
## Logs
2025-12-09T13:57:59.370355 - ComfyUI version: 0.3.76
2025-12-09T13:57:59.373883 - ComfyUI frontend version: 1.33.10
2025-12-09T13:57:59.374578 - [Prompt Server] web root: /home/george/ComfyUI/venv/lib/python3.12/site-packages/comfyui_frontend_package/static
2025-12-09T13:57:59.870762 - Total VRAM 20063 MB, total RAM 64203 MB
2025-12-09T13:57:59.870917 - pytorch version: 2.9.1+cu128
2025-12-09T13:57:59.871020 - xformers version: 0.0.33.post2
2025-12-09T13:57:59.871155 - Set vram state to: NORMAL_VRAM
2025-12-09T13:57:59.871276 - Device: cuda:0 NVIDIA GeForce RTX 3080 : cudaMallocAsync
2025-12-09T13:57:59.871413 - Using async weight offloading with 2 streams
2025-12-09T13:57:59.871600 - Enabled pinned memory 60992.0
2025-12-09T13:58:00.734516 - ======================================== ComfyUI-nunchaku Initialization ========================================
2025-12-09T13:58:00.735539 - Nunchaku version: 1.0.2
2025-12-09T13:58:00.736916 - ComfyUI-nunchaku version: 1.0.2
2025-12-09T13:58:03.931956 - Skipping import of cpp extensions due to incompatible torch version 2.9.1+cu128 for torchao version 0.14.0 Please see GitHub issue #2919 for more info
2025-12-09T13:58:17.840415 - /home/george/ComfyUI/venv/lib/python3.12/site-packages/timm/models/layers/__init__.py:48: FutureWarning: Importing from timm.models.layers is deprecated, please import via timm.layers
warnings.warn(f"Importing from {__name__} is deprecated, please import via timm.layers", FutureWarning)
2025-12-09T13:58:17.849161 - =================================================================================================================
2025-12-09T13:58:17.852773 - ComfyUI-GGUF: Allowing full torch compile
2025-12-09T13:58:18.378234 - Eden_Comfy_Pipelines: WARNING: Could not find OPENAI_API_KEY in .env, disabling gpt prompt generation.2025-12-09T13:58:18.378311 -
2025-12-09T13:58:18.425016 -
[32m🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱[0m
[1m🌱 Eden ComfyUI Pack maintained by [32mhttps://eden.art/ 🌱[0m
[32m🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱🌱[0m
2025-12-09T13:58:18.425108 -
2025-12-09T13:58:18.638230 - [[94mComfyUI-Upscaler-Tensorrt[0m|[92mINFO[0m] - [92mSuccessfully loaded configuration from load_upscaler_config.json[0m
2025-12-09T13:58:18.644606 - [92m[tinyterraNodes] [32mLoaded[0m2025-12-09T13:58:18.644684 -
2025-12-09T13:58:19.172095 - [36;20m[/home/george/ComfyUI/app/custom_nodes/comfy-mtb] | INFO -> loaded [96m108[0m nodes successfuly[0m
2025-12-09T13:58:19.176998 - [36;20m[/home/george/ComfyUI/app/custom_nodes/comfyui_controlnet_aux] | INFO -> Using ckpts path: /media/george/AI/ComfyUI_data/custom_nodes/comfyui_controlnet_aux/ckpts[0m
2025-12-09T13:58:19.177282 - [36;20m[/home/george/ComfyUI/app/custom_nodes/comfyui_controlnet_aux] | INFO -> Using symlinks: False[0m
2025-12-09T13:58:19.177545 - [36;20m[/home/george/ComfyUI/app/custom_nodes/comfyui_controlnet_aux] | INFO -> Using ort providers: ['CUDAExecutionProvider', 'DirectMLExecutionProvider', 'OpenVINOExecutionProvider', 'ROCMExecutionProvider', 'CPUExecutionProvider', 'CoreMLExecutionProvider'][0m
2025-12-09T13:58:19.188278 - DWPose: Onnxruntime with acceleration providers detected2025-12-09T13:58:19.188332 -
2025-12-09T13:58:19.197842 - Adding2025-12-09T13:58:19.197917 - 2025-12-09T13:58:19.197958 - /home/george/ComfyUI/app/custom_nodes2025-12-09T13:58:19.197994 - 2025-12-09T13:58:19.198027 - to sys.path2025-12-09T13:58:19.198059 -
2025-12-09T13:58:19.203352 - Could not find efficiency nodes2025-12-09T13:58:19.203403 -
2025-12-09T13:58:19.207048 - [36;20m[comfyui_controlnet_aux] | INFO -> Using ckpts path: /media/george/AI/ComfyUI_data/custom_nodes/comfyui_controlnet_aux/ckpts[0m
2025-12-09T13:58:19.207442 - [36;20m[comfyui_controlnet_aux] | INFO -> Using symlinks: False[0m
2025-12-09T13:58:19.207847 - [36;20m[comfyui_controlnet_aux] | INFO -> Using ort providers: ['CUDAExecutionProvider', 'DirectMLExecutionProvider', 'OpenVINOExecutionProvider', 'ROCMExecutionProvider', 'CPUExecutionProvider', 'CoreMLExecutionProvider'][0m
2025-12-09T13:58:19.222494 - Loaded comfyui_controlnet_aux nodes from2025-12-09T13:58:19.222547 - 2025-12-09T13:58:19.222588 - /media/george/AI/ComfyUI_data/custom_nodes/comfyui_controlnet_aux2025-12-09T13:58:19.222625 -
2025-12-09T13:58:19.223050 - Could not find AdvancedControlNet nodes2025-12-09T13:58:19.223121 -
2025-12-09T13:58:19.224318 - Could not find AnimateDiff nodes2025-12-09T13:58:19.224368 -
2025-12-09T13:58:19.224794 - Could not find IPAdapter nodes2025-12-09T13:58:19.224853 -
2025-12-09T13:58:19.308150 - Loaded VideoHelperSuite from2025-12-09T13:58:19.308217 - 2025-12-09T13:58:19.308259 - /media/george/AI/ComfyUI_data/custom_nodes/ComfyUI-VideoHelperSuite2025-12-09T13:58:19.308297 -
2025-12-09T13:58:19.310005 - ### Loading: ComfyUI-Impact-Pack (V8.22.2)
2025-12-09T13:58:19.433697 - ### Loading: ComfyUI-Impact-Pack (V8.22.2)
2025-12-09T13:58:19.446141 - Loaded ImpactPack nodes from2025-12-09T13:58:19.446277 - 2025-12-09T13:58:19.446323 - /media/george/AI/ComfyUI_data/custom_nodes/comfyui-impact-pack2025-12-09T13:58:19.446364 -
2025-12-09T13:58:19.447052 - [Impact Pack] Wildcards loading done.
2025-12-09T13:58:19.455582 - [Impact Pack] Wildcards loading done.
2025-12-09T13:58:19.456197 - ### Loading: ComfyUI Connection Helper (V0.1)2025-12-09T13:58:19.456249 -
2025-12-09T13:58:19.587745 - Web extensions folder found at /home/george/ComfyUI/app/web/extensions/ComfyLiterals2025-12-09T13:58:19.587819 -
2025-12-09T13:58:19.703541 - [CRT-Nodes __init__] Importing node classes...2025-12-09T13:58:19.703599 -
2025-12-09T13:58:19.705536 - Loading crt-nodes module2025-12-09T13:58:19.705587 -
2025-12-09T13:58:19.714522 - ✅ ControlNet Aux loaded from 'comfyui_controlnet_aux' with 47 preprocessors2025-12-09T13:58:19.714602 -
2025-12-09T13:58:19.733407 - [CRT-Nodes __init__] Registering custom model paths...2025-12-09T13:58:19.733460 -
2025-12-09T13:58:19.733531 - ... 'ultralytics_bbox' path registered.2025-12-09T13:58:19.733571 -
2025-12-09T13:58:19.733621 - ... 'ultralytics_segm' path registered.2025-12-09T13:58:19.733657 -
2025-12-09T13:58:19.733691 - [CRT-Nodes __init__] INITIALIZATION COMPLETE.2025-12-09T13:58:19.733723 -
2025-12-09T13:58:19.949501 - Set up system temp directory: /tmp/latentsync_11c8fbf42025-12-09T13:58:19.949584 -
2025-12-09T13:58:20.171967 - Initializing ComfyUI-VideoUpscale_WithModel2025-12-09T13:58:20.172039 -
2025-12-09T13:58:20.203334 - set VIDEO_TOTAL_PIXELS: 90316800
2025-12-09T13:58:21.004530 - [34mWAS Node Suite: [0mOpenCV Python FFMPEG support is enabled[0m2025-12-09T13:58:21.004618 -
2025-12-09T13:58:21.004711 - [34mWAS Node Suite [93mWarning: [0m`ffmpeg_bin_path` is not set in `/home/george/ComfyUI/app/custom_nodes/was-node-suite-comfyui/was_suite_config.json` config file. Will attempt to use system ffmpeg binaries if available.[0m2025-12-09T13:58:21.004757 -
2025-12-09T13:58:21.679625 - [34mWAS Node Suite: [0mFinished.[0m [32mLoaded[0m [0m220[0m [32mnodes successfully.[0m2025-12-09T13:58:21.679755 -
2025-12-09T13:58:21.679896 -
[3m[93m"Your time is now. Start where you are and never stop."[0m[3m - Roy T. Bennett[0m
2025-12-09T13:58:21.679984 -
2025-12-09T13:58:21.721163 - 📊 Initial VRAM status: 18.79GB free / 19.59GB total2025-12-09T13:58:21.721285 -
2025-12-09T13:58:21.757622 -
2025-12-09T13:58:21.757721 - [92m[rgthree-comfy] Loaded 48 exciting nodes. 🎉[0m2025-12-09T13:58:21.757792 -
2025-12-09T13:58:21.757843 -
2025-12-09T13:58:21.757911 - [33m[rgthree-comfy] ComfyUI's new Node 2.0 rendering may be incompatible with some rgthree-comfy nodes and features, breaking some rendering as well as losing the ability to access a node's properties (a vital part of many nodes). It also appears to run MUCH more slowly spiking CPU usage and causing jankiness and unresponsiveness, especially with large workflows. Personally I am not planning to use the new Nodes 2.0 and, unfortunately, am not able to invest the time to investigate and overhaul rgthree-comfy where needed. If you have issues when Nodes 2.0 is enabled, I'd urge you to switch it off as well and join me in hoping ComfyUI is not planning to deprecate the existing, stable canvas rendering all together.
[0m2025-12-09T13:58:21.757962 -
2025-12-09T13:58:21.786685 - ### Loading: ComfyUI-Manager (V3.38.1)
2025-12-09T13:58:21.787191 - [ComfyUI-Manager] network_mode: public
2025-12-09T13:58:21.787385 - [ComfyUI-Manager] Since --preview-method is set, ComfyUI-Manager's preview method feature will be ignored.
2025-12-09T13:58:21.850746 - ### ComfyUI Version: v0.3.76-51-g3b0368aa | Released on '2025-12-08'
2025-12-09T13:58:21.927951 - ------------------------------------------2025-12-09T13:58:21.928033 -
2025-12-09T13:58:21.928138 - [34mComfyroll Studio v1.76 : [92m 175 Nodes Loaded[0m2025-12-09T13:58:21.928196 -
2025-12-09T13:58:21.928246 - ------------------------------------------2025-12-09T13:58:21.928293 -
2025-12-09T13:58:21.928339 - ** For changes, please see patch notes at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/blob/main/Patch_Notes.md2025-12-09T13:58:21.928383 -
2025-12-09T13:58:21.928427 - ** For help, please see the wiki at https://github.com/Suzie1/ComfyUI_Comfyroll_CustomNodes/wiki2025-12-09T13:58:21.928469 -
2025-12-09T13:58:21.928512 - ------------------------------------------2025-12-09T13:58:21.928553 -
2025-12-09T13:58:22.415922 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/alter-list.json
2025-12-09T13:58:22.438579 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/model-list.json
2025-12-09T13:58:22.468523 - ### Loading: ComfyUI-Impact-Subpack (V1.3.5)
2025-12-09T13:58:22.469668 - [Impact Pack/Subpack] Using folder_paths to determine whitelist path: /home/george/ComfyUI/app/user/default/ComfyUI-Impact-Subpack/model-whitelist.txt
2025-12-09T13:58:22.469791 - [Impact Pack/Subpack] Ensured whitelist directory exists: /home/george/ComfyUI/app/user/default/ComfyUI-Impact-Subpack
2025-12-09T13:58:22.469929 - [Impact Pack/Subpack] Loaded 0 model(s) from whitelist: /home/george/ComfyUI/app/user/default/ComfyUI-Impact-Subpack/model-whitelist.txt
2025-12-09T13:58:22.520317 - [Impact Subpack] ultralytics_bbox: /home/george/ComfyUI/app/models/ultralytics/bbox
2025-12-09T13:58:22.520509 - [Impact Subpack] ultralytics_segm: /home/george/ComfyUI/app/models/ultralytics/segm
2025-12-09T13:58:22.745460 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/extension-node-map.json
2025-12-09T13:58:22.907348 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/custom-node-list.json
2025-12-09T13:58:22.949394 - [ComfyUI-Manager] default cache updated: https://raw.githubusercontent.com/ltdrdata/ComfyUI-Manager/main/github-stats.json
2025-12-09T13:58:22.963654 - [Crystools [0;32mINFO[0m] Crystools version: 1.27.4
2025-12-09T13:58:22.984655 - [Crystools [0;32mINFO[0m] Platform release: 6.8.0-71-generic
2025-12-09T13:58:22.984837 - [Crystools [0;32mINFO[0m] JETSON: Not detected.
2025-12-09T13:58:22.987250 - [Crystools [0;32mINFO[0m] CPU: Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz - Arch: x86_64 - OS: Linux 6.8.0-71-generic
2025-12-09T13:58:22.987549 - [Crystools [0;32mINFO[0m] pynvml (NVIDIA) initialized.
2025-12-09T13:58:22.987849 - [Crystools [0;32mINFO[0m] GPU/s:
2025-12-09T13:58:22.988134 - [Crystools [0;32mINFO[0m] 0) NVIDIA GeForce RTX 3080
2025-12-09T13:58:22.988357 - [Crystools [0;32mINFO[0m] NVIDIA Driver: 570.195.03
2025-12-09T13:58:23.124100 - [34m[ComfyUI-Easy-Use] server: [0mv1.3.4 [92mLoaded[0m2025-12-09T13:58:23.124178 -
2025-12-09T13:58:23.124252 - [34m[ComfyUI-Easy-Use] web root: [0m/media/george/AI/ComfyUI_data/custom_nodes/ComfyUI-Easy-Use/web_version/v2 [92mLoaded[0m2025-12-09T13:58:23.124303 -
2025-12-09T13:58:23.213920 - ### Loading: ComfyUI-Inspire-Pack (V1.21)
2025-12-09T13:58:23.331545 - [FLOAT_Optimized] Registering 24 node(s).2025-12-09T13:58:23.331811 -
2025-12-09T13:58:23.676405 - ======================================== ComfyUI-QwenImageLoraLoader Initialization ========================================
2025-12-09T13:58:23.677302 - [DEBUG] Added to sys.path: /home/george/ComfyUI/app/custom_nodes/ComfyUI-QwenImageLoraLoader2025-12-09T13:58:23.677373 -
2025-12-09T13:58:23.677454 - [DEBUG] wrappers dir exists: True2025-12-09T13:58:23.677517 -
2025-12-09T13:58:23.677590 - [DEBUG] qwenimage.py exists: True2025-12-09T13:58:23.677640 -
2025-12-09T13:58:23.678075 - ============================================================================================================================
2025-12-09T13:58:23.696051 - ### Loading: ComfyUI-Impact-Pack (V8.22.2)
2025-12-09T13:58:23.711098 - [Impact Pack] Wildcards loading done.
2025-12-09T13:58:23.825776 - [MultiGPU Core Patching] Patching mm.soft_empty_cache for Comprehensive Memory Management (VRAM + CPU + Store Pruning)
2025-12-09T13:58:23.827354 - [MultiGPU Core Patching] Patching mm.get_torch_device, mm.text_encoder_device, mm.unet_offload_device
2025-12-09T13:58:23.827678 - [MultiGPU DEBUG] Initial current_device: cuda:0
2025-12-09T13:58:23.827979 - [MultiGPU DEBUG] Initial current_text_encoder_device: cuda:0
2025-12-09T13:58:23.828303 - [MultiGPU DEBUG] Initial current_unet_offload_device: cpu
2025-12-09T13:58:23.831873 - [MultiGPU] Initiating custom_node Registration. . .
2025-12-09T13:58:23.832200 - -----------------------------------------------
2025-12-09T13:58:23.832503 - custom_node Found Nodes
2025-12-09T13:58:23.832794 - -----------------------------------------------
2025-12-09T13:58:23.833986 - ComfyUI-LTXVideo N 0
2025-12-09T13:58:23.834313 - ComfyUI-Florence2 Y 2
2025-12-09T13:58:23.834631 - ComfyUI_bitsandbytes_NF4 N 0
2025-12-09T13:58:23.834934 - x-flux-comfyui N 0
2025-12-09T13:58:23.835254 - ComfyUI-MMAudio N 0
2025-12-09T13:58:23.835555 - ComfyUI-GGUF Y 18
2025-12-09T13:58:23.835855 - PuLID_ComfyUI N 0
2025-12-09T13:58:23.836177 - ComfyUI-WanVideoWrapper Y 20
2025-12-09T13:58:23.836472 - -----------------------------------------------
2025-12-09T13:58:23.836911 - [MultiGPU] Registration complete. Final mappings: CheckpointLoaderAdvancedMultiGPU, CheckpointLoaderAdvancedDisTorch2MultiGPU, UNetLoaderLP, UNETLoaderMultiGPU, VAELoaderMultiGPU, CLIPLoaderMultiGPU, DualCLIPLoaderMultiGPU, TripleCLIPLoaderMultiGPU, QuadrupleCLIPLoaderMultiGPU, CLIPVisionLoaderMultiGPU, CheckpointLoaderSimpleMultiGPU, ControlNetLoaderMultiGPU, DiffusersLoaderMultiGPU, DiffControlNetLoaderMultiGPU, UNETLoaderDisTorch2MultiGPU, VAELoaderDisTorch2MultiGPU, CLIPLoaderDisTorch2MultiGPU, DualCLIPLoaderDisTorch2MultiGPU, TripleCLIPLoaderDisTorch2MultiGPU, QuadrupleCLIPLoaderDisTorch2MultiGPU, CLIPVisionLoaderDisTorch2MultiGPU, CheckpointLoaderSimpleDisTorch2MultiGPU, ControlNetLoaderDisTorch2MultiGPU, DiffusersLoaderDisTorch2MultiGPU, DiffControlNetLoaderDisTorch2MultiGPU, Florence2ModelLoaderMultiGPU, DownloadAndLoadFlorence2ModelMultiGPU, UnetLoaderGGUFDisTorchMultiGPU, UnetLoaderGGUFAdvancedDisTorchMultiGPU, CLIPLoaderGGUFDisTorchMultiGPU, DualCLIPLoaderGGUFDisTorchMultiGPU, TripleCLIPLoaderGGUFDisTorchMultiGPU, QuadrupleCLIPLoaderGGUFDisTorchMultiGPU, UnetLoaderGGUFDisTorch2MultiGPU, UnetLoaderGGUFAdvancedDisTorch2MultiGPU, CLIPLoaderGGUFDisTorch2MultiGPU, DualCLIPLoaderGGUFDisTorch2MultiGPU, TripleCLIPLoaderGGUFDisTorch2MultiGPU, QuadrupleCLIPLoaderGGUFDisTorch2MultiGPU, UnetLoaderGGUFMultiGPU, UnetLoaderGGUFAdvancedMultiGPU, CLIPLoaderGGUFMultiGPU, DualCLIPLoaderGGUFMultiGPU, TripleCLIPLoaderGGUFMultiGPU, QuadrupleCLIPLoaderGGUFMultiGPU, LoadWanVideoT5TextEncoderMultiGPU, WanVideoTextEncodeMultiGPU, WanVideoTextEncodeCachedMultiGPU, WanVideoTextEncodeSingleMultiGPU, WanVideoVAELoaderMultiGPU, WanVideoTinyVAELoaderMultiGPU, WanVideoBlockSwapMultiGPU, WanVideoImageToVideoEncodeMultiGPU, WanVideoDecodeMultiGPU, WanVideoModelLoaderMultiGPU, WanVideoSamplerMultiGPU, WanVideoVACEEncodeMultiGPU, WanVideoEncodeMultiGPU, LoadWanVideoClipTextEncoderMultiGPU, WanVideoClipVisionEncodeMultiGPU, WanVideoControlnetLoaderMultiGPU, FantasyTalkingModelLoaderMultiGPU, Wav2VecModelLoaderMultiGPU, WanVideoUni3C_ControlnetLoaderMultiGPU, DownloadAndLoadWav2VecModelMultiGPU
2025-12-09T13:58:23.838605 -
Import times for custom nodes:
2025-12-09T13:58:23.838748 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/websocket_image_save.py
2025-12-09T13:58:23.838840 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/tiled_ksampler
2025-12-09T13:58:23.838922 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-WanVideoKsampler
2025-12-09T13:58:23.839006 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-VideoUpscale_WithModel
2025-12-09T13:58:23.839081 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfy-image-saver
2025-12-09T13:58:23.839172 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-MakeFrame
2025-12-09T13:58:23.839243 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/UltimateResupscaler
2025-12-09T13:58:23.839312 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyLiterals
2025-12-09T13:58:23.839379 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui-image-selector
2025-12-09T13:58:23.839447 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-HunyuanVideoMultiLora
2025-12-09T13:58:23.839514 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/wanblockswap
2025-12-09T13:58:23.839581 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui-mxtoolkit
2025-12-09T13:58:23.839648 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/Comfyui-QwenEditUtils
2025-12-09T13:58:23.839714 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-Detail-Daemon
2025-12-09T13:58:23.839782 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-LatentSyncWrapper
2025-12-09T13:58:23.839848 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/connection-helper
2025-12-09T13:58:23.839914 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI_Qwen3-VL-Instruct-abliterated
2025-12-09T13:58:23.839980 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui-lora-auto-trigger-words
2025-12-09T13:58:23.840046 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/cg-use-everywhere
2025-12-09T13:58:23.840127 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-AutoCropFaces
2025-12-09T13:58:23.840202 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-QwenVL
2025-12-09T13:58:23.840270 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI_AceNodes
2025-12-09T13:58:23.840337 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-GGUF
2025-12-09T13:58:23.840403 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI_FaceAnalysis
2025-12-09T13:58:23.840483 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-QwenImageLoraLoader
2025-12-09T13:58:23.840551 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-WanAnimatePreprocess
2025-12-09T13:58:23.840619 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-Chibi-Nodes
2025-12-09T13:58:23.840686 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/teacache
2025-12-09T13:58:23.840753 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui-florence2
2025-12-09T13:58:23.840819 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui-various
2025-12-09T13:58:23.840886 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui_essentials
2025-12-09T13:58:23.840951 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui_patches_ll
2025-12-09T13:58:23.841017 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-Custom-Scripts
2025-12-09T13:58:23.841081 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-Image-Saver
2025-12-09T13:58:23.841162 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui-frame-interpolation
2025-12-09T13:58:23.841228 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/derfuu_comfyui_moddednodes
2025-12-09T13:58:23.841305 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-segment-anything-2
2025-12-09T13:58:23.841373 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui-dream-project
2025-12-09T13:58:23.841438 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI_UltimateSDUpscale
2025-12-09T13:58:23.841502 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-KJNodes
2025-12-09T13:58:23.841567 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-MultiGPU
2025-12-09T13:58:23.841637 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui-seedvr2-tilingupscaler
2025-12-09T13:58:23.841706 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui-impact-pack
2025-12-09T13:58:23.841771 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/gguf
2025-12-09T13:58:23.841835 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/rgthree-comfy
2025-12-09T13:58:23.841900 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui_controlnet_aux
2025-12-09T13:58:23.841964 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-MelBandRoFormer
2025-12-09T13:58:23.842028 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/facerestore_cf
2025-12-09T13:58:23.842101 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI_Searge_LLM
2025-12-09T13:58:23.842172 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI_QwenVL
2025-12-09T13:58:23.842237 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/CRT-Nodes
2025-12-09T13:58:23.842302 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI_Comfyroll_CustomNodes
2025-12-09T13:58:23.842366 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui_tinyterranodes
2025-12-09T13:58:23.842429 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/Comfyui_TTP_Toolset
2025-12-09T13:58:23.842493 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/Comfyui-SecNodes
2025-12-09T13:58:23.842593 - 0.0 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-FLOAT_Optimized
2025-12-09T13:58:23.842665 - 0.1 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-Impact-Subpack
2025-12-09T13:58:23.842731 - 0.1 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-SeedVR2_VideoUpscaler-nightly
2025-12-09T13:58:23.842797 - 0.1 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-Inspire-Pack
2025-12-09T13:58:23.842862 - 0.1 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-MediaMixer
2025-12-09T13:58:23.842934 - 0.1 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-VideoHelperSuite
2025-12-09T13:58:23.843000 - 0.1 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui-ollama
2025-12-09T13:58:23.843066 - 0.1 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-LogicUtils
2025-12-09T13:58:23.843147 - 0.1 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui_layerstyle
2025-12-09T13:58:23.843212 - 0.1 seconds: /home/george/ComfyUI/app/custom_nodes/Vantage-HunyuanFoley
2025-12-09T13:58:23.843275 - 0.1 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui-manager
2025-12-09T13:58:23.843338 - 0.1 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-Easy-Use
2025-12-09T13:58:23.843402 - 0.2 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-SUPIR
2025-12-09T13:58:23.843467 - 0.2 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-Upscaler-Tensorrt
2025-12-09T13:58:23.843532 - 0.3 seconds: /home/george/ComfyUI/app/custom_nodes/comfyui-art-venture
2025-12-09T13:58:23.843597 - 0.3 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-WanVideoWrapper
2025-12-09T13:58:23.843661 - 0.5 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-Crystools
2025-12-09T13:58:23.843724 - 0.5 seconds: /home/george/ComfyUI/app/custom_nodes/comfy-mtb
2025-12-09T13:58:23.843790 - 0.5 seconds: /home/george/ComfyUI/app/custom_nodes/ComfyUI-GIMM-VFI
2025-12-09T13:58:23.843853 - 0.6 seconds: /home/george/ComfyUI/app/custom_nodes/eden_comfy_pipelines
2025-12-09T13:58:23.843919 - 1.5 seconds: /home/george/ComfyUI/app/custom_nodes/was-node-suite-comfyui
2025-12-09T13:58:23.843984 - 17.1 seconds: /home/george/ComfyUI/app/custom_nodes/nunchaku_nodes
2025-12-09T13:58:23.844061 -
2025-12-09T13:58:24.130104 - Context impl SQLiteImpl.
2025-12-09T13:58:24.130256 - Will assume non-transactional DDL.
2025-12-09T13:58:24.131197 - No target revision found.
2025-12-09T13:58:24.181441 - Starting server
2025-12-09T13:58:24.181797 - To see the GUI go to: http://0.0.0.0:8188
2025-12-09T13:58:40.438918 - FETCH ComfyRegistry Data: 5/1102025-12-09T13:58:40.439108 -
2025-12-09T13:58:50.677812 - FETCH ComfyRegistry Data: 10/1102025-12-09T13:58:50.677981 -
2025-12-09T13:58:58.861538 - FETCH ComfyRegistry Data: 15/1102025-12-09T13:58:58.861651 -
2025-12-09T13:59:00.804602 - got prompt
2025-12-09T13:59:00.833229 - WARNING: [Errno 2] No such file or directory: '/home/george/ComfyUI/app/input/Screenshot_20251031-105424.png'
2025-12-09T13:59:00.847247 - WARNING: [Errno 2] No such file or directory: '/home/george/ComfyUI/app/output/Others/concat/uncloth_concat_00041_.png'
2025-12-09T13:59:00.847811 - WARNING: [Errno 2] No such file or directory: '/home/george/ComfyUI/app/output/Others/concat/uncloth_concat_00042_.png'
2025-12-09T13:59:00.848344 - WARNING: [Errno 2] No such file or directory: '/home/george/ComfyUI/app/output/Others/concat/uncloth_concat_00043_.png'
2025-12-09T13:59:00.848888 - WARNING: [Errno 2] No such file or directory: '/home/george/ComfyUI/app/output/Others/concat/uncloth_concat_00044_.png'
2025-12-09T13:59:00.849416 - WARNING: [Errno 2] No such file or directory: '/home/george/ComfyUI/app/output/Others/concat/uncloth_concat_00046_.png'
2025-12-09T13:59:00.849920 - WARNING: [Errno 2] No such file or directory: '/home/george/ComfyUI/app/output/Others/concat/uncloth_concat_00047_.png'
2025-12-09T13:59:00.859642 - WAS Node Suite: 2025-12-09T13:59:00.859905 - Batch 0012025-12-09T13:59:00.860190 - Index: 92025-12-09T13:59:00.860353 -
2025-12-09T13:59:01.366904 - Using xformers attention in VAE
2025-12-09T13:59:01.368886 - Using xformers attention in VAE
2025-12-09T13:59:01.518070 - VAE load device: cuda:0, offload device: cpu, dtype: torch.bfloat16
2025-12-09T13:59:01.533281 - Requested to load WanVAE
2025-12-09T13:59:01.617691 - loaded completely; 6534.85 MB usable, 242.03 MB loaded, full load: True
2025-12-09T13:59:11.880506 - FETCH ComfyRegistry Data: 20/1102025-12-09T13:59:11.881283 -
2025-12-09T13:59:13.680876 - gguf qtypes: Q8_0 (198), F32 (141)2025-12-09T13:59:13.680964 -
2025-12-09T13:59:14.176968 - Using mmproj 'Qwen2.5-VL-7B-Instruct-abliterated.mmproj-Q8_0.gguf' for 'qwen2.5-vl-7b-instruct-abliterated.q8_0.gguf'.2025-12-09T13:59:14.177046 -
2025-12-09T13:59:14.246312 - gguf qtypes: F32 (291), Q8_0 (194), F16 (34)2025-12-09T13:59:14.246384 -
2025-12-09T13:59:14.462698 - [MultiGPU Core Patching] text_encoder_device_patched returning device: cuda:0 (current_text_encoder_device=cuda:0)
2025-12-09T13:59:14.941437 - CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
2025-12-09T13:59:15.333668 - Requested to load QwenImageTEModel_
2025-12-09T13:59:17.718092 - loaded completely; 17421.71 MB usable, 9699.04 MB loaded, full load: True
2025-12-09T13:59:20.212880 - VRAM > 15GiB, disabling CPU offload
2025-12-09T13:59:20.218878 -
WARNING, you accessed scaled_fp8 from the model config object which doesn't exist. Please fix your code.
2025-12-09T13:59:20.345065 - !!! Exception during processing !!! Allocation on device
2025-12-09T13:59:20.347443 - Traceback (most recent call last):
File "/home/george/ComfyUI/app/execution.py", line 515, in execute
output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/execution.py", line 329, in get_output_data
return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, v3_data=v3_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/execution.py", line 303, in _async_map_node_over_list
await process_inputs(input_dict, i)
File "/home/george/ComfyUI/app/execution.py", line 291, in process_inputs
result = f(**inputs)
^^^^^^^^^^^
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/nodes/models/qwenimage.py", line 216, in load_model
model = load_diffusion_model_state_dict(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/nodes/models/qwenimage.py", line 98, in load_diffusion_model_state_dict
model = model_config.get_model(new_sd, "", load_device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/model_configs/qwenimage.py", line 38, in get_model
out = model_base.NunchakuQwenImage(self, device=device, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/model_base/qwenimage.py", line 42, in __init__
super(QwenImage, self).__init__(
File "/home/george/ComfyUI/app/comfy/model_base.py", line 141, in __init__
self.diffusion_model = unet_model(**unet_config, device=device, operations=operations)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/models/qwenimage.py", line 595, in __init__
NunchakuQwenImageTransformerBlock(
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/models/qwenimage.py", line 389, in __init__
self.txt_mlp = NunchakuFeedForward(dim=dim, dim_out=dim, dtype=dtype, device=device, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/models/qwenimage.py", line 119, in __init__
NunchakuGELU(dim, inner_dim, approximate="tanh", bias=bias, dtype=dtype, device=device, **kwargs)
File "/home/george/ComfyUI/app/custom_nodes/nunchaku_nodes/models/qwenimage.py", line 70, in __init__
self.proj = SVDQW4A4Linear(dim_in, dim_out, bias=bias, torch_dtype=dtype, device=device, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/george/ComfyUI/venv/lib/python3.12/site-packages/nunchaku/models/linear.py", line 97, in __init__
torch.empty(out_features, in_features // 2, dtype=torch.int8, device=device), requires_grad=False
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: Allocation on device
2025-12-09T13:59:20.347574 - Got an OOM, unloading all loaded models.
2025-12-09T13:59:22.517294 - FETCH ComfyRegistry Data: 25/1102025-12-09T13:59:22.517525 -
2025-12-09T13:59:26.874517 - Prompt executed in 26.05 seconds
## Attached Workflow
Please make sure that workflow does not contain any sensitive information such as API keys or passwords.
Workflow too large. Please manually upload the workflow from local file system.
## Additional Context
(Please add any additional context or steps to reproduce the error here)
Other
No response
Downgrade Python version to 3.12 in docker container and update cuda to 130 resolve this problem for me