kijai
Add Holocine Structured Prompt & Shot Attention Support
Recreated from #1566 #1566, rebased to align with Upstream.
For those who has installed previously - Please delete the previous main and pull this instead.
@Dango233 Hello, may I ask what this prompt means: "hot attention enabled: running without shot embedding."
Hey, I would really like to test it and thanks for your continous work on it. But I get an error with your version of the wrapper, with the kj wrapper main branch, the problem does not exist. (also: the previous pr worked for me in the same environment (python3.12.10, torch 2.7.1., Cuda 12.8))
2025-11-02 15:04:21.273 [Debug] [ComfyUI-0/STDERR] ------- Scheduler info ------- 2025-11-02 15:04:21.322 [Debug] [ComfyUI-0/STDERR] Total timesteps: tensor([1000.0000, 952.6630, 883.2125, 771.4307, 561.6498, 24.4141], device='cuda:0') 2025-11-02 15:04:21.324 [Debug] [ComfyUI-0/STDERR] Using timesteps: tensor([1000.0000, 952.6630, 883.2125], device='cuda:0') 2025-11-02 15:04:21.325 [Debug] [ComfyUI-0/STDERR] Using sigmas: tensor([1.0000, 0.9527, 0.8832, 0.7714], device='cuda:0') 2025-11-02 15:04:21.325 [Debug] [ComfyUI-0/STDERR] ------------------------------ 2025-11-02 15:04:21.327 [Debug] [ComfyUI-0/STDERR] sigmas: tensor([1.0000, 0.9527, 0.8832, 0.7714], device='cuda:0') 2025-11-02 15:04:21.699 [Debug] [ComfyUI-0/STDERR] Input sequence length: 32760 2025-11-02 15:04:21.699 [Debug] [ComfyUI-0/STDERR] Sampling 81 frames at 480x832 with 3 steps 2025-11-02 15:04:22.018 [Debug] [ComfyUI-0/STDERR] 2025-11-02 15:04:24.802 [Debug] [ComfyUI-0/STDERR] 0%| | 0/3 [00:00<?, ?it/s]Error during model prediction: cannot unpack non-iterable NoneType object 2025-11-02 15:04:25.279 [Debug] [ComfyUI-0/STDERR] 2025-11-02 15:04:25.279 [Debug] [ComfyUI-0/STDERR] 0%| | 0/3 [00:03<?, ?it/s] 2025-11-02 15:04:25.279 [Debug] [ComfyUI-0/STDERR] Error during sampling: cannot unpack non-iterable NoneType object 2025-11-02 15:04:25.281 [Debug] [ComfyUI-0/STDOUT] FETCH ComfyRegistry Data: 60/103 2025-11-02 15:04:25.670 [Debug] [ComfyUI-0/STDERR] !!! Exception during processing !!! cannot unpack non-iterable NoneType object 2025-11-02 15:04:25.677 [Warning] [ComfyUI-0/STDERR] Traceback (most recent call last): 2025-11-02 15:04:25.678 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 510, in execute 2025-11-02 15:04:25.679 [Warning] [ComfyUI-0/STDERR] output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs) 2025-11-02 15:04:25.679 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-11-02 15:04:25.679 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 324, in get_output_data 2025-11-02 15:04:25.680 [Warning] [ComfyUI-0/STDERR] return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs) 2025-11-02 15:04:25.680 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-11-02 15:04:25.681 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 298, in _async_map_node_over_list 2025-11-02 15:04:25.681 [Warning] [ComfyUI-0/STDERR] await process_inputs(input_dict, i) 2025-11-02 15:04:25.682 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 286, in process_inputs 2025-11-02 15:04:25.682 [Warning] [ComfyUI-0/STDERR] result = f(**inputs) 2025-11-02 15:04:25.683 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^ 2025-11-02 15:04:25.683 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper-Multishot\nodes_sampler.py", line 3261, in process 2025-11-02 15:04:25.684 [Warning] [ComfyUI-0/STDERR] raise e 2025-11-02 15:04:25.684 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper-Multishot\nodes_sampler.py", line 3146, in process 2025-11-02 15:04:25.685 [Warning] [ComfyUI-0/STDERR] noise_pred, noise_pred_ovi, self.cache_state = predict_with_cfg( 2025-11-02 15:04:25.685 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^ 2025-11-02 15:04:25.686 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper-Multishot\nodes_sampler.py", line 1768, in predict_with_cfg 2025-11-02 15:04:25.686 [Warning] [ComfyUI-0/STDERR] raise e 2025-11-02 15:04:25.687 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper-Multishot\nodes_sampler.py", line 1618, in predict_with_cfg 2025-11-02 15:04:25.687 [Warning] [ComfyUI-0/STDERR] noise_pred_cond, noise_pred_ovi, cache_state_cond = transformer( 2025-11-02 15:04:25.688 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^ 2025-11-02 15:04:25.688 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl 2025-11-02 15:04:25.689 [Warning] [ComfyUI-0/STDERR] return self._call_impl(*args, **kwargs) 2025-11-02 15:04:25.689 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-11-02 15:04:25.691 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1762, in _call_impl 2025-11-02 15:04:25.691 [Warning] [ComfyUI-0/STDERR] return forward_call(*args, **kwargs) 2025-11-02 15:04:25.692 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-11-02 15:04:25.692 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper-Multishot\wanvideo\modules\model.py", line 3383, in forward 2025-11-02 15:04:25.693 [Warning] [ComfyUI-0/STDERR] x, x_ip, lynx_ref_feature, x_ovi = block(x, x_ip=x_ip, lynx_ref_feature=lynx_ref_feature, x_ovi=x_ovi, **kwargs) #run block 2025-11-02 15:04:25.693 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-11-02 15:04:25.694 [Warning] [ComfyUI-0/STDERR] TypeError: cannot unpack non-iterable NoneType object 2025-11-02 15:04:25.694 [Warning] [ComfyUI-0/STDERR]
Probelm fixed.
@Dango233 Hello, may I ask what this prompt means: "hot attention enabled: running without shot embedding."
it's a debug info - if a ckpt comes with shot embedding it will use it. Holocine released ckpt does not have that but code has it - so I implemented just in case they release ckpts with shot embedding
Loading transformer parameters to cuda:0: 100%|███████████████████████████████████| 1095/1095 [00:07<00:00, 140.43it/s] Using 1053 LoRA weight patches for WanVideo model Input sequence length: 126360 Sampling 321 frames at 832x480 with 5 steps 0%| | 0/5 [00:00<?, ?it/s]Error during model prediction: self and mat2 must have the same dtype, but got Float and Half 0%| | 0/5 [00:02<?, ?it/s] Error during sampling: self and mat2 must have the same dtype, but got Float and Half !!! Exception during processing !!! self and mat2 must have the same dtype, but got Float and Half Traceback (most recent call last): File "E:\ComfyUI_windows_portable\ComfyUI\execution.py", line 510, in execute output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\execution.py", line 324, in get_output_data return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\execution.py", line 298, in _async_map_node_over_list await process_inputs(input_dict, i) File "E:\ComfyUI_windows_portable\ComfyUI\execution.py", line 286, in process_inputs result = f(**inputs) ^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 3261, in process raise e File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 3146, in process noise_pred, noise_pred_ovi, self.cache_state = predict_with_cfg( ^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 1768, in predict_with_cfg raise e File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 1618, in predict_with_cfg noise_pred_cond, noise_pred_ovi, cache_state_cond = transformer( ^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\wanvideo\modules\model.py", line 3387, in forward x, x_ip, lynx_ref_feature, x_ovi = block(x, x_ip=x_ip, lynx_ref_feature=lynx_ref_feature, x_ovi=x_ovi, **kwargs) #run block ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_dynamo\eval_frame.py", line 375, in call return super().call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_dynamo\eval_frame.py", line 736, in compile_wrapper return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\wanvideo\modules\model.py", line 1235, in forward q, k, v = self.self_attn.qkv_fn(input_x) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\wanvideo\modules\model.py", line 469, in qkv_fn q = self.norm_q(self.q(x).to(self.norm_q.weight.dtype)).to(x.dtype).view(b, s, n, d) ^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1773, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1784, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\custom_linear.py", line 107, in forward weight = self.apply_lora(weight).to(self.compute_dtype) File "E:\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\custom_linear.py", line 107, in torch_dynamo_resume_in_forward_at_107 weight = self.apply_lora(weight).to(self.compute_dtype) File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_dynamo\eval_frame.py", line 929, in _fn return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_functorch\aot_autograd.py", line 1241, in forward return compiled_fn(full_args) ^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_functorch_aot_autograd\runtime_wrappers.py", line 384, in runtime_wrapper all_outs = call_func_at_runtime_with_args( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_functorch_aot_autograd\utils.py", line 126, in call_func_at_runtime_with_args out = normalize_as_list(f(args)) ^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_functorch_aot_autograd\runtime_wrappers.py", line 750, in inner_fn outs = compiled_fn(args) ^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_functorch_aot_autograd\runtime_wrappers.py", line 556, in wrapper return compiled_fn(runtime_args) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_inductor\output_code.py", line 584, in call return self.current_callable(inputs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch_inductor\utils.py", line 2716, in run out = model(new_inputs) ^^^^^^^^^^^^^^^^^ File "E:\ComfyUI_windows_portable\temp\torchinductor_admin\5o\c5o465fsiajxgf3k75dsy2kh2ahcnjidecfkbzddpraerz4ev4kz.py", line 89, in call extern_kernels.addmm(arg1_1, reinterpret_tensor(arg2_1, (126360, 5120), (5120, 1), 0), reinterpret_tensor(buf0, (5120, 5120), (1, 5120), 0), alpha=1, beta=1, out=buf1) RuntimeError: self and mat2 must have the same dtype, but got Float and Half
Prompt executed in 12.94 seconds
after updated met error
Oh hmmm. Try not to use torch compile for now. I'll look at if we can use compile at all.
I've tried to recreate the scene with the painter from Holocines project page. with 241 (15sec) frames in 720x408 but encountered an error. I disabled torch.
23:29:22.643 [Info] Sent Comfy backend direct prompt requested to backend #0 (from user local) 23:29:31.049 [Warning] [ComfyUI-0/STDERR] Traceback (most recent call last): 23:29:31.050 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 510, in execute 23:29:31.050 [Warning] [ComfyUI-0/STDERR] output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs) 23:29:31.051 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.051 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 324, in get_output_data 23:29:31.052 [Warning] [ComfyUI-0/STDERR] return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs) 23:29:31.052 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.053 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 298, in _async_map_node_over_list 23:29:31.053 [Warning] [ComfyUI-0/STDERR] await process_inputs(input_dict, i) 23:29:31.054 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 286, in process_inputs 23:29:31.054 [Warning] [ComfyUI-0/STDERR] result = f(**inputs) 23:29:31.055 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^ 23:29:31.055 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 3261, in process 23:29:31.056 [Warning] [ComfyUI-0/STDERR] raise e 23:29:31.056 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 3146, in process 23:29:31.057 [Warning] [ComfyUI-0/STDERR] noise_pred, noise_pred_ovi, self.cache_state = predict_with_cfg( 23:29:31.057 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^ 23:29:31.058 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 1768, in predict_with_cfg 23:29:31.058 [Warning] [ComfyUI-0/STDERR] raise e 23:29:31.059 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 1618, in predict_with_cfg 23:29:31.059 [Warning] [ComfyUI-0/STDERR] noise_pred_cond, noise_pred_ovi, cache_state_cond = transformer( 23:29:31.060 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^ 23:29:31.060 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl 23:29:31.060 [Warning] [ComfyUI-0/STDERR] return self._call_impl(*args, **kwargs) 23:29:31.061 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.061 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1762, in _call_impl 23:29:31.062 [Warning] [ComfyUI-0/STDERR] return forward_call(*args, **kwargs) 23:29:31.063 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.064 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\wanvideo\modules\model.py", line 3387, in forward 23:29:31.065 [Warning] [ComfyUI-0/STDERR] x, x_ip, lynx_ref_feature, x_ovi = block(x, x_ip=x_ip, lynx_ref_feature=lynx_ref_feature, x_ovi=x_ovi, **kwargs) #run block 23:29:31.065 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.066 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl 23:29:31.066 [Warning] [ComfyUI-0/STDERR] return self.call_impl(*args, **kwargs) 23:29:31.067 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.068 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1762, in call_impl 23:29:31.068 [Warning] [ComfyUI-0/STDERR] return forward_call(*args, **kwargs) 23:29:31.069 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.069 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\wanvideo\modules\model.py", line 1237, in forward 23:29:31.070 [Warning] [ComfyUI-0/STDERR] q, k = apply_rope_comfy(q, k, freqs) 23:29:31.070 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.071 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\wanvideo\modules\model.py", line 129, in apply_rope_comfy 23:29:31.071 [Warning] [ComfyUI-0/STDERR] xq_out = freqs_cis[..., 0] * xq[..., 0] + freqs_cis[..., 1] * xq[..., 1] 23:29:31.072 [Warning] [ComfyUI-0/STDERR] ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~ 23:29:31.073 [Warning] [ComfyUI-0/STDERR] RuntimeError: The size of tensor a (71370) must match the size of tensor b (69998) at non-singleton dimension 1 23:29:31.073 [Warning] [ComfyUI-0/STDERR]
I've tried to recreate the scene with the painter from Holocines project page. with 241 (15sec) frames in 720x408 but encountered an error. I disabled torch.
23:29:22.643 [Info] Sent Comfy backend direct prompt requested to backend #0 (from user local) 23:29:31.049 [Warning] [ComfyUI-0/STDERR] Traceback (most recent call last): 23:29:31.050 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 510, in execute 23:29:31.050 [Warning] [ComfyUI-0/STDERR] output_data, output_ui, has_subgraph, has_pending_tasks = await get_output_data(prompt_id, unique_id, obj, input_data_all, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs) 23:29:31.051 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.051 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 324, in get_output_data 23:29:31.052 [Warning] [ComfyUI-0/STDERR] return_values = await _async_map_node_over_list(prompt_id, unique_id, obj, input_data_all, obj.FUNCTION, allow_interrupt=True, execution_block_cb=execution_block_cb, pre_execute_cb=pre_execute_cb, hidden_inputs=hidden_inputs) 23:29:31.052 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.053 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 298, in _async_map_node_over_list 23:29:31.053 [Warning] [ComfyUI-0/STDERR] await process_inputs(input_dict, i) 23:29:31.054 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\ComfyUI\execution.py", line 286, in process_inputs 23:29:31.054 [Warning] [ComfyUI-0/STDERR] result = f(**inputs) 23:29:31.055 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^ 23:29:31.055 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 3261, in process 23:29:31.056 [Warning] [ComfyUI-0/STDERR] raise e 23:29:31.056 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 3146, in process 23:29:31.057 [Warning] [ComfyUI-0/STDERR] noise_pred, noise_pred_ovi, self.cache_state = predict_with_cfg( 23:29:31.057 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^ 23:29:31.058 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 1768, in predict_with_cfg 23:29:31.058 [Warning] [ComfyUI-0/STDERR] raise e 23:29:31.059 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\nodes_sampler.py", line 1618, in predict_with_cfg 23:29:31.059 [Warning] [ComfyUI-0/STDERR] noise_pred_cond, noise_pred_ovi, cache_state_cond = transformer( 23:29:31.060 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^ 23:29:31.060 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl 23:29:31.060 [Warning] [ComfyUI-0/STDERR] return self._call_impl(*args, **kwargs) 23:29:31.061 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.061 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1762, in _call_impl 23:29:31.062 [Warning] [ComfyUI-0/STDERR] return forward_call(*args, **kwargs) 23:29:31.063 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.064 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\wanvideo\modules\model.py", line 3387, in forward 23:29:31.065 [Warning] [ComfyUI-0/STDERR] x, x_ip, lynx_ref_feature, x_ovi = block(x, x_ip=x_ip, lynx_ref_feature=lynx_ref_feature, x_ovi=x_ovi, **kwargs) #run block 23:29:31.065 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.066 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl 23:29:31.066 [Warning] [ComfyUI-0/STDERR] return self.call_impl(*args, **kwargs) 23:29:31.067 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.068 [Warning] [ComfyUI-0/STDERR] File "d:\BILD-AI\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1762, in call_impl 23:29:31.068 [Warning] [ComfyUI-0/STDERR] return forward_call(*args, **kwargs) 23:29:31.069 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.069 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\wanvideo\modules\model.py", line 1237, in forward 23:29:31.070 [Warning] [ComfyUI-0/STDERR] q, k = apply_rope_comfy(q, k, freqs) 23:29:31.070 [Warning] [ComfyUI-0/STDERR] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 23:29:31.071 [Warning] [ComfyUI-0/STDERR] File "D:\BILD-AI\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-WanVideoWrapper\wanvideo\modules\model.py", line 129, in apply_rope_comfy 23:29:31.071 [Warning] [ComfyUI-0/STDERR] xq_out = freqs_cis[..., 0] * xq[..., 0] + freqs_cis[..., 1] * xq[..., 1] 23:29:31.072 [Warning] [ComfyUI-0/STDERR] ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~ 23:29:31.073 [Warning] [ComfyUI-0/STDERR] RuntimeError: The size of tensor a (71370) must match the size of tensor b (69998) at non-singleton dimension 1 23:29:31.073 [Warning] [ComfyUI-0/STDERR]
Would you share your workflow - I can't replicate...
Dango233, thank you for your implementation, it is competent. WanVideo VRAM Management node not working. Can you fix it in the Kijai Wrapper?
issue with lora , it was fixed on main branch , but for now toggle merge lora ON
In latest version, the error is as follows: assert dtype in [torch.float16, torch.bfloat16], "Input tensors must be in dtype of torch.float16 or torch.bfloat16" with Merge Loras is ON. This can be resolved by modifying model.py and adding explicit conversion to half precision: q, k, v = q.half(), k.half(), v.half()
Can multiple different context windows adopt multiple rounds of inference? One-time inference consumes a lot of memory.
issue with lora , it was fixed on main branch , but for now toggle merge lora ON
thanks for the suggestion, but at least with the lora loader in my workflow it did not change the result still getting the error. meanwhile I've installed a fresh comfyui python 3.12.10 , cuda=12.8 , pytorch 2.8
@a3nima thanks for sharing your workflow, I trialed it and got the same error but changing the resolution to the one in the root holocine project fixed it for me at least (480x832).
I sincerely hope that kijai can merge it into the main branch, because holoCine can truly narrow the gap with closed-source models. plz look at this:
https://github.com/user-attachments/assets/1ef729e0-aa11-4bf1-943a-140331f0e7a2
If applying a lora I need to use "base_precision" fp32 for the WanVideo Model Loader , otherwise I get
RuntimeError: self and mat2 must have the same dtype, but got Float and Half
Since Kijai is swamped with other things, I did a small workaround to use HoloCine but also Bindweave and the other updates. I downloaded your repo and deleted the node names in the python node files except the loader (renamed it with HoloCine Loader), the sampler (also renamed it), and the four HoloCine nodes. In the init, I deleted most lines except for the model_loader_nodes and the sampler_nodes. Now, it works for HoloCine and everything else too. Btw.: Had to add the line mentioned above and change the resolution to make it work - q, k, v = q.half(), k.half(), v.half()
@railep can you put the working holocine repo up for trying?
Here you go: https://github.com/railep/ComfyUI-WanVideoWrapper-Multishot Clone the repo into your custom nodes. Ignore the warning about two identical repos (they should not do anything to each other since I lobotomized the repo and deleted every node except for the nodes for HoloCine - and they are renamed accordingly, so there are technically two WanVideoWrapper Sampler but one is called HoloCine and works with HoloCine), use the workflow from example_workflows and it should work.
I was too busy these weeks but i can tidy it up as soon as I have some time...
@railep The memory usage is too high. I used to fit the example fully in VRAM with Dango's repo.
The float/half issue is also present by default
To those experiencing issues with the workflow make the following changes
model loader base precision - > fp32 quantization -> fp8 scaled
(I question the default setting of the text encoder since its at fp8 not scaled)
then if you have a larger video card like myself I have the rtx 6000 pro blackwell , set block swaps to 0 blocks to swap.
I did 6 steps -> 832x480 @241 frames, the video generated according to the prompts but it seems a bit soft with grainy artifacts that could be sharper not sure if it's because i need to gen at higher resolutions , also i noticed the enhance node but it's tied to teacache which as far as I'm aware of is only compatible with wan 2.1 , unsure how to proceed. Lastly the motion of the subject almost seems too fast or unnatural such as taking a jacket off or a hat it just peels off like a fruit rollup.
@Dango233 any chance we can have some multigpu support here , I modded my computer to house a rtx5090 along with a rtx 6000 pro blackwell 96gb vram , on a second note the first pass in the ksampler seems to be insanely long like 15 minutes , while the second sampler takes about 70-120 seconds which makes me wonder what the hell is the computer working on in that span of time I am working with the default number of scenes making each one 5 seconds FYI.
I was too busy these weeks but i can tidy it up as soon as I have some time...
Hey Dango, you think you can make this ready to become part of the wrapper? It would be so nice 🤩
@Dango233 is this project dead ? Also can you go over how your nodes allow holocine to work but a regular t2v workflow isn't that great ? Trying to wrap my head around the mechanics outside of lora issues applying to the entire scene which makes sense to split those out per segment
It's not dead... It's just me too busy with my day job.
I'll try catch up with the issues and sort them.
It's not dead... It's just me too busy with my day job.
I'll try catch up with the issues and sort them.
that would be really awesome!
