ControlNets cause ` RuntimeError: Expected all tensors to be on the same device`

[DragonMarionette]

Here is the exact error:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

Running the latest commits on windows 11 with a 3060 Ti, using a local install with a python 11 venv. I get errors like this every time I try to use a ControlNet, including with the example workflows like this one. I have run some of these exact workflows successfully before, as recently as a week ago, so the breaking change must have happened since then.

Error occurs with all ControlNets, including 1.0, 1.1, and the fp16 ones. But t2i adapters still seem to be working. So my guess was that ControlNets in particular are getting loaded onto my CPU even though there's room on the GPU. Except that gpu-only and vram flags haven't helped, so I'm stumped.

Full traceback:

!!! Exception during processing !!!
Traceback (most recent call last):
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\execution.py", line 145, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\execution.py", line 75, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\execution.py", line 68, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\nodes.py", line 1036, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 174, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 297, in sample
    samples = sampler.sample(noise, positive_copy, negative_copy, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\comfy\samplers.py", line 617, in sample
    samples = uni_pc.sample_unipc(self.model_wrap, noise, latent_image, sigmas, sampling_function=sampling_function, max_denoise=max_denoise, extra_args=extra_args, noise_mask=denoise_mask, callback=callback, disable=disable_pbar)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\comfy\extra_samplers\uni_pc.py", line 882, in sample_unipc
    x = uni_pc.sample(img, timesteps=timesteps, skip_type="time_uniform", method="multistep", order=order, lower_order_final=True, callback=callback, disable_pbar=disable)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\comfy\extra_samplers\uni_pc.py", line 731, in sample
    model_prev_list = [self.model_fn(x, vec_t)]
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\comfy\extra_samplers\uni_pc.py", line 422, in model_fn
    return self.data_prediction_fn(x, t)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\comfy\extra_samplers\uni_pc.py", line 404, in data_prediction_fn
    noise = self.noise_prediction_fn(x, t)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\comfy\extra_samplers\uni_pc.py", line 398, in noise_prediction_fn
    return self.model(x, t)
           ^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\comfy\extra_samplers\uni_pc.py", line 330, in model_fn
    return noise_pred_fn(x, t_continuous)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\comfy\extra_samplers\uni_pc.py", line 298, in noise_pred_fn
    output = sampling_function(model, x, t_input, **model_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\comfy\samplers.py", line 274, in sampling_function
    cond, uncond = calc_cond_uncond_batch(model_function, cond, uncond, x, timestep, max_total_area, cond_concat, model_options)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\comfy\samplers.py", line 232, in calc_cond_uncond_batch
    c['control'] = control.get_control(input_x, timestep_, c, len(cond_or_uncond))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\custom_nodes\AIT\AITemplate\AITemplate.py", line 388, in get_control
    control = self.control_model(x=x_noisy, hint=self.cond_hint, timesteps=t, context=context, y=y)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\comfy\cldm\cldm.py", line 290, in forward
    emb = self.time_embed(t_emb)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\venv\Lib\site-packages\torch\nn\modules\container.py", line 217, in forward
    input = module(input)
            ^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\venv\Lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\CreativeWorks\Graphics\AI\ComfyUI\comfy\ops.py", line 18, in forward
    return torch.nn.functional.linear(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

[comfyanonymous]

This is an issue with the AI template node you are using, make sure you update both comfy and it to the latest and it most likely will work.

[DragonMarionette]

It happens even in workflows that contain no AIT nodes at all. But I see that part of the traceback now that you point it out. And I have just updated both but the issue still occurs. But if I uninstall AIT, the exact same workflows work again. Why might built-in nodes call something inside the AIT folder during sampling?

[DragonMarionette]

Likely an issue with a different repo, but even if not hopefully fixed by commit 46dc050