Не работает на CPU

[MichaelMonashev]

Code to reproduce:

from kandinsky2 import get_kandinsky2

model = get_kandinsky2('cpu', task_type='text2img', cache_dir='/tmp/kandinsky2', model_version='2.1', use_flash_attention=False)
images = model.generate_text2img("red cat, 4k photo", num_steps=100, batch_size=1, guidance_scale=4, h=256, w=256, sampler='p_sampler', prior_cf_scale=4, prior_steps="5")

error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/xxx/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/xxx/.local/lib/python3.8/site-packages/kandinsky2/kandinsky2_1_model.py", line 315, in generate_text2img
    image_emb = self.generate_clip_emb(
  File "/home/xxx/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/michael/.local/lib/python3.8/site-packages/kandinsky2/kandinsky2_1_model.py", line 168, in generate_clip_emb
    img_feat = self.prior(
  File "/home/xxx/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxx/.local/lib/python3.8/site-packages/kandinsky2/model/prior.py", line 373, in forward
    sample = sample_fn(
  File "/home/xxx/.local/lib/python3.8/site-packages/kandinsky2/model/gaussian_diffusion.py", line 413, in p_sample_loop
    for sample in self.p_sample_loop_progressive(
  File "/home/xxx/.local/lib/python3.8/site-packages/kandinsky2/model/gaussian_diffusion.py", line 466, in p_sample_loop_progressive
    out = self.p_sample(
  File "/home/xxx/.local/lib/python3.8/site-packages/kandinsky2/model/gaussian_diffusion.py", line 369, in p_sample
    out = self.p_mean_variance(
  File "/home/xxx/.local/lib/python3.8/site-packages/kandinsky2/model/respace.py", line 102, in p_mean_variance
    return super().p_mean_variance(self._wrap_model(model), *args, **kwargs)
  File "/home/xxx/.local/lib/python3.8/site-packages/kandinsky2/model/gaussian_diffusion.py", line 251, in p_mean_variance
    model_output = model(x, s_t, **model_kwargs)
  File "/home/xxx/.local/lib/python3.8/site-packages/kandinsky2/model/respace.py", line 133, in __call__
    return self.model(x, new_ts, **kwargs)
  File "/home/xxx/.local/lib/python3.8/site-packages/kandinsky2/model/prior.py", line 354, in guided_model_fn
    model_out = self.model(combined, ts, **kwargs)
  File "/home/xxx/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxx/.local/lib/python3.8/site-packages/kandinsky2/model/prior.py", line 240, in forward
    t_emb = self.time_embed(
  File "/home/xxx/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxx/.local/lib/python3.8/site-packages/torch/nn/modules/container.py", line 204, in forward
    input = module(input)
  File "/home/xxx/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/xxx/.local/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: "addmm_impl_cpu_" not implemented for 'Half'

[pablonieto0981]

Hi, yesterday faced this issue, and solved it with a hack. Basically you should go into the unet.py (of the kandinsky library) and change self.use_fp16 = use_fp16 to self.use_fp16 = False #use_fp16; also you might probably need to chnage, in the conv.py of PyTorch library...

return F.conv2d(input, weight, bias, self.stride, self.padding, self.dilation, self.groups)

to

return F.conv2d(input.float(), weight, bias, self.stride, self.padding, self.dilation, self.groups)

...in both cases the problem is that some weights of the model are changed to fp16, which is only accepted by the GPU, thus raising an error. .float() makes them fp32, which are CPU acceptable.

[eddir]

For everyone having issues with running Kantinsky-2 on CPU here is a docker configuration that works for me.

[hdnh2006]

self.use_fp16

Hello, thanks for your contribution, I tried it, I changed the code you suggested and I think you need to do some extra changes: In kandinsky2_1_model.py:

        self.device = "cpu" #device
        self.use_fp16 = False #self.config["model_config"]["use_fp16"]

In configs.py:

        "use_fp16": False,

But unfortunately, there's a part of the model that is still being loaded on GPU:

And each image takes around 30 minutes to be generated:

Hope this helps.