Nvidia Cuda with 8GB of RAM gives out of memory error

[katze42]

How do I have to modify the settings so that it runs on my RTX3060ti card, because now I get OOM? It is a 8GB Nvidia Card, capable to run CUDA.

I'm calling Kandinsky with this script: https://gist.github.com/DanPli/0bfcaed38ed678adf3cd3f6aa0420f46

I get the following errors:

(venv) G:\k\Kandinsky>gen.py --prompt "green car" G:\k\Kandinsky\venv\lib\site-packages\huggingface_hub\file_download.py:637: FutureWarning: cached_download is the legacy way to download files from the HF hub, please consider upgrading to hf_hub_download warnings.warn( making attention of type 'vanilla' with 512 in_channels making attention of type 'vanilla' with 512 in_channels making attention of type 'vanilla' with 512 in_channels Working with z of shape (1, 4, 32, 32) = 4096 dimensions. 0%| | 0/30 [00:01<?, ?it/s] Traceback (most recent call last): File "G:\k\Kandinsky\gen.py", line 66, in generate(**vars(parser.parse_args())) File "G:\k\Kandinsky\gen.py", line 34, in generate images = model.generate_text2img( File "G:\k\Kandinsky\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\kandinsky2_1_model.py", line 341, in generate_text2img return self.generate_img( File "G:\k\Kandinsky\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\kandinsky2_1_model.py", line 247, in generate_img samples = diffusion.p_sample_loop( File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\model\gaussian_diffusion.py", line 413, in p_sample_loop for sample in self.p_sample_loop_progressive( File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\model\gaussian_diffusion.py", line 466, in p_sample_loop_progressive out = self.p_sample( File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\model\gaussian_diffusion.py", line 369, in p_sample out = self.p_mean_variance( File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\model\respace.py", line 102, in p_mean_variance return super().p_mean_variance(self._wrap_model(model), *args, **kwargs) File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\model\gaussian_diffusion.py", line 251, in p_mean_variance model_output = model(x, s_t, **model_kwargs) File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\model\respace.py", line 133, in call return self.model(x, new_ts, **kwargs) File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\kandinsky2_1_model.py", line 225, in model_fn model_out = self.model(combined, ts, **kwargs) File "G:\k\Kandinsky\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\model\text2im_model2_1.py", line 95, in forward h = module(h, emb, xf_out) File "G:\k\Kandinsky\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\model\unet.py", line 42, in forward x = layer(x, encoder_out) File "G:\k\Kandinsky\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\model\unet.py", line 265, in forward h = self.attention(qkv, encoder_out) File "G:\k\Kandinsky\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, **kwargs) File "G:\k\Kandinsky\venv\lib\site-packages\kandinsky2\model\unet.py", line 338, in forward weight = torch.softmax(weight.float(), dim=-1).type(weight.dtype) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 506.00 MiB (GPU 0; 8.00 GiB total capacity; 6.83 GiB already allocated; 0 bytes free; 6.95 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[kodxana]

I think Kandinsky do not have any optimisation yet so you might need use smaller image size instead.

[nonetrix]

Would be nice to see optimization, I don't think much extra is needed to get it to just barley fit on 8GBs of VRAM nicely seems to just use slightly over 8GBs

[trolley813]

However, it works well with 512x512 (768x768 is too large for 8GB card, not only for Kandinsky - it doesn't work with SD too AFAIK). Maybe generate a smaller image and then upscale it with a suitable upscaler (it depends on the generated image type)?

[XmYx]

try installing my fork in whatever ui you are using (I suggest aiNodes of course : ) ), it'll work under 6Gb and do 768's

[pablonieto0981]

try installing my fork in whatever ui you are using (I suggest aiNodes of course : ) ), it'll work under 6Gb and do 768's

Looking at your fork. Quick question: what adjustments did you do to make it fit in 6Gb? Thanks for your guidance.

[nonetrix]

I was able to do SDXL at 1024x1024 when I had my 8GB GTX 1080 without much of a sweat, I think getting this model working still might be doable perhaps. I have since upgraded my GPU to RX 6800 with 16 GBs of VRAM so hopefully that works, but running it on AMD might require some trial and error