[Bug]: NansException: A tensor with all NaNs was produced in VAE.

[alexbespik]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

I was generating picture and after 10-30 pictures i have an error d70>, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, None, False, 50) {} Traceback (most recent call last): File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/call_queue.py", line 57, in f res = list(func(*args, **kwargs)) File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/call_queue.py", line 37, in f res = func(*args, **kwargs) File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/txt2img.py", line 56, in txt2img processed = process_images(p) File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/processing.py", line 515, in process_images res = process_images_inner(p) File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/batch_hijack.py", line 42, in processing_process_images_hijack return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs) File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/processing.py", line 673, in process_images_inner devices.test_for_nans(x, "vae") File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/devices.py", line 156, in test_for_nans raise NansException(message) modules.devices.NansException: A tensor with all NaNs was produced in VAE. Use --disable-nan-check commandline argument to disable this check.

Can i swithc off vae? Tryed to reboot program an system

Steps to reproduce the problem

1. Go to ....
2. Press ....
3. ...

What should have happened?

6 pictures i tried to move to other commits but it did not help

Commit where the problem happens

2c550e4d89

What platforms do you use to access the UI ?

Linux

What browsers do you use to access the UI ?

Mozilla Firefox

Command Line Arguments

--theme dark --precision full --no-half --medvram --xformers --autolaunch --no-half-vae

List of extensions

Console logs

/usr/bin/xdg-open: line 613: 18718 Aborted                 (core dumped) kde-open${KDE_SESSION_VERSION} "$1"
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [01:55<00:00,  3.85s/it]
Error completing request███████████████████████████▎                                                                                                                                                       | 30/180 [01:46<09:14,  3.70s/it]
Arguments: ('task(8xkxahmw5q58k33)', 'painted forest , by Ivan Shishkin,lake,ultra realistic,(detailed:1.6).8k, (Bears in forest:1.5) ,  photorealism bears , bears by  by Ivan Shishkin, move bears to background, Living forest, animals , (owl on tree:1.6), muted colors, hyperealism ', '(deformed, distorted, disfigured:1.3), poorly drawn, bad anatomy, wrong anatomy, extra limb, missing limb, floating limbs, (mutated hands and fingers:1.4), disconnected limbs, mutation, mutated, ugly, disgusting, blurry, amputation, mutated bear, bad bear , deformed bear', [], 30, 16, False, False, 6, 1, 6, 3398830535.0, -1.0, 0, 0, 0, False, 600, 760, False, 0.7, 2, 'Latent', 0, 0, 0, [], 0, 'Send to Canvas Editor', <controlnet.py.UiControlNetUnit object at 0x7fe36cf42140>, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, None, False, 50) {}
Traceback (most recent call last):
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/call_queue.py", line 57, in f
    res = list(func(*args, **kwargs))
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/txt2img.py", line 56, in txt2img
    processed = process_images(p)
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/processing.py", line 515, in process_images
    res = process_images_inner(p)
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/batch_hijack.py", line 42, in processing_process_images_hijack
    return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/processing.py", line 673, in process_images_inner
    devices.test_for_nans(x, "vae")
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/devices.py", line 156, in test_for_nans
    raise NansException(message)
modules.devices.NansException: A tensor with all NaNs was produced in VAE. Use --disable-nan-check commandline argument to disable this check.

100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [01:49<00:00,  3.64s/it]
Error completing request█████████████████████████████████████████████████████████▋                                                                                                                         | 60/180 [04:42<07:23,  3.70s/it]
Arguments: ('task(5pg5tv5skidlywk)', 'painted forest , by Ivan Shishkin,lake,ultra realistic,(detailed:1.6).8k, (Bears in forest:1.5) ,  photorealism bears , bears by  by Ivan Shishkin, move bears to background, Living forest, animals , (owl on tree:1.6), muted colors, hyperealism ', '(deformed, distorted, disfigured:1.3), poorly drawn, bad anatomy, wrong anatomy, extra limb, missing limb, floating limbs, (mutated hands and fingers:1.4), disconnected limbs, mutation, mutated, ugly, disgusting, blurry, amputation, mutated bear, bad bear , deformed bear', [], 30, 16, False, False, 6, 1, 6, 3398830535.0, -1.0, 0, 0, 0, False, 600, 760, False, 0.7, 2, 'Latent', 0, 0, 0, [], 0, 'Send to Canvas Editor', <controlnet.py.UiControlNetUnit object at 0x7fe36cefd8a0>, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, None, False, 50) {}
Traceback (most recent call last):
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/call_queue.py", line 57, in f
    res = list(func(*args, **kwargs))
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/txt2img.py", line 56, in txt2img
    processed = process_images(p)
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/processing.py", line 515, in process_images
    res = process_images_inner(p)
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/batch_hijack.py", line 42, in processing_process_images_hijack
    return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/processing.py", line 673, in process_images_inner
    devices.test_for_nans(x, "vae")
  File "/run/media/alexbespik/e8df4068-7043-49ee-928b-ecb0cf9e68fb/Stable Diffusion/stable-diffusion-webui/modules/devices.py", line 156, in test_for_nans
    raise NansException(message)
modules.devices.NansException: A tensor with all NaNs was produced in VAE. Use --disable-nan-check commandline argument to disable this check.

Additional information

Linux - manjaro GPU - Nvidia 1650 Cpu - i-5 10300h

[alexbespik]

the bug temporarily goes away for a couple of dozen pictures if you generate one picture on another model

[Sakura-Luna]

Does the problem still occur when you set VAE to none?

[xxaier]

Does the problem still occur when you set VAE to none?

still error, I use mac M2

Traceback (most recent call last):
  File "/Users/z/git/stable-diffusion-webui/modules/call_queue.py", line 57, in f
    res = list(func(*args, **kwargs))
  File "/Users/z/git/stable-diffusion-webui/modules/call_queue.py", line 37, in f
    res = func(*args, **kwargs)
  File "/Users/z/git/stable-diffusion-webui/modules/txt2img.py", line 56, in txt2img
    processed = process_images(p)
  File "/Users/z/git/stable-diffusion-webui/modules/processing.py", line 526, in process_images
    res = process_images_inner(p)
  File "/Users/z/git/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/batch_hijack.py", line 42, in processing_process_images_hijack
    return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
  File "/Users/z/git/stable-diffusion-webui/modules/processing.py", line 680, in process_images_inner
    samples_ddim = p.sample(conditioning=c, unconditional_conditioning=uc, seeds=seeds, subseeds=subseeds, subseed_strength=p.subseed_strength, prompts=prompts)
  File "/Users/z/git/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/hook.py", line 252, in process_sample
    return process.sample_before_CN_hack(*args, **kwargs)
  File "/Users/z/git/stable-diffusion-webui/modules/processing.py", line 907, in sample
    samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
  File "/Users/z/git/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 377, in sample
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "/Users/z/git/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 251, in launch_sampling
    return func()
  File "/Users/z/git/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 377, in <lambda>
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
  File "/Users/z/git/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Users/z/git/stable-diffusion-webui/repositories/k-diffusion/k_diffusion/sampling.py", line 594, in sample_dpmpp_2m
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "/Users/z/git/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/z/git/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 167, in forward
    devices.test_for_nans(x_out, "unet")
  File "/Users/z/git/stable-diffusion-webui/modules/devices.py", line 156, in test_for_nans
    raise NansException(message)
modules.devices.NansException: A tensor with all NaNs was produced in Unet. This could be either because there's not enough precision to represent the picture, or because
 your video card does not support half type. Try setting the "Upcast cross attention layer to float32" option in Settings > Stable Diffusion or using the --no-half comman
dline argument to fix this. Use --disable-nan-check commandline argument to disable this check.

[Sakura-Luna]

still error, I use mac M2

Your problem is different from this one. Try --no-half.

[Arnaud3013]

Same issue here, maybe --no-half work but it's like 7x slower... no other solution? edit: i've tried using lora inside prompt and not in extension, using sdp, and --disable-nan-check in order to not break when i launch many generation (black image sometimes but with previous parameter, seems good)

[2blackbar]

--no half will reduce your speed by half, this is not the way, you didnt bought expensive gpu to use half its power... Heres the fix i found , works 100% of the time https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/12292

[catboxanon]

There was quite a bit of rework done for this in 1.6.0, and should have fixed some edge cases. If the issue still persists and there is a consistent way to reproduce it, feel free to open a new issue.