[bug]: v4.0.0 inpainting RuntimeError: expected scalar type float but found c10::Half

[pravbk100]

Is there an existing issue for this problem?

• [X] I have searched the existing issues

Operating system

macOS

GPU vendor

Apple Silicon (MPS)

GPU model

M1 mac mini

GPU VRAM

16gb

Version number

4.0.0

Browser

safari 17.1.2

Python dependencies

No response

What happened

I am getting

RuntimeError: expected scalar type float but found c10::Half

whenever i try to use unified canvas inpainting or outpainting. It works with SD 1.5 inpainting model but throws this error if i use any other model.

What you expected to happen

inpainting should work

How to reproduce the problem

No response

Additional context

No response

Discord username

No response

[grunblatt-git]

This error seems to be dependent on the scheduler - i get the same error consistently with Euler, Euler a & Heun. I haven't noticed this to behave differently for different models yet.

[psychedelicious]

@grunblatt-git Are you on a mac too?

Assuming you are both on macOS, it's possible that this could be resolved using the PYTORCH_ENABLE_MPS_FALLBACK environment variable. This tells torch to use CPU when a specific operation isn't available on macOS (there are many that are not available).

The invoke.sh script sets this for you, so if you are using that, then we definitely have a bug - possibly introduced by diffusers.

If you run the app using the invokeai-web command, please try running it like this and see if that fixes the issue:

export PYTORCH_ENABLE_MPS_FALLBACK=1; invokeai-web

Paging @Vargol @gogurtenjoyer in case they have run into this on MPS.

[grunblatt-git]

@psychedelicious yes I am on Mac too, however so far I started invoke exclusively with the invoke.sh script anyways. So I guess the torch fallback was already set.

Could you reproduce it already or do you need anything else to troubleshoot this?

[psychedelicious]

Thanks. I haven't tried this out on macOS yet, but given your experience I have no doubt it will be a problem for me too. This kind of issue pops up fairly regularly for MPS unfortunately.

I'm not confident in my ability to resolve this. It could be an issue with diffusers. I see one open issue with the same error message, but it's related to text-to-video: https://github.com/huggingface/diffusers/issues/2785

I'll defer to @Vargol and/or @gogurtenjoyer for better understanding of the problem and if we need to raise this upstream or not.

[Vargol]

My install is a little none standard at the moment, a nightly PyTorch and a quick hack to enable bfloat16 and I've not noticed this issue. I've run a few inpainting tests with SDXL base and Euler that worked okay.

I've revert back to float16 and if it still works I'll knock up 'pure' 4.0rc environment to have a quick test.

[Vargol]

Okay...clean environment I get the error.

SDXL base with Euler scheduler - inpainting.

[2024-03-25 09:53:00,026]::[InvokeAI]::ERROR --> Error while invoking session 56557ff5-78f8-4c83-a00f-825b85bb59f1, invocation 0e4a70d0-44f5-4800-91db-ecddeab0909e (denoise_latents):
expected scalar type float but found c10::Half
[2024-03-25 09:53:00,026]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/app/services/session_processor/session_processor_default.py", line 160, in _process
    outputs = self._invocation.invoke_internal(
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/app/invocations/baseinvocation.py", line 281, in invoke_internal
    output = self.invoke(context)
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/app/invocations/latent.py", line 787, in invoke
    result_latents = pipeline.latents_from_embeddings(
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 354, in latents_from_embeddings
    latents = self.generate_latents_from_embeddings(
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 432, in generate_latents_from_embeddings
    step_output = self.step(
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 580, in step
    step_output.pred_original_sample = guidance(
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 89, in __call__
    return self.apply_mask(latents, t)
  File "/Volumes/SSD2TB/AI/TestInvoke/lib/python3.10/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 107, in apply_mask
    masked_input = torch.where(mask_bool, latents, mask_latents)
RuntimeError: expected scalar type float but found c10::Half

[Vargol]

A bit more testing, it seems to be fixed in whatever version of 2.3.0 is in the test channel (2.2.2 in the same channel is a bust).

So

pip uninstall torch torchvision
pip3 install --pre torch==2.3.0 torchvision --index-url https://download.pytorch.org/whl/test/cpu

should work around the issue if nothing can be done in the apply_mask function.

[pjparkjd]

I am getting the same error after installing the latest version today. I don't know if these additional details will help, but I do launch using invoke.sh. The above fix did not work for me, but switching to the PNDM schedule did.

Description

Attempting to expand an image's aspect ratio using Unified Canvas results in a RuntimeError. This occurs with both the Cyberrealistic_v4.1_FP16 and Juggernaut-XL-v9 models when invoking the model after setting a mask.

Environment

• OS: macOS 14.4.1 (Build 23E224)
• Machine: iMac21,1 with Apple M1 Chip, 8 GB RAM
• Python Version: Python 3.11.6
• Installed Packages:
  • pip 23.2.1
  • protobuf 4.24.4
  • setuptools 68.2.2
  • six 1.16.0
  • wheel 0.41.2

Steps to Reproduce

1. Select "Unified Canvas" mode.
2. Upload an image.
3. Adjust the canvas to a wider width.
4. Set "Base/Mask" dropdown to "Mask".
5. Use the Brush tool to paint over the areas to keep.
6. Click "Invoke".

Expected Behavior

The image should be expanded to the specified aspect ratio without altering the masked area.

Actual Behavior

The process fails with a RuntimeError indicating a type mismatch: "expected scalar type float but found c10::Half".

Error Message

Error Traceback (most recent call last): File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/app/services/session_processor/session_processor_default.py", line 185, in _process outputs = self._invocation.invoke_internal( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/app/invocations/baseinvocation.py", line 281, in invoke_internal output = self.invoke(context) ^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/app/invocations/latent.py", line 940, in invoke result_latents = pipeline.latents_from_embeddings( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 339, in latents_from_embeddings latents = self.generate_latents_from_embeddings( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 419, in generate_latents_from_embeddings step_output = self.step( ^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 551, in step step_output.pred_original_sample = guidance( ^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 88, in __call__ return self.apply_mask(latents, t) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Users/redact/invokeai/.venv/lib/python3.11/site-packages/invokeai/backend/stable_diffusion/diffusers_pipeline.py", line 106, in apply_mask masked_input = torch.where(mask_bool, latents, mask_latents) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: expected scalar type float but found c10::Half

[Kaputmaher]

Same was for M2 Max in Pinokio environment. Was fixed by: cd /Users/admin/pinokio/api/invoke.git/app source env/bin/activate pip uninstall torch torchvision pip install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cpu

[deseven]

Same exact error for me. None of the solutions above worked, I guess nightly and test channels moved forward since then. Maybe some specific version would work, but I'm not sure how to determine that.

M1, macOS Sonoma

[Vargol]

still working here with torch==2.3.1

[deseven]

With 2.3.1 I'm just getting a different kind of error (while simply trying to outpaint a 256x256 image:

[2024-06-24 21:41:44,220]::[InvokeAI]::ERROR --> Error while invoking session a916852f-3483-4481-8935-b6ac92ab2b94, invocation 27260781-24a0-4fa9-9831-fc3551069040 (canvas_paste_back): images do not match
[2024-06-24 21:41:44,220]::[InvokeAI]::ERROR --> Traceback (most recent call last):
  File "/Users/deseven/invokeai/.venv/lib/python3.11/site-packages/invokeai/app/services/session_processor/session_processor_default.py", line 129, in run_node
    output = invocation.invoke_internal(context=context, services=self._services)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/deseven/invokeai/.venv/lib/python3.11/site-packages/invokeai/app/invocations/baseinvocation.py", line 297, in invoke_internal
    return self.invoke(context)
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/deseven/invokeai/.venv/lib/python3.11/site-packages/invokeai/app/invocations/image.py", line 954, in invoke
    source_image.paste(target_image, (0, 0), mask)
  File "/Users/deseven/invokeai/.venv/lib/python3.11/site-packages/PIL/Image.py", line 1760, in paste
    self.im.paste(im, box, mask.im)
ValueError: images do not match

[Vargol]

Thats a different bug, you'll need to raise it as a new issue.

[smile4yourself]

I also can not get the canvas to work; same error "expected scalar type float but found c10::Half; running v4.2.4 on a Mac Ultra, using the install script to recreate the working folder but over-writing the previous version so that I still have my photos in the new version.

[Vargol]

Have you tried using torch 2.3.1 ? from the dev console or activated venv run

pip uninstall torch torchvision
pip install torch torchvision

that should install torch 2.3.1 assuming its supported on you macOS version

Ignore the warning about them being incompatible.

If it doesn't work then the devs will need all the details, what model, scheduler. steps, infill etc.

[smile4yourself]

Thanks so much! That fixed it even though I apparently have other issues. I include my console log fyi below

What would you like to do?
1: Generate images using the browser-based interface
2: Open the developer console
3: Command-line help
Q: Quit

To update, download and run the installer from https://github.com/invoke-ai/InvokeAI/releases/latest.

Please enter 1-4, Q: [1] 2
Open the developer console
Python 3.10.14
Press ^D to exit

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.

(InvokeAI) MAC_at_home@Mac-Studio ~/invoke4> pip uninstall torch torchvision
Found existing installation: torch 2.2.2
Uninstalling torch-2.2.2:
  Would remove:
    /Users/MAC_at_home/invoke4/.venv/bin/convert-caffe2-to-onnx
    /Users/MAC_at_home/invoke4/.venv/bin/convert-onnx-to-caffe2
    /Users/MAC_at_home/invoke4/.venv/bin/torchrun
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/functorch/*
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torch-2.2.2.dist-info/*
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torch/*
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torchgen/*
  Would not remove (might be manually added):
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torch/.DS_Store
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torch/include/.DS_Store
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torch/include/ATen/.DS_Store
Proceed (Y/n)? Y

  Successfully uninstalled torch-2.2.2
  
Found existing installation: torchvision 0.17.2
Uninstalling torchvision-0.17.2:
  Would remove:
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torchvision-0.17.2.dist-info/*
    /Users/MAC_at_home/invoke4/.venv/lib/python3.11/site-packages/torchvision/*
Proceed (Y/n)? y

  Successfully uninstalled torchvision-0.17.2
  
(InvokeAI) MAC_at_home@Mac-Studio ~/invoke4> pip install torch torchvision

Collecting torch
  Downloading torch-2.3.1-cp311-none-macosx_11_0_arm64.whl.metadata (26 kB)
Collecting torchvision
  Downloading torchvision-0.18.1-cp311-cp311-macosx_11_0_arm64.whl.metadata (6.6 kB)
Requirement already satisfied: filelock in ./.venv/lib/python3.11/site-packages (from torch) (3.15.1)
Requirement already satisfied: typing-extensions>=4.8.0 in ./.venv/lib/python3.11/site-packages (from torch) (4.12.2)
Requirement already satisfied: sympy in ./.venv/lib/python3.11/site-packages (from torch) (1.12.1)
Requirement already satisfied: networkx in ./.venv/lib/python3.11/site-packages (from torch) (3.3)
Requirement already satisfied: jinja2 in ./.venv/lib/python3.11/site-packages (from torch) (3.1.4)
Requirement already satisfied: fsspec in ./.venv/lib/python3.11/site-packages (from torch) (2024.3.1)
Requirement already satisfied: numpy in ./.venv/lib/python3.11/site-packages (from torchvision) (1.26.4)
Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in ./.venv/lib/python3.11/site-packages (from torchvision) (10.3.0)
Requirement already satisfied: MarkupSafe>=2.0 in ./.venv/lib/python3.11/site-packages (from jinja2->torch) (2.1.5)
Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in ./.venv/lib/python3.11/site-packages (from sympy->torch) (1.3.0)
Downloading torch-2.3.1-cp311-none-macosx_11_0_arm64.whl (61.0 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.0/61.0 MB 61.9 MB/s eta 0:00:00
Downloading torchvision-0.18.1-cp311-cp311-macosx_11_0_arm64.whl (1.6 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 49.9 MB/s eta 0:00:00
Installing collected packages: torch, torchvision

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
invokeai 4.2.4 requires torch==2.2.2, but you have torch 2.3.1 which is incompatible.
invokeai 4.2.4 requires torchvision==0.17.2, but you have torchvision 0.18.1 which is incompatible.

Successfully installed torch-2.3.1 torchvision-0.18.1

[notice] A new release of pip is available: 24.1 -> 24.1.1
[notice] To update, run: pip install --upgrade pip

(InvokeAI) MAC_at_home@Mac-Studio ~/invoke4>

[hsm207]

I had the same issue too after upgrading my developer install to commit 35f8781ea22cccbbbb6fab92180ee2b7bca35a70.

Upgrading to pytorch 2.3.1 from 2.2.2 fixed it for me.

[smile4yourself]

After upgrading invokeAI to 4.2.6post1 on the MAC, this error came back. I resolved it the same way as you suggested before,

pip uninstall torch torchvision
pip install torch torchvision

regards

[magnusviri]

I had to install 2.3 (which is not the latest). The latest didn't error out but it produced garbage.

pip uninstall torch torchvision
pip install torch==2.3 torchvision

[smile4yourself]

this error came back for me after installing v4.2.6 and then 4.2.6post1; I got the same runtime error trying to inpaint, reinstalled torch as per above, but this time pip reinstalled torch v 2.4.0 -- and invoke now fails to create promt-to-image and produces junk with inpainting. I tried again: uninstall and then 'pip install torch==2.3.1 torchvision' and torchvision 0.18.1 was loaded along with torch 2.3.1. ; Inpainting is working now, as well as prompt-to-image. It looks to me that invoke won't work on a MAC unless these versions are used...

[hsm207]

It looks to me that invoke won't work on a MAC unless these versions are used...

I'm on Mac M1 and Invoke 4.2.7. I'm using torch 2.4.0 and torchvision 0.19.0. I have no issues inpainting and prompt-to-image so far.