Instruct pix2pix RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.HalfTensor) should be the same

[SuperMaximus1984]

When I run InstructPix2Pix_cuda:0 I receive an error RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.HalfTensor) should be the same Whenver I run it InstructPix2Pix_cpu everything works well, but slow.

I have torch 2 with CUDA, GroundingDINO installed with GPU. How do I fix the RuntimeError and run InstructPix2Pix_cuda:0?

[shugerdou]

same here

[liujiabin20810]

same problem.

Entering new AgentExecutor chain... Action: Segment the Image Action Input: image/c4ca40ac.png Observation: image/02c3_segment-image_c4ca40ac_c4ca40ac.png Thought:Action: Remove Something From The Photo Action Input: image/02c3_segment-image_c4ca40ac_c4ca40ac.png, small brown dogimage_path=image/02c3_segment-image_c4ca40ac_c4ca40ac.png, to_be_replaced_txt= small brown dog 100% 50/50 [00:10<00:00, 4.86it/s] Traceback (most recent call last): File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 437, in run_predict output = await app.get_blocks().process_api( File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1352, in process_api result = await self.call_function( File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1077, in call_function prediction = await anyio.to_thread.run_sync( File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread return await future File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 807, in run result = context.run(func, *args) File "/content/TaskMatrix/visual_chatgpt.py", line 1520, in run_text res = self.agent({"input": text.strip()}) File "/usr/local/lib/python3.10/dist-packages/langchain/chains/base.py", line 168, in call raise e File "/usr/local/lib/python3.10/dist-packages/langchain/chains/base.py", line 165, in call outputs = self._call(inputs) File "/usr/local/lib/python3.10/dist-packages/langchain/agents/agent.py", line 503, in _call next_step_output = self._take_next_step( File "/usr/local/lib/python3.10/dist-packages/langchain/agents/agent.py", line 420, in _take_next_step observation = tool.run( File "/usr/local/lib/python3.10/dist-packages/langchain/tools/base.py", line 71, in run raise e File "/usr/local/lib/python3.10/dist-packages/langchain/tools/base.py", line 68, in run observation = self._run(tool_input) File "/usr/local/lib/python3.10/dist-packages/langchain/agents/tools.py", line 17, in _run return self.func(tool_input) File "/content/TaskMatrix/visual_chatgpt.py", line 1384, in inference_remove return self.inference_replace_sam(f"{image_path},{to_be_removed_txt},background") File "/content/TaskMatrix/visual_chatgpt.py", line 1408, in inference_replace_sam updated_image = self.inpaint(prompt=replace_with_txt, image=image_pil, File "/content/TaskMatrix/visual_chatgpt.py", line 1180, in call update_image = self.inpaint(prompt=prompt, image=image.resize((width, height)), File "/usr/local/lib/python3.10/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py", line 1056, in call image, has_nsfw_concept = self.run_safety_checker(image, device, prompt_embeds.dtype) File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py", line 536, in run_safety_checker image, has_nsfw_concept = self.safety_checker( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion/safety_checker.py", line 51, in forward pooled_output = self.vision_model(clip_input)[1] # pooled_output File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/clip/modeling_clip.py", line 940, in forward return self.vision_model( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/clip/modeling_clip.py", line 865, in forward hidden_states = self.embeddings(pixel_values) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(input, **kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/clip/modeling_clip.py", line 195, in forward patch_embeds = self.patch_embedding(pixel_values) # shape = [, width, grid, grid] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 463, in forward return self._conv_forward(input, self.weight, self.bias) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 459, in _conv_forward return F.conv2d(input, weight, bias, self.stride, RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.cuda.FloatTensor) should be the same

[yuanjiechen]

Use search and replace tools to replace all torch.float16 to torch.float32!

[SuperMaximus1984]

Use search and replace tools to replace all torch.float16 to torch.float32! In this file? pipeline_stable_diffusion_inpaint.py

[stipplelabs]

Setting self.torch_dtype = torch.float32 instead self.torch_dtype = torch.float16 if 'cuda' in device else torch.float32 in all model class initialization @ visual_chatgpt.py worked on RTX 3090 CUDA 11.7!