AUTOMATIC1111
[Bug]: Face Restoration and --device-id different than 0 results in error
Is there an existing issue for this?
- [X] I have searched the existing issues and checked the recent builds/commits
What happened?
It appears that the Face Restoration portion of the code is ignoring the --device-id flag and defaulting to GPU 0 no matter what is set for --device-id. Testing on instances running --device-id 1 and --device-id 2 result in the same error, except for id 0.
Steps to reproduce the problem
Execute the instance using multiple GPUs and use the flag for selecting GPU other than 0, then attempt to use Face Restoration (Either Codeformer or GFPGAN)
What should have happened?
Face Restore Script should follow --device-id flag and use the selected GPU
Commit where the problem happens
3e15f8e0f5cc87507f77546d92435670644dbd18
What platforms do you use to access UI ?
Windows
What browsers do you use to access the UI ?
Microsoft Edge
Command Line Arguments
-gradio-auth XX:YYYYYYYYYYYYYYYYYYYYY --share --xformers --deepdanbooru --freeze-settings --device-id 1
-gradio-auth XX:YYYYYYYYYYYYYYYYYYYYY --share --xformers --deepdanbooru --freeze-settings --device-id 2
Additional information, context and logs
100%|██████████████████████████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 14.35it/s] Error completing request███████████████████████████████████████████████████████████████| 20/20 [00:01<00:00, 15.01it/s] Arguments: ('test', '', 'None', 'None', 20, 0, True, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 0, 0, 0, 0.9, 5, '0.0001', False, 'None', '', 0.1, False, False, False, None, '', 1, '', 0, '', True, False, False) {} Traceback (most recent call last): File "C:\stable-diffusion-webui-1.5inpainting\modules\ui.py", line 221, in f res = list(func(*args, **kwargs)) File "C:\stable-diffusion-webui-1.5inpainting\webui.py", line 63, in f res = func(*args, **kwargs) File "C:\stable-diffusion-webui-1.5inpainting\modules\txt2img.py", line 48, in txt2img processed = process_images(p) File "C:\stable-diffusion-webui-1.5inpainting\modules\processing.py", line 464, in process_images x_sample = modules.face_restoration.restore_faces(x_sample) File "C:\stable-diffusion-webui-1.5inpainting\modules\face_restoration.py", line 19, in restore_faces return face_restorer.restore(np_image) File "C:\stable-diffusion-webui-1.5inpainting\modules\codeformer_model.py", line 93, in restore self.face_helper.get_face_landmarks_5(only_center_face=False, resize=640, eye_dist_threshold=5) File "C:\stable-diffusion-webui-1.5inpainting\repositories\CodeFormer\facelib\utils\face_restoration_helper.py", line 151, in get_face_landmarks_5 bboxes = self.face_det.detect_faces(input_img) File "C:\stable-diffusion-webui-1.5inpainting\repositories\CodeFormer\facelib\detection\retinaface\retinaface.py", line 209, in detect_faces loc, conf, landmarks, priors = self.__detect_faces(image) File "C:\stable-diffusion-webui-1.5inpainting\repositories\CodeFormer\facelib\detection\retinaface\retinaface.py", line 156, in __detect_faces loc, conf, landmarks = self(inputs) File "C:\stable-diffusion-webui-1.5inpainting\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "C:\stable-diffusion-webui-1.5inpainting\repositories\CodeFormer\facelib\detection\retinaface\retinaface.py", line 121, in forward out = self.body(inputs) File "C:\stable-diffusion-webui-1.5inpainting\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "C:\stable-diffusion-webui-1.5inpainting\venv\lib\site-packages\torchvision\models\_utils.py", line 69, in forward x = module(x) File "C:\stable-diffusion-webui-1.5inpainting\venv\lib\site-packages\torch\nn\modules\module.py", line 1130, in _call_impl return forward_call(*input, **kwargs) File "C:\stable-diffusion-webui-1.5inpainting\venv\lib\site-packages\torch\nn\modules\conv.py", line 457, in forward return self._conv_forward(input, self.weight, self.bias) File "C:\stable-diffusion-webui-1.5inpainting\venv\lib\site-packages\torch\nn\modules\conv.py", line 453, in _conv_forward return F.conv2d(input, weight, bias, self.stride, RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! (when checking argument for argument weight in method wrapper__cudnn_convolution)
The same exact error happens on machine using --device-id 2, except for the very last line:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:2! (when checking argument for argument weight in method wrapper__cudnn_convolution)
Not only face restoration, but hi-res fix and img2img fail to utilize the same device.
This occurs in chrome as well as Firefox. Just adding to the list.
