Getting "CUDA" errors on SD Extension on Mac After f797c07 Update

[jet3004]

Hi, since the f797c07 update, (now that I got the tab to show back up) all generations (that were working perfectly before) are erroring out..You had helped me with non-CUDA support before, like for Mac users, and so I thought maybe something changed in last code that rendered it non-working now...Also, not sure if relevant but do we still need to be adding --no-gradio-queue and --disable-safe-unpickle arguments?

My error:

/Users/jet/stable-diffusion-webui/extensions/SadTalker/checkpoints/auido2pose_00140-model.pth /Users/jet/stable-diffusion-webui/extensions/SadTalker/checkpoints/shape_predictor_68_face_landmarks.dat Traceback (most recent call last): File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/routes.py", line 394, in run_predict output = await app.get_blocks().process_api( File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/blocks.py", line 1075, in process_api result = await self.call_function( File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/gradio/blocks.py", line 884, in call_function prediction = await anyio.to_thread.run_sync( File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, *args) File "/Users/jet/stable-diffusion-webui/modules/call_queue.py", line 15, in f res = func(*args, **kwargs) File "/Users/jet/stable-diffusion-webui/extensions/SadTalker/src/gradio_demo.py", line 69, in test self.preprocess_model = CropAndExtract(self.path_of_lm_croper, self.path_of_net_recon_model, self.dir_of_BFM_fitting, self.device) File "/Users/jet/stable-diffusion-webui/extensions/SadTalker/src/utils/preprocess.py", line 51, in __init__ self.kp_extractor = KeypointExtractor(device) File "/Users/jet/stable-diffusion-webui/extensions/SadTalker/src/face3d/extract_kp_videos_safe.py", line 34, in __init__ self.detector = init_alignment_model('awing_fan',device=device) File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/facexlib/alignment/__init__.py", line 19, in init_alignment_model model.load_state_dict(torch.load(model_path)['state_dict'], strict=True) File "/Users/jet/stable-diffusion-webui/extensions/sd_dreambooth_extension/reallysafe.py", line 117, in load return load_with_extra(filename, *args, **kwargs) File "/Users/jet/stable-diffusion-webui/extensions/sd_dreambooth_extension/reallysafe.py", line 164, in load_with_extra return unsafe_torch_load(filename, *args, **kwargs) File "/Users/jet/stable-diffusion-webui/modules/safe.py", line 106, in load return load_with_extra(filename, extra_handler=global_extra_handler, *args, **kwargs) File "/Users/jet/stable-diffusion-webui/modules/safe.py", line 151, in load_with_extra return unsafe_torch_load(filename, *args, **kwargs) File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/serialization.py", line 713, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/serialization.py", line 930, in _legacy_load result = unpickler.load() File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/pickle.py", line 1213, in load dispatch[key[0]](self) File "/opt/homebrew/Caskroom/miniconda/base/lib/python3.10/pickle.py", line 1254, in load_binpersid self.append(self.persistent_load(pid)) File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/serialization.py", line 876, in persistent_load wrap_storage=restore_location(obj, location), File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/serialization.py", line 175, in default_restore_location result = fn(storage, location) File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/serialization.py", line 152, in _cuda_deserialize device = validate_cuda_device(location) File "/Users/jet/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/serialization.py", line 136, in validate_cuda_device raise RuntimeError('Attempting to deserialize object on a CUDA ' RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

[vinthony]

Hi, thanks for this information, in recent updates, we reduce the requirements of 3rd library so we do not need to add the unsafe flags in sd-webui. It seems the problem of new face alignment package.

[jet3004]

Hi, thanks for this information, in recent updates, we reduce the requirements of 3rd library so we do not need to add the unsafe flags in sd-webui. It seems the problem of new face alignment package.

Ah, thanks...Ok so no --disable-safe-unpickle needed anymore, I thought that is what the new notes were saying.

So a face alignment issue, I see... Yeah, didn't exist in previous versions. Anything I can do or must wait for a fix? Updated to e4fa605 and it persists. Thanks so much, @vinthony.

[vinthony]

just change from src.face3d.extract_kp_videos_safe import KeypointExtractor to from src.face3d.extract_kp_videos import KeypointExtractor, make sure adding the flag of :--disable-safe-unpickle

https://github.com/Winfredy/SadTalker/blob/e4fa6050f8e3b093ef2dd8d98cea8928d567e1bb/src/utils/preprocess.py#L11-L15

[jet3004]

just change from src.face3d.extract_kp_videos_safe import KeypointExtractor to from src.face3d.extract_kp_videos import KeypointExtractor, make sure adding the flag of :--disable-safe-unpickle

https://github.com/Winfredy/SadTalker/blob/e4fa6050f8e3b093ef2dd8d98cea8928d567e1bb/src/utils/preprocess.py#L11-L15

Thanks, that change had it go further and generate without error…BUT…

The generated .mp4 isn't right ( and it took a lot longer than before).

Tested multiple times just now, In trying with my same face .jpg and 2 second audio test clip that I have tested SadTalked with each time (even with no CUDA support on Mac obv and expecting slower results), the generations are taking far longer than the [2023.04.08] build and didn't match the face (you can only see animation in the wrong spot in the top left).

Was the Face Renderer changed? Or is it because of the from src.face3d.extract_kp_videos import KeypointExtractor change? It doesn't work now but it definitely did in the [2023.04.08] build. Thank you @vinthony

[vinthony]

which preprocess method are you using?

[jet3004]

which preprocess method are you using?

It definitely takes much, much longer now even to test a few but I tried Crop, Resize, and Full just now multiple times – all take much longer (literally 2 second audio is taking, on any process, is taking what 15 second audio clips were taking before) but crop/resize worked on same test image but FULL (with and without face enhance) which was working yesterday on it, produced only an animation in top left.

I tried with two more images and FULL and none work now with FULL. I am still testing on the other two but FULL doesn't work as it didn't before.

[vinthony]

it seem your image is very large? can you give some examples for me to test?

[jet3004]

Maybe, yes. Tried the one attached that I know is tinier and it worked okay now in FULL...

But it's def still slower, I will say that: my knowledge is surface level but is this now just completely using 'CPU' so it's mad slow vs PyTorch implementation? I'm still trying to figure out what images are vs aren't working but it's definitely night and day slower now. Nothing changed in that area from previous builds? Again, a 2 second test .mp3 is taking 3x as long now.

[vinthony]

This photo works on my computer at regular speed.

About the changes, in the latest version( update since 2023.04.08), we make the face enhancer a post-process than before. which might be slower than previous ones but has much better visual quality.

If it is because of the face enhancement, you can tune it off.

We do not change anything in face render, it already very slow in CPU as we tested.