Add face landmark preprocess

[OedoSoldier]

Add support for face landmark CotrolNet: https://huggingface.co/georgefen/Face-Landmark-ControlNet

Diff model extracted: https://huggingface.co/OedoSoldier/Face-Landmark-ControlNet-diff

[catboxanon]

Surely not all of this is needed? Of note, the whole C++ source for MMCV ops is being included, but I'm sure this can be stripped down much further.

[OedoSoldier]

Surely not all of this is needed? Of note, the whole C++ source for MMCV ops is being included, but I'm sure this can be stripped down much further.

I think we should add mmcv, mmsegmentation, and mmpose to dependencies in requirements.txt. Although I tried using an existing mmcv package, it wasn't compatible with the mmpose.

This issue is related to git issues #485 and #70. Now that the naming conflict has been resolved, we can install mmcv and other packages properly without needing to rely on local packages.

[OedoSoldier]

I attempted to install pre-built packages of mmcv-full through pip, but was unsuccessful. So I resorted to using the existing local mmcv packages and was able to resolve the 'scope exists' error.

[deepfates]

Testing this inside webui, doesn't seem to have a Model attribute? The preprocessor shows up in the preprocessor drop-down but nothing in the model drop-down. Annorator preview works.

Also get this error while running:

Error running process: /content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
Traceback (most recent call last):
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/modules/scripts.py", line 386, in process
    script.process(p, *script_args)
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 659, in process
    self.unloadable.get(unit.module, lambda:None)()
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/processor.py", line 248, in unload_face_model
    unload_face_model()
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/face/__init__.py", line 40, in unload_face_model
    face_model.model.cpu()
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1269, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'TopDown' object has no attribute 'model'

[OedoSoldier]

Testing this inside webui, doesn't seem to have a Model attribute? The preprocessor shows up in the preprocessor drop-down but nothing in the model drop-down. Annorator preview works.

Also get this error while running:

Error running process: /content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
Traceback (most recent call last):
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/modules/scripts.py", line 386, in process
    script.process(p, *script_args)
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 659, in process
    self.unloadable.get(unit.module, lambda:None)()
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/processor.py", line 248, in unload_face_model
    unload_face_model()
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/face/__init__.py", line 40, in unload_face_model
    face_model.model.cpu()
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1269, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'TopDown' object has no attribute 'model'

Ah, didn't notice that, fixed. You can download the face landmark model here: https://huggingface.co/georgefen/Face-Landmark-ControlNet Also I've extracted a diff model here: https://huggingface.co/OedoSoldier/Face-Landmark-ControlNet-diff

[zhixiongzuo]

@OedoSoldier, I can use diff_control_sd15_landmarks_fp16.safetensors to generate face landmark in webui, but when I generate images(enable controlnet-face) just same error with "AttributeError: 'TopDown' object has no attribute 'model' ".

[OedoSoldier]

@OedoSoldier, I can use diff_control_sd15_landmarks_fp16.safetensors to generate face landmark in webui, but when I generate images(enable controlnet-face) just same error with "AttributeError: 'TopDown' object has no attribute 'model' ".

Try cherry-pick 7555c558cd98547d9e888aa00f4cb37bb72c2f84, it should be fixed.

[zhixiongzuo]

good job ! it works now.

[deepfates]

Thank you! This does indeed work for txt2img model now. However when using it with the img2img or inpainting interfaces I get the following error:

Loading model from cache: diff_control_sd15_landmarks_fp16 [1b57219f]
Loading preprocessor: face_landmark
Error running process: /content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
Traceback (most recent call last):
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/modules/scripts.py", line 386, in process
    script.process(p, *script_args)
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 728, in process
    control, detected_map = self.detectmap_proc(detected_map, unit.module, unit.rgbbgr_mode, resize_mode, h, w)
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 556, in detectmap_proc
    detected_map = HWC3(detected_map)
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/util.py", line 6, in HWC3
    assert x.dtype == np.uint8
  File "/usr/local/lib/python3.9/dist-packages/PIL/Image.py", line 529, in __getattr__
    raise AttributeError(name)
AttributeError: dtype

Something about the datatypes coming from the annotator are not matching, I think? Have you had success with img2img on face landmarks before?

[CCpt5]

Thank you! This does indeed work for txt2img model now. However when using it with the img2img or inpainting interfaces I get the following error:

> File "/usr/local/lib/python3.9/dist-packages/PIL/Image.py", line 529, in __getattr__ > raise AttributeError(name) > AttributeError: dtype > ``` > > Something about the datatypes coming from the annotator are not matching, I think? Have you had success with img2img on face landmarks before? ___

Annnd GPT-4 says:

[OedoSoldier]

Thank you! This does indeed work for txt2img model now. However when using it with the img2img or inpainting interfaces I get the following error:

Loading model from cache: diff_control_sd15_landmarks_fp16 [1b57219f]
Loading preprocessor: face_landmark
Error running process: /content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py
Traceback (most recent call last):
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/modules/scripts.py", line 386, in process
    script.process(p, *script_args)
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 728, in process
    control, detected_map = self.detectmap_proc(detected_map, unit.module, unit.rgbbgr_mode, resize_mode, h, w)
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 556, in detectmap_proc
    detected_map = HWC3(detected_map)
  File "/content/drive/MyDrive/stable-diffusion-webui-colab/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/util.py", line 6, in HWC3
    assert x.dtype == np.uint8
  File "/usr/local/lib/python3.9/dist-packages/PIL/Image.py", line 529, in __getattr__
    raise AttributeError(name)
AttributeError: dtype

Something about the datatypes coming from the annotator are not matching, I think? Have you had success with img2img on face landmarks before?

Thanks for reporting, it's a bug, now fixed.

[95864668]

感谢大江的无私分享，在colab上运行提示错误信息： `Traceback (most recent call last): File "init.pxd", line 942, in numpy.import_array RuntimeError: module compiled against API version 0x10 but this version of numpy is 0xf . Check the section C-API incompatibility at the Troubleshooting ImportError section at https://numpy.org/devdocs/user/troubleshooting-importerror.html#c-api-incompatibility for indications on how to solve this problem .

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/local/lib/python3.9/dist-packages/gradio/routes.py", line 337, in run_predict output = await app.get_blocks().process_api( File "/usr/local/lib/python3.9/dist-packages/gradio/blocks.py", line 1015, in process_api result = await self.call_function( File "/usr/local/lib/python3.9/dist-packages/gradio/blocks.py", line 833, in call_function prediction = await anyio.to_thread.run_sync( File "/usr/local/lib/python3.9/dist-packages/anyio/to_thread.py", line 31, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "/usr/local/lib/python3.9/dist-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/usr/local/lib/python3.9/dist-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, *args) File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/controlnet.py", line 362, in run_annotator result, is_image = preprocessor(img, res=pres, thr_a=pthr_a, thr_b=pthr_b) File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/scripts/processor.py", line 238, in face from annotator.face import apply_face_model File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/face/init.py", line 8, in from annotator.mmpkg.mmpose.apis import inference_top_down_pose_model, init_pose_model File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/mmpkg/mmpose/apis/init.py", line 2, in from .inference import (collect_multi_frames, inference_bottom_up_pose_model, File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/mmpkg/mmpose/apis/inference.py", line 17, in from annotator.mmpkg.mmpose.datasets.dataset_info import DatasetInfo File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/mmpkg/mmpose/datasets/init.py", line 7, in from .datasets import ( # isort:skip File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/mmpkg/mmpose/datasets/datasets/init.py", line 2, in from ...deprecated import (TopDownFreiHandDataset, TopDownOneHand10KDataset, File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/mmpkg/mmpose/deprecated.py", line 5, in from .datasets.datasets.base import Kpt2dSviewRgbImgTopDownDataset File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/mmpkg/mmpose/datasets/datasets/base/init.py", line 2, in from .kpt_2d_sview_rgb_img_bottom_up_dataset import
File "/content/stable-diffusion-webui/extensions/sd-webui-controlnet/annotator/mmpkg/mmpose/datasets/datasets/base/kpt_2d_sview_rgb_img_bottom_up_dataset.py", line 8, in from xtcocotools.coco import COCO File "/usr/local/lib/python3.9/dist-packages/xtcocotools/coco.py", line 58, in from . import mask as maskUtils File "/usr/local/lib/python3.9/dist-packages/xtcocotools/mask.py", line 3, in import xtcocotools._mask as _mask File "xtcocotools/_mask.pyx", line 23, in init xtcocotools._mask File "init.pxd", line 944, in numpy.import_array ImportError: numpy.core.multiarray failed to import`

[95864668]

numpy版本问题，已解决 !pip install numpy==1.24.2

[figurehunt]

HI dalao i got a problem like this,

Loading model from cache: diff_control_sd15_landmarks_fp16 [1b57219f] Loading preprocessor: face_landmark Error running process: C:\AI\extensions\sd-webui-controlnet\scripts\controlnet.py Traceback (most recent call last): File "C:\AI\modules\scripts.py", line 417, in process script.process(p, *script_args) File "C:\AI\extensions\sd-webui-controlnet\scripts\controlnet.py", line 727, in process detected_map, is_image = preprocessor(input_image, res=unit.processor_res, thr_a=unit.threshold_a, thr_b=unit.threshold_b) File "C:\AI\extensions\sd-webui-controlnet\scripts\processor.py", line 238, in face from annotator.face import apply_face_model File "C:\AI\extensions\sd-webui-controlnet\annotator\face_init_.py", line 8, in from annotator.mmpkg.mmpose.apis import inference_top_down_pose_model, init_pose_model File "C:\AI\extensions\sd-webui-controlnet\annotator\mmpkg\mmpose\apis_init_.py", line 2, in from .inference import (collect_multi_frames, inference_bottom_up_pose_model, File "C:\AI\extensions\sd-webui-controlnet\annotator\mmpkg\mmpose\apis\inference.py", line 19, in from annotator.mmpkg.mmpose.models import build_posenet File "C:\AI\extensions\sd-webui-controlnet\annotator\mmpkg\mmpose\models_init_.py", line 6, in from .detectors import * # noqa File "C:\AI\extensions\sd-webui-controlnet\annotator\mmpkg\mmpose\models\detectors_init_.py", line 2, in from .associative_embedding import AssociativeEmbedding File "C:\AI\extensions\sd-webui-controlnet\annotator\mmpkg\mmpose\models\detectors\associative_embedding.py", line 13, in from annotator.mmpkg.mmpose.core.post_processing.group import HeatmapParser File "C:\AI\extensions\sd-webui-controlnet\annotator\mmpkg\mmpose\core\post_processing\group.py", line 8, in from munkres import Munkres ModuleNotFoundError: No module named 'munkres' 提示：Python 运行时抛出了一个异常。请检查疑难解答页面。

could you pls help me? thank you!