feat: add inpaint mask generation functionality

[mashb1t]

Implements https://github.com/lllyasviel/Fooocus/discussions/1987 Uses rembg, see https://github.com/danielgatis/rembg

EDIT: Shopify just announced this exact same feature in https://www.shopify.com/magic, see also https://www.theverge.com/24056980/shopify-generative-ai-image-editing-search

This feature allows users to upload an inpaint image and automatically generate the mask for it based on rembg models, so they don't need to switch tools for this.

Process:

1. enable mask in advanced inpainting settings
2. upload inpaint image
3. select mask generation model (rembg)
4. click generate mask from image button

Possible extension (in general, not necessarily regarding this feature): add mask handling for image mask gradio canvas, so users can add an additive mask to the mask image OR disable mask canvas for this image.

[diaolulu1]

Where should I place these mask generation models?

[mashb1t]

@diaolulu1 the models should automatically be downloaded when selecting the model and using it the first time (see screenshot below, default mask generation model is isnet-general-use.

[rayronvictor]

I'm getting an error when using sam as mask generation model

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1103, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 707, in wrapper
    response = f(*args, **kwargs)
  File "/content/Fooocus/webui.py", line 205, in generate_mask
    return generate_mask_from_image(image, mask_model)
  File "/content/Fooocus/extras/inpaint_mask.py", line 11, in generate_mask_from_image
    return remove(
  File "/usr/local/lib/python3.10/dist-packages/rembg/bg.py", line 257, in remove
    masks = session.predict(img, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/rembg/sessions/sam.py", line 169, in predict
    validate(instance=prompt, schema=schema)
  File "/usr/local/lib/python3.10/dist-packages/jsonschema/validators.py", line 1306, in validate
    raise error
jsonschema.exceptions.ValidationError: '{}' is not of type 'array'

Failed validating 'type' in schema:
    {'items': {'properties': {'data': {'items': {'type': 'number'},
                                       'type': 'array'},
                              'label': {'type': 'integer'},
                              'type': {'type': 'string'}},
               'type': 'object'},
     'type': 'array'}

On instance:
    '{}'

[diaolulu1]

I'm getting an error when using sam as mask generation model

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 488, in run_predict
    output = await app.get_blocks().process_api(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1431, in process_api
    result = await self.call_function(
  File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1103, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/usr/local/lib/python3.10/dist-packages/gradio/utils.py", line 707, in wrapper
    response = f(*args, **kwargs)
  File "/content/Fooocus/webui.py", line 205, in generate_mask
    return generate_mask_from_image(image, mask_model)
  File "/content/Fooocus/extras/inpaint_mask.py", line 11, in generate_mask_from_image
    return remove(
  File "/usr/local/lib/python3.10/dist-packages/rembg/bg.py", line 257, in remove
    masks = session.predict(img, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/rembg/sessions/sam.py", line 169, in predict
    validate(instance=prompt, schema=schema)
  File "/usr/local/lib/python3.10/dist-packages/jsonschema/validators.py", line 1306, in validate
    raise error
jsonschema.exceptions.ValidationError: '{}' is not of type 'array'

Failed validating 'type' in schema:
    {'items': {'properties': {'data': {'items': {'type': 'number'},
                                       'type': 'array'},
                              'label': {'type': 'integer'},
                              'type': {'type': 'string'}},
               'type': 'object'},
     'type': 'array'}

On instance:
    '{}'

Me too😭

[mashb1t]

Yes, segment map models will return segments (duh) and not one map, so sadly i'll have to remove them later today.

[rayronvictor]

@mashb1t

https://github.com/lllyasviel/Fooocus/assets/1683922/5a030d4e-280e-46cb-a8b1-50d264a70d2d

[rayronvictor]

PR https://github.com/mashb1t/Fooocus/pull/5

[rayronvictor]

@mashb1t, how about SAM with prompt?

https://github.com/lllyasviel/Fooocus/assets/1683922/696b97e8-bc05-4d52-86f6-0f1693a7dd25

[mashb1t]

@rayronvictor sure, looks really good! I'd propose to put the advanced settings in developer debug mode inpainting, but don't see why we shouldn't use this! Feel free to provide your code in an additional PR to the fork branch (like last time), so we can merge it here.

[rayronvictor]

PR created https://github.com/mashb1t/Fooocus/pull/6

[JulienZLQ]

When I start, it prompts CMD Failed requirements: install -r "requirements_versions.txt"

and when I use the inpaint mask generation, just error

Traceback (most recent call last): File "D:\Fooocus\python_embeded\lib\site-packages\gradio\routes.py", line 488, in run_predict output = await app.get_blocks().process_api( File "D:\Fooocus\python_embeded\lib\site-packages\gradio\blocks.py", line 1431, in process_api result = await self.call_function( File "D:\Fooocus\python_embeded\lib\site-packages\gradio\blocks.py", line 1103, in call_function prediction = await anyio.to_thread.run_sync( File "D:\Fooocus\python_embeded\lib\site-packages\anyio\to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "D:\Fooocus\python_embeded\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread return await future File "D:\Fooocus\python_embeded\lib\site-packages\anyio_backends_asyncio.py", line 807, in run result = context.run(func, *args) File "D:\Fooocus\python_embeded\lib\site-packages\gradio\utils.py", line 707, in wrapper response = f(*args, **kwargs) File "D:\Fooocus\FooocusNov\webui.py", line 245, in generate_mask from extras.inpaint_mask import generate_mask_from_image File "D:\Fooocus\FooocusNov\extras\inpaint_mask.py", line 4, in from rembg import remove, new_session ModuleNotFoundError: No module named 'rembg'

[mashb1t]

As you can see rembg is in the requirements_versions.txt, see https://github.com/lllyasviel/Fooocus/pull/2032/files#diff-2284b86f286dc7e0ea4bd09a0ec20c78fbb17d6724d7f0053e78428d0715bbb1R19

Please make sure to manually install the dependencies if you're not using the automated way. Is there a specific error in the install?

[JulienZLQ]

As you can see rembg is in the requirements_versions.txt, see https://github.com/lllyasviel/Fooocus/pull/2032/files#diff-2284b86f286dc7e0ea4bd09a0ec20c78fbb17d6724d7f0053e78428d0715bbb1R19

Please make sure to manually install the dependencies if you're not using the automated way. Is there a specific error in the install?

I just followed the instructions in this link to manually modify and add them. rembg is sure in the requirements_versions.txt

[JulienZLQ]

[System ARGV] ['FooocusNov\entry_with_update.py'] Python 3.10.9 (tags/v3.10.9:1dd9be6, Dec 6 2022, 20:01:21) [MSC v.1934 64 bit (AMD64)] Fooocus version: 2.1.862 Installing requirements Couldn't install requirements. Command: "D:\Fooocus\python_embeded\python.exe" -m pip install -r "requirements_versions.txt" --prefer-binary Error code: 1 stdout: Requirement already satisfied: torchsde==0.2.5 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 1)) (0.2.5) Requirement already satisfied: einops==0.4.1 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 2)) (0.4.1) Requirement already satisfied: transformers==4.30.2 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 3)) (4.30.2) Requirement already satisfied: safetensors==0.3.1 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 4)) (0.3.1) Requirement already satisfied: accelerate==0.21.0 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 5)) (0.21.0) Requirement already satisfied: pyyaml==6.0 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 6)) (6.0) Requirement already satisfied: Pillow==9.2.0 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 7)) (9.2.0) Requirement already satisfied: scipy==1.9.3 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 8)) (1.9.3) Requirement already satisfied: tqdm==4.64.1 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 9)) (4.64.1) Requirement already satisfied: psutil==5.9.5 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 10)) (5.9.5) Requirement already satisfied: pytorch_lightning==1.9.4 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 11)) (1.9.4) Requirement already satisfied: omegaconf==2.2.3 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 12)) (2.2.3) Requirement already satisfied: gradio==3.41.2 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 13)) (3.41.2) Requirement already satisfied: pygit2==1.12.2 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 14)) (1.12.2) Requirement already satisfied: opencv-contrib-python==4.8.0.74 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 15)) (4.8.0.74) Requirement already satisfied: httpx==0.24.1 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 16)) (0.24.1) Requirement already satisfied: onnxruntime==1.16.3 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 17)) (1.16.3) Requirement already satisfied: timm==0.9.2 in d:\fooocus\python_embeded\lib\site-packages (from -r requirements_versions.txt (line 18)) (0.9.2) Collecting rembg==2.0.53 (from -r requirements_versions.txt (line 19)) Obtaining dependency information for rembg==2.0.53 from https://files.pythonhosted.org/packages/55/6e/5a336d1308105fbe2a9738e7b99e79549628e80595d62142c4334e319b67/rembg-2.0.53-py3-none-any.whl.metadata Using cached rembg-2.0.53-py3-none-any.whl.metadata (14 kB) Collecting groundingdino-py==0.4.0 (from -r requirements_versions.txt (line 20)) Using cached groundingdino-py-0.4.0.tar.gz (82 kB) Preparing metadata (setup.py): started Preparing metadata (setup.py): finished with status 'error'

stderr: error: subprocess-exited-with-error

python setup.py egg_info did not run successfully. exit code: 1

[thiner]

Why does the seg models are saved to a different folder?

Downloading data from 'https://github.com/danielgatis/rembg/releases/download/v0.0.0/u2netp.onnx' to file '/root/.u2net/u2netp.onnx'.

This is really inconvenient for Fooocus running in Docker or remote server. Is it possible move the models to path_inpaint?

[mashb1t]

@thiner it seems like we can set and override the env var U2NET_HOME in https://github.com/danielgatis/rembg/blob/49d1686f65b71b72df1c3b420551406d13074426/rembg/sessions/base.py#L78, will test this in the evening (in ~6-8h). @rayronvictor Feel free to already adjust and provide your feedback!

[mashb1t]

I was already able to test it, works perfectly fine. @thiner thanks for the hint! Please check again yourself.

[rayronvictor]

@mashb1t I fixed an error that occurs when groundingDINO doesn't find boxes PR https://github.com/mashb1t/Fooocus/pull/7. In this case we are using the entire image as a box.

[thiner]

@mashb1t the model path issue is fixed, thanks.

[siriume]

Would it be better if we could integrate rembg into the GPU memory management(from ComfyUI) process?

[mashb1t]

Would it be better if we could integrate rembg into the GPU memory management(from ComfyUI) process?

In general it does already as text_encoder_device respects all settings such as --always-gpu or --always-high-vram, see https://github.com/lllyasviel/Fooocus/blob/main/ldm_patched/modules/model_management.py#L523. I've used the BLIP implementation as a reference. Does this answer your question?

[mashb1t]

Shopify just announced this exact same feature in https://www.shopify.com/magic

[thiner]

Shopify just announced this exact same feature in https://www.shopify.com/magic

I don't know why this pr has been pending for such long... I hope it can be merged into main branch soon.

[metercai]

I merged this PR in my fork: https://github.com/metercai/SimpleSDXL. A pretty good feature. But there are still some unexpected situations in the merger for Chinese user. It may be due to the two components it references not being refined enough, which makes it impossible to merge it.

[mashb1t]

@metercai this PR does not (yet) include translations for the en.json file, didn't have time to do so as i'm working on https://github.com/lllyasviel/Fooocus/pull/1940 since. Maybe somebody can support here, if not i can do it after finishing metadata handling. Feel free to contribute the translations 👍

[metercai]

@metercai this PR does not (yet) include translations for the en.json file, didn't have time to do so as i'm working on #1940 since. Maybe somebody can support here, if not i can do it after finishing metadata handling. Feel free to contribute the translations 👍

Sorry, I don't understand which specific tasks need to be supported on this PR? What's translations for the en.json file?

[mashb1t]

@metercai currently the labels are not in the translation blueprint of en.json. If you've already extracted them for Chinese translations it would be great if you could also add them to en.json.

[thiner]

I got below error while trying generate mask with sam

 requests.exceptions.ConnectTimeout: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /bert-base-uncased/resolve/main/config.json 

I guess there is some dependencies missing. Maybe it's better to list out all dependencies, so we can prevent from runtime error. @rayronvictor

[metercai]

@metercai currently the labels are not in the translation blueprint of en.json. If you've already extracted them for Chinese translations it would be great if you could also add them to en.json.

This is simple, how can I submit it to you? There is too much content in my fork...

[mashb1t]

@metercai you can fork the fork and make a PR in https://github.com/mashb1t/Fooocus