ComfyUI Image Artifacts in Negative Masked Area

Please see attached workflow ... not sure if PNG workflows come through in GH issues so I'll also copy workflow as a JSON below.

Summary - when using an all-black mask as input to VAE Encode (for Inpainting), the resultant image is nonetheless altered. I have confirmed this issue occurs across many SDXL checkpoints, although I'd assume the issue is with the application of the mask, not the model.

Workflow is in this PNG and also this shows the issue - the 1024 square in the top left was inpainted with an all-black mask. In particular, a hotspot is visible right near the center of the image but the entire inpainted area is easy to see.

ComfyUI_inpaint_mask_problem_230926v01_00002_

{ "last_node_id": 61, "last_link_id": 96, "nodes": [ { "id": 5, "type": "GetImageSize", "pos": [ 1397, 505 ], "size": { "0": 210, "1": 46 }, "flags": {}, "order": 12, "mode": 0, "inputs": [ { "name": "image", "type": "IMAGE", "link": 58 } ], "outputs": [ { "name": "width", "type": "INT", "links": [ 5, 6 ], "shape": 3, "slot_index": 0 }, { "name": "height", "type": "INT", "links": [ 7, 8 ], "shape": 3, "slot_index": 1 } ], "properties": { "Node name for S&R": "GetImageSize" } }, { "id": 8, "type": "CLIPTextEncodeSDXL", "pos": [ 1826, 468 ], "size": { "0": 400, "1": 270 }, "flags": {}, "order": 14, "mode": 0, "inputs": [ { "name": "clip", "type": "CLIP", "link": 94 }, { "name": "width", "type": "INT", "link": 5, "widget": { "name": "width", "config": [ "INT", { "default": 1024, "min": 0, "max": 8192 } ] } }, { "name": "target_width", "type": "INT", "link": 6, "widget": { "name": "target_width", "config": [ "INT", { "default": 1024, "min": 0, "max": 8192 } ] } }, { "name": "height", "type": "INT", "link": 7, "widget": { "name": "height", "config": [ "INT", { "default": 1024, "min": 0, "max": 8192 } ] } }, { "name": "target_height", "type": "INT", "link": 8, "widget": { "name": "target_height", "config": [ "INT", { "default": 1024, "min": 0, "max": 8192 } ] } }, { "name": "text_g", "type": "STRING", "link": 4, "widget": { "name": "text_g", "config": [ "STRING", { "multiline": true, "default": "CLIP_G" } ] } } ], "outputs": [ { "name": "CONDITIONING", "type": "CONDITIONING", "links": [ 9 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "CLIPTextEncodeSDXL" }, "widgets_values": [ 1024, 1024, 0, 0, 1024, 1024, "CLIP_G", "CLIP_L" ] }, { "id": 7, "type": "Text", "pos": [ 1347, 720 ], "size": { "0": 315, "1": 58 }, "flags": {}, "order": 0, "mode": 0, "outputs": [ { "name": "STRING", "type": "STRING", "links": [ 12 ], "shape": 3, "slot_index": 0 } ], "title": "Negative Prompt", "properties": { "Node name for S&R": "Text" }, "widgets_values": [ "wrong" ] }, { "id": 9, "type": "CLIPTextEncode", "pos": [ 1911, 822 ], "size": { "0": 239.5443572998047, "1": 54 }, "flags": {}, "order": 6, "mode": 0, "inputs": [ { "name": "clip", "type": "CLIP", "link": 95 }, { "name": "text", "type": "STRING", "link": 12, "widget": { "name": "text", "config": [ "STRING", { "multiline": true } ] } } ], "outputs": [ { "name": "CONDITIONING", "type": "CONDITIONING", "links": [ 13 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "CLIPTextEncode" }, "widgets_values": [ "" ] }, { "id": 6, "type": "Text", "pos": [ 1357, 608 ], "size": { "0": 315, "1": 58 }, "flags": {}, "order": 1, "mode": 0, "outputs": [ { "name": "STRING", "type": "STRING", "links": [ 4 ], "shape": 3, "slot_index": 0 } ], "title": "Positive Prompt", "properties": { "Node name for S&R": "Text" }, "widgets_values": [ "a large robot arm in a manufacturing facility" ] }, { "id": 11, "type": "VAEEncodeForInpaint", "pos": [ 1866, 973 ], "size": { "0": 315, "1": 98 }, "flags": {}, "order": 13, "mode": 0, "inputs": [ { "name": "pixels", "type": "IMAGE", "link": 59 }, { "name": "vae", "type": "VAE", "link": 15 }, { "name": "mask", "type": "MASK", "link": 86 } ], "outputs": [ { "name": "LATENT", "type": "LATENT", "links": [ 18 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "VAEEncodeForInpaint" }, "widgets_values": [ 0 ] }, { "id": 13, "type": "VAEDecode", "pos": [ 2742, 791 ], "size": { "0": 210, "1": 46 }, "flags": {}, "order": 16, "mode": 0, "inputs": [ { "name": "samples", "type": "LATENT", "link": 19 }, { "name": "vae", "type": "VAE", "link": 14 } ], "outputs": [ { "name": "IMAGE", "type": "IMAGE", "links": [ 83 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "VAEDecode" } }, { "id": 12, "type": "KSampler", "pos": [ 2353, 461 ], "size": { "0": 315, "1": 262 }, "flags": {}, "order": 15, "mode": 0, "inputs": [ { "name": "model", "type": "MODEL", "link": 96 }, { "name": "positive", "type": "CONDITIONING", "link": 9 }, { "name": "negative", "type": "CONDITIONING", "link": 13 }, { "name": "latent_image", "type": "LATENT", "link": 18 } ], "outputs": [ { "name": "LATENT", "type": "LATENT", "links": [ 19 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "KSampler" }, "widgets_values": [ 47, "increment", 20, 5, "dpmpp_3m_sde_gpu", "karras", 1 ] }, { "id": 37, "type": "PreviewImage", "pos": [ 3234, 534 ], "size": [ 695.3995681370016, 681.5476737632193 ], "flags": {}, "order": 11, "mode": 0, "inputs": [ { "name": "images", "type": "IMAGE", "link": 55 } ], "title": "Start", "properties": { "Node name for S&R": "PreviewImage" } }, { "id": 49, "type": "Image Paste Crop", "pos": [ 2842, 1257 ], "size": { "0": 315, "1": 122 }, "flags": {}, "order": 17, "mode": 0, "inputs": [ { "name": "image", "type": "IMAGE", "link": 92 }, { "name": "crop_image", "type": "IMAGE", "link": 83 }, { "name": "crop_data", "type": "CROP_DATA", "link": 67 } ], "outputs": [ { "name": "IMAGE", "type": "IMAGE", "links": [ 68 ], "shape": 3, "slot_index": 0 }, { "name": "IMAGE", "type": "IMAGE", "links": [ 85 ], "shape": 3, "slot_index": 1 } ], "properties": { "Node name for S&R": "Image Paste Crop" }, "widgets_values": [ 0, 0 ] }, { "id": 51, "type": "PreviewImage", "pos": [ 3008, 1462 ], "size": [ 210, 246 ], "flags": {}, "order": 19, "mode": 0, "inputs": [ { "name": "images", "type": "IMAGE", "link": 85 } ], "properties": { "Node name for S&R": "PreviewImage" } }, { "id": 50, "type": "PreviewImage", "pos": [ 3239, 1263 ], "size": [ 696.7448162863502, 680.6636407515141 ], "flags": {}, "order": 18, "mode": 0, "inputs": [ { "name": "images", "type": "IMAGE", "link": 68 } ], "title": "Result", "properties": { "Node name for S&R": "PreviewImage" } }, { "id": 3, "type": "CheckpointLoaderSimple", "pos": [ 1355, 269 ], "size": { "0": 315, "1": 98 }, "flags": {}, "order": 3, "mode": 0, "outputs": [ { "name": "MODEL", "type": "MODEL", "links": [ 96 ], "shape": 3, "slot_index": 0 }, { "name": "CLIP", "type": "CLIP", "links": [ 94, 95 ], "shape": 3, "slot_index": 1 }, { "name": "VAE", "type": "VAE", "links": [ 14, 15 ], "shape": 3, "slot_index": 2 } ], "properties": { "Node name for S&R": "CheckpointLoaderSimple" }, "widgets_values": [ "sdxl\\sd_xl_base_1.0_0.9vae.safetensors" ] }, { "id": 54, "type": "PreviewImage", "pos": [ 1980, 1210 ], "size": [ 210, 246 ], "flags": {}, "order": 9, "mode": 0, "inputs": [ { "name": "images", "type": "IMAGE", "link": 79 } ], "properties": { "Node name for S&R": "PreviewImage" } }, { "id": 53, "type": "MaskToImage", "pos": [ 1720, 1180 ], "size": { "0": 210, "1": 26 }, "flags": {}, "order": 7, "mode": 0, "inputs": [ { "name": "mask", "type": "MASK", "link": 78 } ], "outputs": [ { "name": "IMAGE", "type": "IMAGE", "links": [ 79 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "MaskToImage" } }, { "id": 52, "type": "SolidMask", "pos": [ 1380, 1120 ], "size": { "0": 315, "1": 106 }, "flags": {}, "order": 4, "mode": 0, "outputs": [ { "name": "MASK", "type": "MASK", "links": [ 78, 86 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "SolidMask" }, "widgets_values": [ 0, 1024, 1024 ] }, { "id": 36, "type": "Image Crop Square Location", "pos": [ 724, 554 ], "size": { "0": 315, "1": 126 }, "flags": {}, "order": 10, "mode": 0, "inputs": [ { "name": "image", "type": "IMAGE", "link": 90 } ], "outputs": [ { "name": "IMAGE", "type": "IMAGE", "links": [ 55, 58, 59 ], "shape": 3, "slot_index": 0 }, { "name": "CROP_DATA", "type": "CROP_DATA", "links": [ 67 ], "shape": 3, "slot_index": 1 } ], "properties": { "Node name for S&R": "Image Crop Square Location" }, "widgets_values": [ 50, 50, 1024 ] }, { "id": 61, "type": "Reroute", "pos": [ 571, 1192 ], "size": [ 82, 26 ], "flags": {}, "order": 8, "mode": 0, "inputs": [ { "name": "", "type": "*", "link": 89 } ], "outputs": [ { "name": "IMAGE", "type": "IMAGE", "links": [ 90, 92 ], "slot_index": 0 } ], "properties": { "showOutputText": true, "horizontal": false } }, { "id": 60, "type": "MaskToImage", "pos": [ 305, 1189 ], "size": { "0": 210, "1": 26 }, "flags": {}, "order": 5, "mode": 0, "inputs": [ { "name": "mask", "type": "MASK", "link": 87 } ], "outputs": [ { "name": "IMAGE", "type": "IMAGE", "links": [ 89 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "MaskToImage" } }, { "id": 59, "type": "SolidMask", "pos": [ -68, 1125 ], "size": { "0": 315, "1": 106 }, "flags": {}, "order": 2, "mode": 0, "outputs": [ { "name": "MASK", "type": "MASK", "links": [ 87 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "SolidMask" }, "widgets_values": [ 0.5, 2048, 2048 ] } ], "links": [ [ 4, 6, 0, 8, 5, "STRING" ], [ 5, 5, 0, 8, 1, "INT" ], [ 6, 5, 0, 8, 2, "INT" ], [ 7, 5, 1, 8, 3, "INT" ], [ 8, 5, 1, 8, 4, "INT" ], [ 9, 8, 0, 12, 1, "CONDITIONING" ], [ 12, 7, 0, 9, 1, "STRING" ], [ 13, 9, 0, 12, 2, "CONDITIONING" ], [ 14, 3, 2, 13, 1, "VAE" ], [ 15, 3, 2, 11, 1, "VAE" ], [ 18, 11, 0, 12, 3, "LATENT" ], [ 19, 12, 0, 13, 0, "LATENT" ], [ 55, 36, 0, 37, 0, "IMAGE" ], [ 58, 36, 0, 5, 0, "IMAGE" ], [ 59, 36, 0, 11, 0, "IMAGE" ], [ 67, 36, 1, 49, 2, "CROP_DATA" ], [ 68, 49, 0, 50, 0, "IMAGE" ], [ 78, 52, 0, 53, 0, "MASK" ], [ 79, 53, 0, 54, 0, "IMAGE" ], [ 83, 13, 0, 49, 1, "IMAGE" ], [ 85, 49, 1, 51, 0, "IMAGE" ], [ 86, 52, 0, 11, 2, "MASK" ], [ 87, 59, 0, 60, 0, "MASK" ], [ 89, 60, 0, 61, 0, "*" ], [ 90, 61, 0, 36, 0, "IMAGE" ], [ 92, 61, 0, 49, 0, "IMAGE" ], [ 94, 3, 1, 8, 0, "CLIP" ], [ 95, 3, 1, 9, 0, "CLIP" ], [ 96, 3, 0, 12, 0, "MODEL" ] ], "groups": [], "config": {}, "extra": {}, "version": 0.4 }

Sep 26 '23 12:09 ancillarymagnet

Following up on this - I've posted a more real-world example of this issue with an accompanying workflow on reddit: https://www.reddit.com/r/comfyui/comments/16tiend/comfyui_inpaint_color_shenanigans_workflow/

Sep 27 '23 14:09 ancillarymagnet

Might be related: https://github.com/comfyanonymous/ComfyUI/issues/1841

I made this thread just now, but after searching around a little bit more, I stumbled on your post here. This artifacting issue seems to get worse and worse the more times you do this process in a row. I typically only work on a single area at a time, then save it, then load that saved image into the slot, mask something new, sample, save, repeat.

Oct 25 '23 22:10 RandomGitUser321

Thanks for the link - I should have posted here and I'll post there as well - I came to the same conclusion you did - that compression through the VAE isn't lossless and isn't masked. With that said, you can ameliorate this issue by, rather than using VAE Encode (for Inpaint), rather using VAE Encode (not the inpaint one) -> Set Latent Noise Mask.

Oct 26 '23 10:10 ancillarymagnet

Bağlantı için teşekkürler - Buraya göndermeliydim ve oraya da göndereceğim - Ben de sizinle aynı sonuca vardım - VAE aracılığıyla sıkıştırmanın kayıpsız olmadığı ve maskelenmediği. Bununla birlikte, VAE Encode (Inpaint için) kullanmak yerine, VAE Encode (inpaint olanı değil) -> Gizli Gürültü Maskesini Ayarla'yı kullanarak bu sorunu iyileştirebilirsiniz.

I tested it and the result is very bad. Could you share your Workflow results?

Dec 31 '23 20:12 kadirnar