[enhancement]: SD upscale face correction

[whosawhatsis]

Is there an existing issue for this?

• [X] I have searched the existing issues

Contact Details

No response

What should this feature add?

Codeformer and GFPGAN work pretty well for correcting faces, but I'm finding that I can often get even better ones by using embiggen (only when faces don't fall on the border between tiles, though), even with face correction disabled. When a face is close to 512x512, SD can do a much better job. This is particularly true in the 2.0 models, which (with a little negative prompting) have been giving me great faces without any restoration.

A nice third option to have for face restoration would be to pick faces out of the the image (using opencv or whatever the existing face correction options use), upscale that square to 512x512 (or even 768x768, once 2.0 models are supported) and run a few steps of img2img on them (5-7 steps seems to be enough), maybe run GFPGAN for safety, then scale the image back to fit where it was, with the edges feathered like embiggen does so that you don't get those boxes around them like you sometimes get with GFPGAN.

Alternatives

No response

Aditional Content

No response

[netsvetaev]

I think it’s very nice option for the embiggen UI.

[whosawhatsis]

Face-aware embiggen is a bit of a different idea. Having that identify faces and do a similar thing to ensure that the face doesn't end up on a border between two tiles would help a lot (and would use most of the same code needed for this), but what I'm suggesting here is to add it as an alternative option with the other face restore settings. Scale up the face to be large enough for SD to do a good job with it, img2img, then scale the tile back down to fit the original image, all happening automatically on each generated image (as long as a face is detected in it).

[netsvetaev]

Yeah, I see. I’m sure we’ll implement this in the UI with Embiggen. It also very useful for the selective img2img on canvas I propose earlier, to get rid of edges with embiggen-like feathering. Also, we already have similar option for canvas part regeneration, when selection is smaller than 512px. They're all connected features. BTW, I still get no restoration effect when have more than one face in the image and want some kind of manual face detection :-(

[whosawhatsis]

BTW, I still get no restoration effect when have more than one face in the image and want some kind of manual face detection :-(

Come to think of it, that's something missing from the unified canvas... There should be a button on that interface to do face correction on the part of the image in the bounding box. That would let you put the box over a face you want to restore and run restoration on it, excluding any other faces that might be on the canvas.

[whosawhatsis]

Btw, I a similar system might work for hand correction, though it might require a finetuned model trained specifically on images zoomed-in on hands to be effective.

[netsvetaev]

Btw, I a similar system might work for hand correction, though it might require a finetuned model trained specifically on images zoomed-in on hands to be effective.

Exactly: select anywhere, fix. But 3-4 months later we may get SD 3.0 which will fix these problems. Or not.

[whosawhatsis]

Exactly: select anywhere, fix. But 3-4 months later we may get SD 3.0 which will fix these problems. Or not.

If we do, we'll want this to use the new model to fix the hands in all of our best 1.5 images.

[netsvetaev]

to fix the hands in all of our best 1.5 images

don’t know how…it must be kind of «face fix», but «hand fix»?

[github-actions[bot]]

There has been no activity in this issue for 14 days. If this issue is still being experienced, please reply with an updated confirmation that the issue is still being experienced with the latest release.