Fooocus
Fooocus copied to clipboard
lllyasviel
[Feature Request]: Character consistency
Is there an existing issue for this?
- [X] I have searched the existing issues and checked the recent builds/commits
What would your feature do?
It would generate images of a character with the same face and clothes in any environments.
Take a look at this: https://www.youtube.com/watch?v=7BO_TocisAc
This is an experimental feature in Midjourney v6, but it's already working very well. Note how different this is to the current Face Swap feature. Face Swap keeps the face angle in the generated image the same as the input image, and it doesn't keep the clothes in the generated image. The Character Consistency feature in Midjourney attempts to generate the face at a different angle, and keeps the clothes the same.
Proposed workflow
- Go to Image Prompt
- Press Advanced, Character Consistency (new feature)
- Add a character image
Additional information
No response
@IamTirion i saw that already this morning, nice new feature in MJ. Fooocus can already do almost the same, it wasn't difficult to change the angle of a person in a photo using FaceSwap (no effort was put into the prompt whatsoever, just doing this for the angle). I'm not certain in what you'd propose would differ from the already existing implementation, maybe you can provide more feedback.
Examples (all with different models)
1/1
3/4
3/4
One thing i found using the face swap networks on other UIs, is that starting them later allows them to match faces to new angles and positions.
@IamTirion i saw that already this morning, nice new feature in MJ. Fooocus can already do almost the same, it wasn't difficult to change the angle of a person in a photo using FaceSwap (no effort was put into the prompt whatsoever, just doing this for the angle). I'm not certain in what you'd propose would differ from the already existing implementation, maybe you can provide more feedback. Examples (all with different models)
- Clothes consistency
- Hair consistency
- Face angle. Let me explain the difference I see between Face Swap and Midjourney's Character Consistency. I see in Midjourney's version, the character can look in many directions, and it can generate a side view of the face even if the input image has a frontal view of the face. In my experience, and I think in your examples too, Face Swap will make the character nearly always look in the same direction as the input image, so I can't generate an image that is looking at the character from a different camera angle. There can be slight changes to the face angle when using Face Swap, but not as much as the Midjourney one.
You can see the girl looking down at the camera, looking away from the camera, while she is looking straight at the camera in the input image.
ScionoicS suggested by starting Face Swap later, we can get bigger changes to the face angle, but that sacrifices the likeliness of the generated face to the input face. In my experience, it also doesn't work that well. There was once I first generated an image, then inpainted it with Face Swap. In the end, it still gave me the same face angle as the input image.
Is something possible to do with inswapper_128, I heard that MJ is also using a non-open-source high-resolution inswapper_ version by Insightface, I have used inswapper_128 it's in (roop-unleashed) and the result is amazing but resolution is low
One thing i found using the face swap networks on other UIs, is that starting them later allows them to match faces to new angles and positions.
Maybe Ive overlooked something but surely thats just because they use controlnet too? Instant ID is basically IP adapter insightface and controlnets. An idea maybe to use insightface twice, once with IP adapter then once at the end alone when the image has been generated. Its a quick process so wouldnt add much time onto each generation.
One thing i found using the face swap networks on other UIs, is that starting them later allows them to match faces to new angles and positions.
Maybe Ive overlooked something but surely thats just because they use controlnet too? Instant ID is basically IP adapter insightface and controlnets. An idea maybe to use insightface twice, once with IP adapter then once at the end alone when the image has been generated. Its a quick process so wouldnt add much time onto each generation.
I don't know, but maybe it's not worth replicating their results anymore. There are new researches on character consistency every day. Everything we have feels like a workaround, rather than a real solution.
