major change : multiple face, face checkpoints, gender detection, face blending, face extraction, face comparison

[glucauze]

This is a first version of an important change:

• allow reuse of faces via checkpoints (~ 2kb). Checkpoints can be built in the FaceTools tab. Checkpoints have precedence over reference img (reference source image is discarded but not the batch source images).
• gender detection of faces (face numbers only concern gender, can be disabled)
• face combination (blending) sources, references and checkpoints. Blending is activated by default. Allow the use of multiple version of a face to improve the result. Blending is also performed when building face checkpoint. This is done via the batch source image option when blend is activated.
• notion of face unit as for controlNet. For the moment, 3 possible units (this is hard coded but could be an option)
• several versions of the same face can be used for replacement: several images generated if blend is not activated, mixed faces otherwise. This is done via the batch source image option.
• face similarity and filtering (against reference and/or against source). This is configured via the 2 sliders (min sim and min ref sim). Ref is the checkpoint or the ref image.
• face comparison in tools
• face extraction (using upscaling or not)

Performance improvement : don't use temp files anymore (useless), process the source face only once, checkpoints are also faster since no analysis is required.

This is based on a slightly older version of the faceswap plugin (which is no longer public) that evolved separately. I have tried my best to reintegrate the roop changes. I haven't yet reimplemented the API part. It works for me, but there may still be bugs. Think of it as a proposal of what roop could be.

TODO : save image, reactivate API, cleanup the mess in the interface (the parsing of arguments is tricky).

[SwatMessiah]

for version - major change #118 sha - 9c6baf11b1131c6161e33f5b46e68db762543fe6

i testet your version, very nice changes!

one problem i have is its not autosaving the "restored" image (windows 11). maybe you can add a path in the roop-settings tab or a checkbox "save restored image"

great work ;)

[SwatMessiah]

i also found after the tex2img with Highres fix - the face get not swapped and in the a1111 ui there is a empty preview after complete

[glucauze]

should be fixed in the last commit. Also added an API example (API is not really easy to use right now) in example directory. API should work. Note that hires.fix will happen before swapping in the current implementation.

Saving of images should also be fixed.

[SwatMessiah]

let you know if i found an error, but works very nice - love the idea of the "face model" works great!

maybe a QoL feature:

• use the same image name and just add swapped?
• add gen info from original image

keep up the good work! 👍 :)

---

Version 1 "simple"

• i modified this on my script - so it also save the gen info and continues with the same name pattern but with 1 higher number in the filename

            # Modified with geninfo and continue same name pattern
            save_image(result_images[i], p.outpath_samples, "", seed=int(p.seed), info=orig_infos[i], suffix="-swapped")
  

Bug:

• only works for 1 image not for higher batchcount/size
• generate forever - sometime doesnt save the seed in the swapped imagename

[glucauze]

Hey, thanks for the feedback. Didn't see your last message before committing. I have a solution based on your first suggestion (it don't breaks the batch).

I added some documentation inside the extension I renamed upscaler to enhancements.

There is now an option to perform inpainting of each faces before upscaling. This is useful to modify the faces after the swapping occur.

[SwatMessiah]

Hey, no problem :) thanks for your work. realy like your "roop" version

i removed it because it was a failed code ;) thought: if p.batch_size > 1 or p.batch_count > 1: would work but after restart it throws an error ('StableDiffusionProcessingTxt2Img' object has no attribute 'batch_count')

---

Update - think i found it:

current only for Batch size

      if p.batch_size > 1:

replace with (batchsize and batch count)

      if p.batch_size > 1 or p.n_iter > 1:

---

Bug: i will continue testing it later but for now, enable inpaint (denoise value > 0) throws an error

\AppData\Local\Programs\Python\Python310\lib\ntpath.py", line 104, in join path = os.fspath(path) TypeError: expected str, bytes or os.PathLike object, not NoneType 2023-07-04 17:34:02,472 - Roop - ERROR - Failed to upscale expected str, bytes or os.PathLike object, not NoneType

[dhwz]

@glucauze a few complaints :-)

1. swapped Images are stored even if "Always save all generated images" is disabled
2. Image is stored as "swapped-0000-1389764686-[prompt_spaces]" instead of storing it to the original filename e.g. 02709-1389764686-woman.jpg, why is isn't it copied to the final filename I don't want the duplicated one (without swap). You have already an option "Show Unmodified (original)", maybe also add "Save Unmodified (original)".
3. Face Tools in menu and also in the roop section, why?
4. Settings should be moved to settings and are wrongly placed in that section (in my opinion).

[glucauze]

@SwatMessiah should be fixed. This was due to the fact that when inpainting tried to save the image (useless).

@dhwz :

1. should be fixed
2. a bit of refactoring is needed for that, will wait. It's anoying but it doesn't hinder use
3. Renamed FaceTools to Roop. Removed from accordion. There was no tab at first and I left it at that.
4. I'll look at adding it to the settings panel if I have time. For the moment, it allows you to easily activate/deactivate the saving of images before the swap. And it would make it possible to quickly change models in the unlikely event that a new one becomes available.

[SwatMessiah]

inpaint works now👍

i was also struggeling with the name for saving the image (as you saw in my reply "version 2" xD)

• you could also make bool checkbox in the settings (Swap Gridimage faces) not sure ;)

Bug: when you set Batch count to > 1 it creates a "Grid Image" and swappes the faces:

Bugfix add p.niter > 1:

        if p.batch_size > 1 or p.n_iter > 1:
            # Remove grid image if batch size is greater than 1 :
            result_images = result_images[1:]
            result_infos = result_infos[1:]
            logger.info("Discard grid image from swapping process. This could induce bugs with some extensions.")

[dhwz]

@glucauze Probably a bug, the Enhancements section is also displayed on the "Face Tools (now Roop)" menu?

[glucauze]

It's not a bug, it allows to play with it during extraction. Last commit add face units configuration in sd settings and some inpaintings config (sampler and when it occurs)

[SwatMessiah]

just an idea - in the enhancement section you could also add something like add ISO noise with a strength slider as last step

maybe like:

image_array = np.array(image) # NumPy array

# Set the strength of the noise (between 0 and 1)
strength = 0.5 

# Generate ISO noise
noise = np.random.normal(loc=0, scale=25, size=image_array.shape).astype(np.uint8)
scaled_noise = (strength * noise).astype(np.uint8)

# Add the noise
noisy_image_array = np.clip(image_array + scaled_noise, 0, 255).astype(np.uint8)

[glucauze]

@SwatMessiah Could you elaborate a bit more on the usefulness of this?

If we add noise (which img2img must already have done), won't that mean deviating a lot from the original model?

Apart from that, does the extension work normally?

[SwatMessiah]

@glucauze it was just an idea when you add some fine "iso noise / film grain / Gaussian noise" over the image it looks for the eye more "realistic" because the skin is less smooth

[SwatMessiah]

@glucauze here is my prototype.. to see what i mean ;) with the "hardcoded" settings it makes the image more real maybe you can put them as sliders

its not optimized but you can run it - for a1111 it would need some changes try global_strength = 0.1 up to 0.3 and compare the skin it breaks little bit the smoothnes

put the file in a folder create subfolder "source" and paste a image there python noisetest.py

------------ removed the sample script to not spam hundred lines ;) ------------

[glucauze]

Hey thanks for your work. I did try it a bit, but i think the noise was too heavy and i was not completly convinced. I will give it another try latter and see if it's possible to integrate.

I added an custom inswapper class that use ldsr to upscale the 128*128 swapped face by a 4 factor and a add a bit of sharpness. This is done only on faces. It is disabled by default but can be tested by enabling upscaled swapper in sd settings (roop). It does not requires restart. It's not perfect but it tends to give more realistic results as far as i have seen. It takes a bit longer to process (of course).

[SwatMessiah]

i would also keep the noise low like remap a slider form 0-1 to the noise value (0-0.3) would also call it more like "film grain" but it was just an idea ;) feel free to use/implement it or not its no problem

[Fenrir200678]

Just checked this fork out a couple of hours ago and played around with it. Really nice changes! Only thing I couldn't get to work is using this with regular A1111 inpainting. So I want the faceswap only in my masked area, but it's always doing a full swap no matter what and where I mask. This worked fine in the original version.

I know I can inpaint via Post Inpainting in Enhancement tab, but also there is no way to mask out an area.

Am I missing something or is that just not (yet?) implemented?

[glucauze]

No, you're probably right. To allow an option to view the original images, I had to migrate the moment the processing was done. It has to be said that a1111's documentation on extensions isn't always very clear to me.

As a side effect, this has probably changed the way impainting is done. I'll have to have a look.

[glucauze]

@Fenrir200678 the problem should be fixed. I've changed the way the processing is carried out back to the method I used previously.

I've also solved the file naming problem as far as i have seen.

[Fenrir200678]

Seems to work fine after a few quick tests. Thank you!

[glucauze]

Had to close and restart the code base for cleaner git project. I resubmitted a pr. You will likely have to remove/reinstall to test.