s0md3v
add a check box which allows people use codeformer to upscale img and.video
@cuong29102003 it's possible that you have conflicting dependencies or versions. try pip install --upgrade codeformer-pip
and don't rememmber reinstall requirements.txt
Hey, that is a nice PR... please resolved the conflicts towards nest branch. Also extend the pre_check() in case upscale is active but models are missing.
I've also been working on a similar implementation. However my approach is a bit different.
The problem I see with the implementation here is that the entire image will be processed through the upscaling. In my tests it causes artifacts immediately around the face and head of the person being swapped due to the low res 128 swapping model. InSwapper alters more of the image than CodeFormer does. So CodeFormer leaves additional artifacts untouched (see samples below).
Instead of upscaling the entire image after the fact, I'm upscaling each face as they are swapped. The main issue I'm dealing with is memory contention between InsightFace (OnnxRuntime) and CodeFormer (Torch). I have to implement a way to limit the GPU mem usage for each framework so that they can share the total mem available.
Additionally, we need to get the exact face bounds (very easy, just needs implementation. I'll do it when I have some time) and then generate a mask using the exact bounds. Apply a small alpha blur to the edge of the mask. Then use the mask to apply the upscaled face to the original frame image. This method will copy a clean upscaled and swapped face on to the frame. I just haven't gotten to that point yet.
Another technique would be to save each crop of a swapped face to a temporary file along with a txt file with the crop details, while leaving the original frame image intact. Then when each swap is complete, get Onnxruntime to free GPU memory (not sure how to do this, people have been asking how to do this on the Onnxruntime forums. It seems ORT holds on to memory until the parent process exits). Then load CodeFormer, run the logic I outlined above. Merge all of it back to the original frame and output a video.
Source Face
Target Image
Face after swap
Face after upscaling
Final result. Pay particular attention to Jeff Daniels' right cheek. It has heavily aliasing.
@CeeBeeEh @Moeblack Could you both align on Discord and cooo in that issue? I have no direct comparison but suggest we work together on one solution.
@CeeBeeEh thank for your response.
Actually, I'm not satisfied with my implementation either. My approach is to upscale the face after all processing is complete, and everything is handled automatically by the Codeformer, as you know. The reason I did let Codeformer handle the post-processing automatically is that I don't have a clear idea of how to paste the processed faces back into the original image.
In fact, InSwapper can export all the swapped and aligned faces without pasting them back into the original image (there are relevant code snippets in the InsightFace Demo Code), but it only exports the images without their corresponding coordinates. You can directly process these extracted faces and paste them back into the original image.
My coding skills are currently not sufficient to implement this functionality, but I think it can be achieved using the InsightFace SCRFD module. If you have any better ideas, I welcome your suggestions and would be happy to discuss further.