diffusers icon indicating copy to clipboard operation
diffusers copied to clipboard

add pipeline of DifFace

Open zsyOAOA opened this issue 1 year ago • 5 comments

Please integrate the pipeline of DifFace (https://arxiv.org/abs/2212.06512) to diffusers. DifFace aims to solve the problem of face image enhancement using a pre-trained diffusion model. Thanks.

zsyOAOA avatar Feb 02 '23 04:02 zsyOAOA

Hey @zsyOAOA,

Thanks for opening the PR. Seems like quite some tests are still failing. Also in case you have question regarding the design this doc might help: https://huggingface.co/docs/diffusers/main/en/conceptual/philosophy

patrickvonplaten avatar Feb 03 '23 17:02 patrickvonplaten

@patrickvonplaten Hi, My code depends on the library of opencv, which causes some failed tests. Is it possible to include opencv in diffusers since it is a very common tool in CV?

zsyOAOA avatar Feb 09 '23 10:02 zsyOAOA

cc @williamberman

patrickvonplaten avatar Feb 13 '23 10:02 patrickvonplaten

Hey @zsyOAOA, super cool model! It might be helpful to open an issue first where we discuss the model components and what code changes we need to add them. Similar to this discussion in the control net issue https://github.com/huggingface/diffusers/issues/2331#issuecomment-1431990832.

A few preliminaries that I see,

I think there's some custom code for loading weights from google drive, we build diffusers pipelines with the idea that models can be serialized to git lfs repos with particular file locations within the repos. This helps support uniform apis for loading weights. This also eases integration with existing libraries that we have for saving an loading to the huggingface hub (huggingface.co/models). If there needs to be some way of saving/loading a particular model from google drive, that saving and loading code probably shouldn't be a part of diffusers.

Some of the modules in the pipeline seem to be initialized outside of the constructor when the pipeline is ran. We always pass models into the pipeline constructor pre-initialized because this supports our saving/loading apis.

I see some pretty heavy weight dependencies under the difface module. We do include custom models for particular pipelines but usually we should first see if they can be integrated with existing model components. I think the additional models would be a good candidate for discussion in an issue for adding the pipeline. Our main rationale here is that we need to understand the future maintenance from newly included models.

williamberman avatar Feb 15 '23 21:02 williamberman

@williamberman Our project depends on another deep image super-resolution model Real-ESRGAN and face detection model RetinaFace. They are not necessary for diffusers but specified for our project. I don't know how to add them in diffusers.

zsyOAOA avatar Feb 17 '23 02:02 zsyOAOA

awesome, thanks for the context @zsyOAOA ! for auxiliary models, we have a few options

  1. the auxiliary model can be used to pre-process input such that the processed input fits with existing APIs

We keep the model in an external dependency and document how to use it to pre-process input. This is the preferred method as there's no additional maintenance burden on us and APIs stay consistent

A good example of this is some of the auxiliary controlnet models i.e. image segmentation. The image segmentation model is maintained outside of diffusers. The pipeline user manually creates the segmentation image and passes it in to the pipeline. This means the pipeline api stays consistent and we don't have to maintain the auxiliary models in source. Usually these auxiliary models are already in transformers or provided as a one-off package.

  1. the auxiliary model must by used inside the pipeline

We implement the model such that it fits with the existing diffusers design (i.e. model loading, configuration). This is doable if there is sufficient demand for the pipeline but requires additional maintenance burden.

A good example of this is the prior network for our unclip pipeline.

williamberman avatar Mar 07 '23 01:03 williamberman

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

github-actions[bot] avatar Mar 31 '23 15:03 github-actions[bot]