diffusers
diffusers copied to clipboard
add pipeline of DifFace
Please integrate the pipeline of DifFace (https://arxiv.org/abs/2212.06512) to diffusers. DifFace aims to solve the problem of face image enhancement using a pre-trained diffusion model. Thanks.
Hey @zsyOAOA,
Thanks for opening the PR. Seems like quite some tests are still failing. Also in case you have question regarding the design this doc might help: https://huggingface.co/docs/diffusers/main/en/conceptual/philosophy
@patrickvonplaten Hi, My code depends on the library of opencv, which causes some failed tests. Is it possible to include opencv in diffusers since it is a very common tool in CV?
cc @williamberman
Hey @zsyOAOA, super cool model! It might be helpful to open an issue first where we discuss the model components and what code changes we need to add them. Similar to this discussion in the control net issue https://github.com/huggingface/diffusers/issues/2331#issuecomment-1431990832.
A few preliminaries that I see,
I think there's some custom code for loading weights from google drive, we build diffusers pipelines with the idea that models can be serialized to git lfs repos with particular file locations within the repos. This helps support uniform apis for loading weights. This also eases integration with existing libraries that we have for saving an loading to the huggingface hub (huggingface.co/models). If there needs to be some way of saving/loading a particular model from google drive, that saving and loading code probably shouldn't be a part of diffusers.
Some of the modules in the pipeline seem to be initialized outside of the constructor when the pipeline is ran. We always pass models into the pipeline constructor pre-initialized because this supports our saving/loading apis.
I see some pretty heavy weight dependencies under the difface module. We do include custom models for particular pipelines but usually we should first see if they can be integrated with existing model components. I think the additional models would be a good candidate for discussion in an issue for adding the pipeline. Our main rationale here is that we need to understand the future maintenance from newly included models.
@williamberman Our project depends on another deep image super-resolution model Real-ESRGAN and face detection model RetinaFace. They are not necessary for diffusers but specified for our project. I don't know how to add them in diffusers.
awesome, thanks for the context @zsyOAOA ! for auxiliary models, we have a few options
- the auxiliary model can be used to pre-process input such that the processed input fits with existing APIs
We keep the model in an external dependency and document how to use it to pre-process input. This is the preferred method as there's no additional maintenance burden on us and APIs stay consistent
A good example of this is some of the auxiliary controlnet models i.e. image segmentation. The image segmentation model is maintained outside of diffusers. The pipeline user manually creates the segmentation image and passes it in to the pipeline. This means the pipeline api stays consistent and we don't have to maintain the auxiliary models in source. Usually these auxiliary models are already in transformers or provided as a one-off package.
- the auxiliary model must by used inside the pipeline
We implement the model such that it fits with the existing diffusers design (i.e. model loading, configuration). This is doable if there is sufficient demand for the pipeline but requires additional maintenance burden.
A good example of this is the prior network for our unclip pipeline.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.