Push to Hub Design

[patrickvonplaten]

It would be nice to discuss a bit the push to hub design of the library.

IMO we have two different use cases for push_to_hub.

1. The complement of from_pretrained(...). Every model, scheduler, and diffusion pipeline should eventually have a push_to_hub(...) method that allows one to push the class to the Hub. IMO this method can mirror 1-to-1 the API we have for Transformers: https://huggingface.co/docs/transformers/v4.21.2/en/main_classes/model#transformers.PreTrainedModel.push_to_hub maybe with the repo_url argument. This function has a very clear purpose and often doesn't need customization. It's as simple as "push the object that you're currently holding to the Hub exactly like as is". We don't have this functionality yet, but it would be cool to integrate it soon. There IMO we can abstract away most of the logic from the user

2. Push to Hub in the example scripts. Our example scripts should foremost be "easy-to-understand" and "easy-to-tweak" and the ability to push to the Hub should be included in this philosophy. There are a lot of different use cases how someone might want to push to the Hub during training (e.g. only some parts of the pipeline, only when a condition is met, etc....) => here let's give the user maximum flexibility by just showing the fundamental blocks that are needed from huggingface_hub - just like it's done in: https://github.com/huggingface/transformers/blob/f719c0377f7f97c4bf9b6b54de209f4aad0aef4b/examples/pytorch/text-classification/run_glue_no_trainer.py#L242 . I think this is even more important for diffusion models because they have many more components

What do you think @patil-suraj @pcuenca @anton-l ?

[anton-l]

2. Something to consider here is also where we should put the model card generator (with training hypermarameters, per-epoch metric logs, etc.). But otherwise I'm very eager to remove the git-related complexity from the examples, especially with the upcoming git-less push_to_hub in huggingface_hub==1.0

[pcuenca]

I agree with both points: I think it's very useful that all objects have a simple push_to_hub functionality, and that during training users should be able to configure how they want to proceed. Very intrigued about the git-less approach pointed out by @anton-l, it would be cool to adopt it soon.

[Wauplin]

Hey there :wave: @osanseviero pointed this issue to me (as well as https://github.com/huggingface/diffusers/issues/81). In general, I'd be glad to help in integrating more huggingface_hub in diffusers when needed. This issue seems a very good start.

To come back to @patrickvonplaten message:

1. That's definitely what push_to_hub is meant to be. Since v0.9, hfh has a new "http-based push-to-hub mixin" called ModelHubMixin. This is not well documented yet but the idea is to inherit your model/scheduler/pipeline classes from ModelHubMixin and implement the private _from_pretrained() and _save_pretrained() methods. By doing this, you automatically have a class with public methods from_pretrained, save_pretrained and push_to_hub. This leverages the http API which means no need to clone a repo locally. => I would be very up to help on this integration. This would be a good exercise for me on a "real" integration and a good occasion to update the hfh documentation.

2. For the record, http-based methods cannot be run in the background at the moment. There is no blocking=False argument implemented (doesn't mean it will never be :) ). So we can either push to use the Repository object or work on making the http-based helpers usable in a training environment. I'm not sure to be the right person in this context (documenting push during training possibilities) but let me know if you have hfh-related questions here :)

[pcuenca]

Hi @Wauplin!

That would be awesome! I think @anton-l might be the most knowledgeable person about this area in diffusers, but he's off this week.

We currently have some basic push_to_hub and init_git_repo functions that are used from the example training script. There's also a model card template that is used when training.

I believe that improving hub features will be very useful when more training and fine-tuning scripts are added.

I'm not sure about the design here, but let's open the discussion.

As @patrickvonplaten said, every object in the library should have the capability to be pushed to the hub. There might be some additional complexity (I don't know, just guessing here) due to the fact that pipelines are a composition of different modules, typically a scheduler and 2 models: UNet as (de)noiser and a decoder. There might be more, like the safety checker in Stable Diffusion. If the user pushes a fine-tuned pipeline to the hub, all components should end up nicely structured in folders like this: https://huggingface.co/CompVis/stable-diffusion-v1-4/tree/main. If they manually push one of the components instead of the full pipeline, I suppose that the model will end up in its own repo in the Hub.

We are also planning to allow community pipelines that don't need to be bundled with their own fine-tuned models. Instead, they are meant for new tasks and work by reusing other modules. For example, a user could create a community pipeline for out-painting by cleverly configuring the modules of the Stable Diffusion pipeline and manipulating the inputs. I'm not sure how this is going to work, but in this case it would be nice to push just the pipeline configuration and code, and refer to other Hub repos for the rest of the modules.

These are just a couple of initial thoughts. Were you planning to work on this yourself @Wauplin, or help us do it? Either way your opinions are very valuable!

[Wauplin]

@pcuenca ok so there is definitely some good stuff existing for pushing to the Hub and improvements that would be welcomed :)

Thanks for the explanation about the different parts of the pipeline ! I guess one way of implementing some community pipelines would be to adapt the model_index.json to not only define the library+model type for each component but also an optional model_id to tell where to find it. So that some repo can contain only some fine tuned components but not everything.

{
  "feature_extractor": [ # <- optional `model_id` somewhere here
    "transformers",
    "CLIPFeatureExtractor"
  ],
(...)

In any case, this will have to be discussed :) From a time/planning perspective, I can definitely do some integration myself when it makes sense (and for sure with some guidance :smile:). I will be off next week but over October I will have some time to allocate to it.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[miquel-espinosa]

Would be very nice and handy to have this working

[patrickvonplaten]

cc @sayakpaul think you were already working on something like this no?

[sayakpaul]

Yup. I am on this. Assigning.