Integration with the Hugging Face Hub

[osanseviero]

Is your feature request related to a problem? Please describe. As I train models, I would like to easily be able to share them with other people and document them well. I would also like to be able to access other trained models from the community.

Describe the solution you'd like I would like to have an integration with the Hugging Face Hub (disclaimer: I'm a member of the OS team there). I would like to be able to do model.push_to_hub("osanseviero/my_vae") and get a model directly in the Hub. Some of the benefits of sharing models through the Hub:

• versioning, commit history and diffs
• repos provide useful metadata about their tasks, languages, metrics, etc that make them discoverable
• multiple features from TensorBoard visualizations, leaderboards, and more

[clementchadebec]

Hello @osanseviero, Thanks for opening this issue! I was actually considering integrating this awesome feature into the library! Are there any resources I should use first to do this integration? Thanks!

[osanseviero]

Yes, we have a guide for this at https://huggingface.co/docs/hub/models-adding-libraries

My suggestion is to begin with adding one-two models manually, then add downstream functionality to download models from the Hub. The automatic push_to_hub could be done with https://huggingface.co/docs/huggingface_hub/how-to-upstream#managing-files-in-a-repo-without-git-with-the-createcommit-api

[clementchadebec]

Hi @osanseviero, Thank you for requesting this feature. It was added in #28 and we can now share and download models though the HuggingFace Hub! To do so you need:

• a valid HuggingFace account
• the package huggingface_hub installed in your virtual env. If not you can install it with

$ python -m pip install huggingface_hub

• to be logged in to your HuggingFace account using

$ huggingface-cli login

Uploading a model to the Hub

Any Pythae model can be easily uploaded using the method push_to_hf_hub

>>> my_vae_model.push_to_hf_hub(hf_hub_path="your_hf_username/your_hf_hub_repo")

Note: If your_hf_hub_repo already exists and is not empty, files will be overridden. In case, the repo your_hf_hub_repo does not exist, a folder having the same name will be created.

Downloading models from the Hub

Equivalently, you can download or reload any Pythae model directly from the Hub using the method load_from_hf_hub

>>> from pythae.models import AutoModel
>>> my_downloaded_vae = AutoModel.load_from_hf_hub(hf_hub_path="path_to_hf_repo")

[osanseviero]

This is very very cool, nice work :partying_face: :hugs:

One recommendation: Models have model cards which are just README.md files which can have metadata at the top. This can include things such as license, metrics, etc. It would be very cool if all models have a tag for pythae as it would allow users to filter models for it. We can even add an automatic code snippet + filter with list of libraries at left of hf.co/models.

[clementchadebec]

Re-opening this issue to add model cards and tags

[clementchadebec]

Hi @osanseviero, I am currently trying to integrate the model cards and tags to the library. I understand that for pythae to be added as a library in hf.co/models I need to add it in Libraries.ts together with the corresponding code snippet. However, from what I understand, the tags must be mentioned in the README.md metadata to be displayed on the hub. This would require other users to always include the following when they push their own README.md to the hub.

---
tags:
- pythae
---

Is there any other way to add the library label automatically when pushing the model without using the README.md? Moreover, I was wondering if there exists an easy way to check if there is an existing README.md for a given model to avoid erasing it when pushing a new updated model.

Thank you for your help!

[osanseviero]

cc @nateraw @nimaboscarino

Yes, the way we add metadata is by creating a README.md file with yaml metadata at the top. Having the small snippet you suggested would work great. huggingface_hub has a couple of utilities to make updating the metadata very very easy. For example, you can use metadata_update to just add the tag. We also have a couple of nice utilities being added to huggingface_hub for easier programmatic model card creation (https://github.com/nateraw/modelcards being merged right now), but the existing tools should work well already. Let us know if you have any feedback!

[NimaBoscarino]

Are there any more tasks to take on for this issue to close it? I'm happy to help out!

[clementchadebec]

Hi @NimaBoscarino, No I think we are good to go. I was just waiting https://github.com/huggingface/hub-docs/pull/227 to be merged before closing this issue in case anything was needed on your side :)

[osanseviero]

The PR is now merged. Thanks a lot for your contribution!

[clementchadebec]

Cool! Thanks for letting me know!