[Docs] Doc Sprint on Wednesday Sep 7

[patrickvonplaten]

** There is still more TODO, to see open options, see this comment**

Let's add them docs.

• Create a minimum toctree with mostly blank pages to get it working
• Merge: https://github.com/huggingface/moon-landing/pull/3415
• Do a doc sprint right after

cc @anton-l

[patrickvonplaten]

Docs are now live :tada:

Thanks to @natolambert we have a rough template of diffusers docs: https://huggingface.co/docs/diffusers/index

In two days, we want to kick-start a doc-sprint where we add as many doc-strings as possible. Here the rough draft: https://docs.google.com/document/d/1TwqMlQrjWXi5a0y-C-Wbzx70R4ZQP8f8mq-ARSqB5fI/edit?usp=sharing

Tomorrow we will share more details on what exactly what could be worked on. Let us know if you'd like to participate in the doc-sprint on Wednesday with a :+1:

[patrickvonplaten]

The general doc structure (which is defined here) should look more or less as follows:

"1. Get started" I think the following subsections for "1. Get started" would be nice: "1.1" Diffusers (as is there already), but needs to be adapted "1.2" Quicktour (as is there already), but needs to be adapted “1.3” Installation

“2. Using Diffusers” We only have vision at the moment so let’s maybe just focus on this:

2.1 Loading pretrained instances (explain from_pretrained for all pipelines, models, schedulers)

2.2 Pipelines for Inference 2.1.1 Unconditional Generation 2.1.2 Conditional Generation 2.1.3 Image2Image 2.1.4 In-painting 2.1.5 Create a custom pipeline

"3. Custom Hardware / Optimizing"

pipelines for inference (fp16, autocast, onnx, mps, OpenVino)

“4. Training Diffusers”

3.1 Unconditional Training 3.2 Text-inversion 3.3 Conditional Training (TODO)

“5. Conceptual guides”

"4.1" Here I'd maybe now make a whole section for "Stable Diffusion" with subsections like ethics, optimization, the Hub, ... "4.2" Philosophy (as is there already), but would need to be adapted "4.3" Contribution (we can more or less copy what we have here: )

“5. API”

This is pretty self-explanatory - all public methods need some nice docstring. For now I’d put all models (UNet2D, UNet2DCondition) in one doc string file and also put all schedulers in one doc string file. The each pipeline should get a “Pipeline card” just like we do for Transformers models -> we could just copy-paste the format from: https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines/stable_diffusion (which could be the stable_diffusion.mdx model card) to all other models.

[patrickvonplaten]

For section 5.

We need to adapt both the code files, e.g. add nice doc-strings to each public function and class as well as add some .mdx files for the main API classes.

In general section 5 will look as follows:

API
- Main classes 
  - Models (one file with info and docstrings about `ModelMixin`, `Unet2DModel` ,`Unet2DConditionModel`, `VQModel`, `AutoencoderKLModel` 
  - Schedulers (one file with info and docstring about all scheduler classes - cc @natolambert)
  - DiffusionPipeline (one file - only with `DiffusionPipeline` class)
  - Logging (can copy from transformers: https://huggingface.co/docs/transformers/main_classes/logging)
  - Config (one file - only with `ConfigMixin`) and `register_to_config`
  - Outputs (one file with all `BaseOutput` classes)
  
 - Pipelines (similar in spirit to Transformers models): https://huggingface.co/docs/transformers/model_doc/albert
  - Stable Diffusion (one file which can be a copy of: https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines/stable_diffusion)
  - DDPM (we should create a README similar to https://github.com/huggingface/diffusers/tree/main/src/diffusers/pipelines/stable_diffusion

[patrickvonplaten]

Communication tomorrow will happen over a discord and we'll also hold a call at 9h30am Paris time. Please make sure to join discord to join the call and to participate :-)

To join: Follow this link https://discord.gg/bYA2G9yJtP, then click on the 🎨 emoji to get access to the diffusers group. There you'll find the "core-contributions" channel where we'll post all the important info!

[natolambert]

Working on scheduler updates in #376.

[patrickvonplaten]

For anybody who wants to join - ideally we start with 5. API docs for pipelines. Just pick one of the following pipelines and follow the PR: https://github.com/huggingface/diffusers/pull/377/files

• You should add a pipeline card, e.g. latent_diffusion.mdx
• add the file to the _toctree.yml: https://github.com/huggingface/diffusers/blob/main/docs/source/_toctree.yml
• Complete all pipeline file in your pipeline folder with correct docstrings in the code, e.g. https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/latent_diffusion/pipeline_latent_diffusion.py
  • Here make sure that at least the following functions and classes have a nice docstring: def __init__ and def __call__
  • Make sure to add the classes you want to expose to your pipeline card so that they are correctly displayed in the docs

Pick one of the following pipelines to get started :rocket: & please note below which pipeline you would like to work on with an open PR :-)

• 1. [x] 1. ddim
• 2. [x] 2. ddpm
• 3. [x] 3. latent_diffusion
• 4. [x] 4. latent_diffusion_uncond
• 5. [x] 5. pndm
• 6. [x] 6. score_sde_ve
• 7. [x] 7. stable_diffusion
• 8. [x] 8. stochatic_karras_ve

[patrickvonplaten]

Taking latent diffusion in this PR: https://github.com/huggingface/diffusers/pull/377/files

[pcuenca]

I can take stable diffusion if that's ok.

[kashif]

I can take 2 ddpm and pndm

[satpalsr]

I'd like to take latent_diffusion_uncond.

[patrickvonplaten]

Template is ready: https://github.com/huggingface/diffusers/pull/385

=> We can start filling it out :-) (Note that I copy-pasted quicktour.mdx to all doc pages and models.mdx to all API pages to be faster. Make sure to delete / overwrite those :-)

[patrickvonplaten]

First example you can copy from: https://github.com/huggingface/diffusers/pull/377#pullrequestreview-1098963887

[kashif]

i can do ddim karras and score_sde_ve next

[natolambert]

Okay, seems like there is some interest from North America timezone folks! Here's a list of pages that could use some contributions by category. These are a little more open-ended than the Pipelines which focus on one system.

You can see the latest docs here where one can easily find blank pages.

Additionally, doc strings are very important! You can choose some code that no one mentioned above and write docstrings for every public method (I have a lot unmerged yet for schedulers).

Important docstrings:

• [ ] UNet2DModel: https://github.com/huggingface/diffusers/blob/1a431ae886b516d13f49c1f8a1e1e68d3159eab4/src/diffusers/models/unet_2d.py#L25
• [ ] UNet2DConditionModel: https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/unet_2d_condition.py#L25
• [ ] VQModel: https://github.com/huggingface/diffusers/blob/1a431ae886b516d13f49c1f8a1e1e68d3159eab4/src/diffusers/models/vae.py#L370
• [ ] AutoencoderKL: https://github.com/huggingface/diffusers/blob/1a431ae886b516d13f49c1f8a1e1e68d3159eab4/src/diffusers/models/vae.py#L450

Doc updates by section

Using diffusers

Loading

• [ ] loading models / pipelines / schedulers
• [ ] configuration models / pipelines / schedulers

Pipelines for inference

These are conceptual guides covering different applications -- could be based off previous colabs released with diffusers:

• [ ] Unconditional Image Generation
• [ ] Text-to-Image Generation
• [ ] Text-Guided Image-to-Image
• [ ] Text-Guided Image-Inpainting
• [ ] Create a custom pipeline

Special hardware

These may be a bit harder unless you have experience with them (All except fp16 are available, fp16 covered in #404 )

[natolambert]

_toctree had a typo that made the API heading undefined.

Fix here #406 .

[patrickvonplaten]

Thanks!

[natolambert]

Large schedulers push done here in #376, ready for review.

[natolambert]

Did a quick scan for typos in #408

[satpalsr]

@patrickvonplaten Can you check if #413 and #414 look good?

[utsavnandi]

Hi, there is a minor mistake in https://github.com/huggingface/diffusers/blob/main/docs/source/installation.mdx #402

git clone https://github.com/huggingface/diffusers.git cd transformers pip install -e .

The user should cd to diffusers not transformers ! Don't know where to report this, so mentioning this here.

[daspartho]

@utsavnandi Thank you for pointing it out; I've created a PR to address it :) #453