Scale and shift in Resnet block

[lukovnikov]

Implemented optional scale and shift (instead of currently only shift) based on timestep embedding, similarly to OpenAI's improved diffusion code.

However, it is currently not being used and I would appreciate some suggestions on how to best hook it up. Was time_embedding_norm intended for this (it is not being used)?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[patrickvonplaten]

Hey @lukovnikov :-)

What exactly is the use case here? I think we should only add such features if it enables a new model addition :-) If the changes make it possible to run a new model architecture, then totally fine to add it !

[lukovnikov]

Hey @lukovnikov :-)

What exactly is the use case here? I think we should only add such features if it enables a new model addition :-) If the changes make it possible to run a new model architecture, then totally fine to add it !

@patrickvonplaten I think it's a very standard model variant, it's there by default in lucidrains/Phil Wang diffusion code, which is covered in this Huggingface blog post and it's there in OpenAI's code for the improved diffusion paper. Currently, I'm only using it to train from scratch.

[patrickvonplaten]

Hey @lukovnikov,

I think we would need to port the whole OpenAI model to implement this change. Note that this code is used by all diffusion models so it's hard to allow adapting the core diffusion model code just for training experiments.

If we official support a new model architecture like OpenAI's GLIDE model, I'm fine with doing such a change (if we include a bunch of tests), but it's sadly too much maintenance to allow these changes just for training experiments.

Would you maybe be interested in porting the whole OpenAI model and then one has already a clear use case for this config?

[lukovnikov]

Hi @patrickvonplaten I understand. I don't think I'll be able to port GLIDE alone right now.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.