Want to use "strict=False" in from_pretrained

[CuddleSabe]

What API design would you like to have changed or added to the library? Why? I want to add param "strict" in from_pretrained. I want to add zerosft controlnet, but it can only implement by adding the set layers to unet class, when I use from_pretrained, it can't use "strict=False"

What use case would this enable or better enable? Can you give us a code example?

def from_pretrained(cls, pretrained_model_name_or_path: Optional[Union[str, os.PathLike]], strict: bool=True, **kwargs):

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yiyixuxu]

Hi @CuddleSabe can you provide a link to the zerosft controlnet you want to use? the short answer is that we would not want to add "strict=False" to from_pretrained but we can look into supporting this new controlnet model from diffusers

cc @asomoza here, have you checked out this model? do we want to support it?

[asomoza]

For what I understand it's not a controlnet but a connector used for the controlnet used in SUPIR:

It designs an effective 620 million parameter adaptor to steer the generative prior for image restoration. The paper introduces techniques like a robust encoder, trimmed architecture, and a new ZeroSFT connector to make model scaling feasible.

I think @CuddleSabe is trying to port SUPIR to diffusers but I don't know if as a contribution or as a closed source private solution.

[CuddleSabe]

For what I understand it's not a controlnet but a connector used for the controlnet used in SUPIR:

It designs an effective 620 million parameter adaptor to steer the generative prior for image restoration. The paper introduces techniques like a robust encoder, trimmed architecture, and a new ZeroSFT connector to make model scaling feasible.

I think @CuddleSabe is trying to port SUPIR to diffusers but I don't know if as a contribution or as a closed source private solution.

Hi, I solve it by create new class inherited from "CrossAttnUpBlock2D" 、“UpBlock2D” and "UNetMidBlock2DCrossAttn" but my edition is used to implement sdv1.5 SUPIR (the official is SDXL)

[CuddleSabe]

For what I understand it's not a controlnet but a connector used for the controlnet used in SUPIR:

It designs an effective 620 million parameter adaptor to steer the generative prior for image restoration. The paper introduces techniques like a robust encoder, trimmed architecture, and a new ZeroSFT connector to make model scaling feasible.

I think @CuddleSabe is trying to port SUPIR to diffusers but I don't know if as a contribution or as a closed source private solution.

The SUPIR ControlNet can directly load into diffusers' Controlnet class, their outputs have same shapes. To support SUPIR, maybe we have to create new classes inherited from diffusers, new classes have "zerosft".

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.