[Feature Request]: SmirkingFace model support

[Evil-Dragon]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What would your feature do ?

The EBL model given here, requires an update to A111 to load the model correctly: https://www.patreon.com/smirkingface

Code here: https://github.com/smirkingface/stable-diffusion

Sorry, i can't code so if someone wants to take this one as a pull request.

Proposed workflow

Thank you.

Additional information

No response

[B34STW4RS]

So this model with zero patrons? You want support integrated? This really looks like something the creator of the patreon would be doing, no?

[Chilluminati91]

So this model with zero patrons? You want support integrated? This really looks like something the creator of the patreon would be doing, no?

Seems like you can download the model for free. Patreon just for support if I understand it right.

[DenkingOfficial]

@B34STW4RS EBL model has a new extended architecture, so it will produce better results. It can help other people train better models, and I think SmirkingFace model support will be nice in this webui

[B34STW4RS]

@B34STW4RS EBL model has a new extended architecture, so it will produce better results. It can help other people train better models, and I think SmirkingFace model support will be nice in this webui

Oh I'm all for it, It just seems like a strange way to go about asking for support with a patreon first approach considering this is an open source repo.

[Evil-Dragon]

I downloaded the model today and thought it would just work straight away in A111 but then realised it needed extra code. Just as extra coding was needed to support NAI/VAE I thought this would make a useful additional model for A111 and support additional models that needed this in the future.

[Ehplodor]

Maybe it's naive, but shouldn't support for this or that model architecture be the prerogative of those who actually modify the architecture?

[aleksusklim]

shouldn't support for this or that model architecture be the prerogative of those who actually modify the architecture?

You can say the same about every single feature of any recent research like VAE or Hypernetworks too.

[Ehplodor]

shouldn't support for this or that model architecture be the prerogative of those who actually modify the architecture?

You can say the same about every single feature of any recent research like VAE or Hypernetworks too.

You are most probably completely right.

[smirkingface]

Here's how to get it running:

1. Edit launch.py: Find this line:

    run_pip(f"install -r {requirements_file}", "requirements for Web UI")

Add the following line after it:

   run_pip(f"install git+https://github.com/smirkingface/stable-diffusion", "smirkingface")

2. Copy v1-inference.yaml from the configs directory of the original stable-diffusion repository, name it SF_EBL_1.0_vae.yaml and place in the webui models/Stable-diffusion directory along with SF_EBL_1.0_vae.ckpt, and modify unet_config:

    unet_config:
      target: sd.modules.extensions.unet.ExtendedUNetModel
      params:
        kernel_size: 5
        skip_connection_conv: True

3. Run webui like normal, the model will show up in the model list.

• Comment edited to reflect the rather hidden functionality of adding a yaml file along with a model checkpoint.

Disclaimer: Not every webui functionality may work

[ilikenwf]

What is the actual diff between mainline SD and this that you've adjusted?

I ask because your fork doesn't have all our optimizations from here and so it is half or less the normal speed.

[uwusensei]

i think can this be refactored as an extension pretty easily.

looks like the only big difference is the reliance on the ExtendedUNetModel and the ExtendedSpatialTransformer, both of which subclass normal SD classes and we already have in the repo. as long as they're loaded on startup and we can reference the class (e.g. extensions.smirkingface.sd.modules.extensions.unet.ExtendedUNetModel if we put it under a smirkingface folder under extensions/ ) there doesn't superficially appear to be many other changes needed to make this work with this repo, though my guess is we'll always need the yaml file just because it needs the custom class

that said I don't have an interest in this model soo.. not it.

[smirkingface]

With the yaml files I don't think there's much need to make adaptations specifically for this model. In general you can't expect those optimizations to work for all modified architectures anyway.

There are a few general issues with models using custom configuration files that I've noticed. When I have time I'll get around to make a pull request or two.

[SeleDreams]

I still end up getting this kind of errors when loading it with the config etc

size mismatch for model.diffusion_model.output_blocks.7.0.in_layers.2.weight: copying a param with shape torch.Size([640, 1280, 5, 5]) from checkpoint, the shape in current model is torch.Size([640, 1280, 3, 3]).
size mismatch for model.diffusion_model.output_blocks.7.0.out_layers.3.weight: copying a param with shape torch.Size([640, 640, 5, 5]) from checkpoint, the shape in current model is torch.Size([640, 640, 3, 3]).
size mismatch for model.diffusion_model.output_blocks.8.0.in_layers.2.weight: copying a param with shape torch.Size([640, 960, 5, 5]) from checkpoint, the shape in current model is torch.Size([640, 960, 3, 3]).
size mismatch for model.diffusion_model.output_blocks.8.0.out_layers.3.weight: copying a param with shape torch.Size([640, 640, 5, 5]) from checkpoint, the shape in current model is torch.Size([640, 640, 3, 3]).
size mismatch for model.diffusion_model.output_blocks.9.0.in_layers.2.weight: copying a param with shape torch.Size([320, 960, 5, 5]) from checkpoint, the shape in current model is torch.Size([320, 960, 3, 3]).
size mismatch for model.diffusion_model.output_blocks.9.0.out_layers.3.weight: copying a param with shape torch.Size([320, 320, 5, 5]) from checkpoint, the shape in current model is torch.Size([320, 320, 3, 3]).
size mismatch for model.diffusion_model.output_blocks.10.0.in_layers.2.weight: copying a param with shape torch.Size([320, 640, 5, 5]) from checkpoint, the shape in current model is torch.Size([320, 640, 3, 3]).
size mismatch for model.diffusion_model.output_blocks.10.0.out_layers.3.weight: copying a param with shape torch.Size([320, 320, 5, 5]) from checkpoint, the shape in current model is torch.Size([320, 320, 3, 3]).
size mismatch for model.diffusion_model.output_blocks.11.0.in_layers.2.weight: copying a param with shape torch.Size([320, 640, 5, 5]) from checkpoint, the shape in current model is torch.Size([320, 640, 3, 3]).
size mismatch for model.diffusion_model.output_blocks.11.0.out_layers.3.weight: copying a param with shape torch.Size([320, 320, 5, 5]) from checkpoint, the shape in current model is torch.Size([320, 320, 3, 3]).

[SeleDreams]

never mind, it didn't save the config modifications properly

however now it crashes without any specific error, it just says that the process has stopped

making attention of type 'vanilla' with 512 in_channels
Loading weights [758fda47] from /home/seledreams/stable-diffusion-webui/models/Stable-diffusion/model.ckpt
./webui.sh : ligne 140 :  4811 Processus arrêté      "${python_cmd}" "${LAUNCH_SCRIPT}" "$@"

[medledan]

Disclaimer: Not every webui functionality may work

I'll wait until the devs integrate this. Every time I mess with programming code I destroy the program I'm editing.

[catboxanon]

Closing as stale.