Merged model artifacts

[Astropulse]

Some merged models are giving strange results:

Expected result (via Automatic1111): prompt "detailed photo of a bowl of fruit" negative prompt "blurry" steps 20 seed 812980511 cfg 7 sampler ddim

Result from ImaginAIry: prompt "detailed photo of a bowl of fruit" negative prompt "blurry" steps 20 seed 812980511 cfg 7 sampler ddim

Model was a merge of a dreambooth model based on 1.5 and https://huggingface.co/Corruptlake/Sci-Fi-Diffusion

No errors, other models work fine, including other merges. This may just be an issue with Sci-Fi-Diffusion, but as it doesn't cause the same issues in base stable diffusion environment, or automatic1111's webui, I thought it may have something to do with how models are being loaded.

[brycedrennan]

I'd guess it's using the wrong config somehow. Does scifi diffusion work by itself?

[Astropulse]

Yep, both models work on their own, it only causes this issue when merged.

Same when using other models + scifi diffusion.

[brycedrennan]

What format is the dreambooth model in? How many GB is it? What software created it? How big is the merged model? What format is it in?

[brycedrennan]

I'd probably need to see the merged weights to figure out what was going on

[Astropulse]

Both models are pickled, the base model is 5.57 GB, and the merged model created in Automatic1111 is a few KB larger. Here's a google drive link to the model causing issues: https://drive.google.com/file/d/1LLQhYQgeF5CqlO2mn8ELQDiKaV32pyvz/view?usp=share_link

[brycedrennan]

Thanks for providing everything asked. I can't "trust" the pickle so I need to find some sort of pickle scanner first before I try it out.

[Astropulse]

Completely understandable, thank you for looking into this.

[brycedrennan]

Looks like the weights contain some data in float16 and some float32. Just guessing that's the issue

[Astropulse]

Are you manually loading VAE's at all? This kind of artifact can be caused by the wrong VAE being loaded.

[brycedrennan]

I'm talking about the weights in the file you linked to. I have not opened it but I did use a pickle inspector to look at the datatypes. It has mixed types.

[Astropulse]

Other merges with fp16 and fp32 were working, here's some confirmation that it's the VAE.

Default VAE:

VAE from https://huggingface.co/stabilityai/sd-vae-ft-mse:

Is there a way to manually specify the VAE file to load from?

[Astropulse]

I've fixed the issue by replacing the VAE contained in the model, but a way to do this on the fly would be quite helpful.