[Bug]: txt2img hires fix in non 1:1 ratios is deforming the image

[miaouxtoo]

Is there an existing issue for this?

• [X] I have searched the existing issues and checked the recent builds/commits

What happened?

MacOS M2 Max 32Gb Ventura 13.2.1 Trying out txt2img + hires fix with Latent (bicubic antialiased) as the upscaler. I'm copying a generation from civitai, and I've seen two other people do this in a forum w. normal results.

To debug this, I've set save before hires fix to on, and I have both results attached. The hires fix works totally fine if I use a 1:1 aspect ratio (e.g. 512x512) It does NOT work if the ratio is not 1:1

Specifically in my case I tried 448 x 640

Steps to reproduce the problem

Model: neverendingdreamNED_bakedVAE LORAs: hipoly3DModelLora_v10 & samdoesartsSamYang_normal

` beautiful, masterpiece, best quality, hiqcgbody, anime, 1girl, sam yang, halter cut dress, mascara, flirty, blunt bangs, portrait shot, looking at viewer, glossy lipstick, nightclub, intricate details,lora:hipoly3DModelLora_v10:0.5 lora:samdoesartsSamYang_normal:0.5

Negative prompt: (painting by bad-artist:0.9), easynegative, (worst quality, low quality:1.4), watermark, text, error, blurry, jpeg artifacts, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, artist name, bad anatomy, deformed iris.

Size: 448x640, Seed: 3686603218, Steps: 28, Sampler: DPM++ SDE Karras, CFG scale: 7, Clip skip: 2, Model hash: d8722b4a4d, Hires steps: 20, Hires upscale: 1.75, Hires upscaler: Latent (bicubic antialiased), Denoising strength: 0.55 `

What should have happened?

1:1 aspect ratio generates and upscales directly from txt2img just fine with hires fix. (with 1.75x)

Any other aspect ratio squashes the pre-hires generation out of shape. I've seen examples of it working, I just don't know why mine does not upscale correctly with hires fix.

I have tested this with Latent (bicubic antialiased) and R-ESRGAN 4x+

Commit where the problem happens

0cc0ee1bcb4c24a8c9715f66cede06601bfc00c8

What platforms do you use to access the UI ?

No response

What browsers do you use to access the UI ?

No response

Command Line Arguments

no

List of extensions

clip-interrogator-ext openpose-editor sd-dynamic-prompts sd-webui-additional-networks sd-webui-controlnet sd-webui-depth-lib

Console logs

./webui.sh --device=cpu 

################################################################
Install script for stable-diffusion + Web UI
Tested on Debian 11 (Bullseye)
################################################################

################################################################
Running on mia user
################################################################

################################################################
Repo already cloned, using it as install directory
################################################################

################################################################
Create and activate python venv
################################################################

################################################################
Launching launch.py...
################################################################
Python 3.10.10 (main, Feb 16 2023, 02:49:39) [Clang 14.0.0 (clang-1400.0.29.202)]
Commit hash: 0cc0ee1bcb4c24a8c9715f66cede06601bfc00c8
Installing requirements for Web UI


Installing sd-dynamic-prompts requirements.txt

Launching Web UI with arguments: --device=cpu --upcast-sampling --no-half-vae --use-cpu interrogate
Warning: caught exception 'Torch not compiled with CUDA enabled', memory monitor disabled
No module 'xformers'. Proceeding without it.
==============================================================================
You are running torch 1.12.1.
The program is tested to work with torch 1.13.1.
To reinstall the desired version, run with commandline flag --reinstall-torch.
Beware that this will cause a lot of large files to be downloaded, as well as
there are reports of issues with training tab on the latest version.

Use --skip-version-check commandline argument to disable this check.
==============================================================================
[AddNet] Updating model hashes...
0it [00:00, ?it/s]
[AddNet] Updating model hashes...
0it [00:00, ?it/s]
Loading weights [d8722b4a4d] from /Users/mia/stable-diffusion-webui/models/Stable-diffusion/neverendingDreamNED_bakedVae.safetensors
Creating model from config: /Users/mia/stable-diffusion-webui/configs/v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: /Users/mia/stable-diffusion-webui/models/VAE/vae-ft-mse-840000-ema-pruned.safetensors
Applying cross attention optimization (InvokeAI).
Textual inversion embeddings loaded(4): ulzzang-6500-v1.1, EasyNegative, bad-artist-anime, bad-artist
Model loaded in 3.1s (create model: 0.5s, apply weights to model: 1.5s, apply half(): 0.3s, load VAE: 0.2s, move model to device: 0.4s).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.

Additional information

No response

[Xyem]

Does this still happen if one of the base dimensions is 512 e.g. 384x512 -> HighRes @ 2x for 768x1024?

[miaouxtoo]

Test1: base 512x768, hires 2x = some kind of weird primary color discoloration in the lower half, the upper half looks like a sepia view of the face through frosted glass. Doesn't seem to deform (yet), just looks like a botched hires-post. Possibly a memory issue?

Test2: base 512x768, hires 1.5x = works. Upscale is good. Her facial dimensions are good. Test3: base 448x640, hires 1.75x = deformed face. Test4: base 448x640, hires 1.5x = works. Upscale is good. Her facial dimensions are good.

[Xyem]

Just had a look through your log (which doesn't include the generation, could you add one?) and I noticed that you are using a model with a baked VAE and also loading another VAE. Your mentioning that the colours are misbehaving in two separate ways in the same image makes me suspect this is playing a part.

Loading weights [d8722b4a4d] from /Users/mia/stable-diffusion-webui/models/Stable-diffusion/neverendingDreamNED_bakedVae.safetensors
[..snip..]
Loading VAE weights specified in settings: /Users/mia/stable-diffusion-webui/models/VAE/vae-ft-mse-840000-ema-pruned.safetensors

Does this deformation happen with base SD model and/or with no VAE?

[miaouxtoo]

I removed the VAE from the model (set to automatic) 00006-765230736.txt 00007-3346021222.txt

The colours misbehaving are, I think due to memory mismanagement somehow. It only does it for larger hires fix generations, and always look similar in how they discolor. It does this once I go over 1024 / 1128 ish regardless of ratio.

E.g. (this is with default base SD model w. automatic VAE) 448x640 at 2x hi res = discoloured 512x512 at 2x hi res = okay. 512x512 at 2.5x hi res = discoloured

[miaouxtoo]

Closing this bug. Some of the results are repeated for other Mac users in a more detailed optimisation thread here:

https://github.com/AUTOMATIC1111/stable-diffusion-webui/discussions/7453