Mikubill
[Bug]: CN v1.1.173, Reference Only does not work
Is there an existing issue for this?
- [X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui
What happened?
I tried using the latest updated CN and Reference Only to regenerate the picture I got on NIJI, but obviously it doesn't work. I'm pretty sure I have CN enabled and pixel perfect turned on and CN weight set to 1.25
Steps to reproduce the problem
-
In txt2img, write some Prompts normally, set the sampler and the number of steps.
-
Open CN and enable it, and check Pixel Perfect at the same time. Set the preprocessor to Reference Only. Import my original image.
-
Press Generate. Get an unrelated new image.
What should have happened?
Normally, using Reference Only will get an image that is close to, or related to, the original image. But I use this preprocessor and it doesn't have any effect it should.
Commit where the problem happens
CN: d831043c (Mon May 15 07:18:51 2023)
WebUI: 5ab7f21
python: 3.10.8 • torch: 2.0.0+cu118 • xformers: N/A • gradio: 3.28.1 • commit: [5ab7f213]
What browsers do you use to access the UI ?
Google Chrome, Microsoft Edge
Command Line Arguments
Launching Web UI with arguments: --medvram --theme dark --opt-sdp-attention --disable-nan-check --api --autolaunch
Console logs
Python 3.10.8 (tags/v3.10.8:aaaf517, Oct 11 2022, 16:50:30) [MSC v.1933 64 bit (AMD64)]
Commit hash: 5ab7f213bec2f816f9c5644becb32eb72c8ffb89
Installing requirements
Launching Web UI with arguments: --medvram --theme dark --opt-sdp-attention --disable-nan-check --api --autolaunch
No module 'xformers'. Proceeding without it.
[AddNet] Updating model hashes...
[AddNet] Updating model hashes...
Loading weights [eac6c08a19] from E:\NovelAI-Webui\novelai-webui-aki-v3\models\Stable-diffusion\meinamix_meinaV9.safetensors
Creating model from config: E:\NovelAI-Webui\novelai-webui-aki-v3\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: E:\NovelAI-Webui\novelai-webui-aki-v3\models\VAE\orangemix.vae.pt
Applying scaled dot product cross attention optimization.
Textual inversion embeddings loaded(0):
Model loaded in 107.7s (load weights from disk: 0.2s, create model: 0.4s, apply weights to model: 99.7s, apply half(): 0.5s, load VAE: 6.9s).
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Startup time: 116.0s (import torch: 1.4s, import gradio: 0.9s, import ldm: 0.4s, other imports: 0.7s, list SD models: 0.1s, load scripts: 1.1s, load SD checkpoint: 108.1s, create ui: 2.9s, gradio launch: 0.2s).
Closing server running on port: 7860
Restarting UI...
[AddNet] Updating model hashes...
ControlNet v1.1.173
ControlNet v1.1.173
Running on local URL: http://127.0.0.1:7860
To create a public link, set `share=True` in `launch()`.
Startup time: 4.2s (load scripts: 1.0s, create ui: 2.7s, gradio launch: 0.3s).
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 1024
raw_W = 1024
target_H = 872
target_W = 872
estimation = 872.0
preprocessor resolution = 872
ControlNet used torch.float16 VAE to encode torch.Size([1, 4, 109, 109]).
1it [00:20, 20.74s/it]
2it [00:22, 9.33s/it]
3it [00:23, 5.53s/it]
4it [00:24, 3.95s/it]
5it [00:25, 2.89s/it]
Total progress: 1it [01:42, 102.40s/it]
......
156it [03:01, 1.08s/it]
156it [03:02, 1.17s/it]
Total progress: 52it [04:20, 3.21s/it]
Total progress: 52it [04:23, 3.21s/it]
Total progress: 52it [04:23, 5.07s/it]
Loading preprocessor: reference_adain
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 1024
raw_W = 1024
target_H = 872
target_W = 872
estimation = 872.0
preprocessor resolution = 872
ControlNet used torch.float16 VAE to encode torch.Size([1, 4, 109, 109]).
Loading preprocessor: reference_adain
Pixel Perfect Mode Enabled.
resize_mode = ResizeMode.INNER_FIT
raw_H = 1024
raw_W = 1024
target_H = 872
target_W = 872
estimation = 872.0
preprocessor resolution = 872
Additional information
No response
ah But I need SDP... My PC can't update xFormers, maybe because of some environment configuration, its update always fails. I couldn't solve it, so I used SDP and uninstalled xFormers
ah remove --opt-sdp-attention --disable-nan-check
An observation. I use --opt-sdp-attention in my startup and it all works fine. So I don't think this one is the issue.
@E7admin Any chance you could upload the source image and I can test it out on my machine with the same settings as you.
but there are reports that depth models do not work in sdp. so sdp is very risky
Yes I can upload the original image for everyone to test, but I would say that the CN Tile model is also not friendly to me, I can hardly get any usable results from the use of the CN Tile model. I'm not sure if this has something to do with my use of SDP as I can't test with xFormers (at least not right now).
The original image:
The CN Tile Model - txt2img image:
I don't know why the CN Tile Model needs to add so many wacky details and strokes to the image, I've never seen such results in others and videos. And not just this one, all pictures will have this effect. I'm pretty sure my CN has been updated to v1.1.173 and I'm using the official Tile model.
The results for "Reference_Only" can vary quite a bit depending on your model and prompts.
I used a very simple prompt with the default ControlNet Settings and got this result, no cherry picking, this is the first thing that came out.
1girl, standing, painting
Negative prompt: loli, claws, (Worst quality, bad quality:1.4)
Steps: 35, Sampler: DPM++ SDE Karras, CFG scale: 5, Seed: 572577993, Size: 872x872, Model hash: 61e23e57ea, Model: mixProV4_v4, Clip skip: 2, Version: v1.2.1, ControlNet 0: "preprocessor: reference_only, model: None, weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (64, 0.5, 64)"
Replicating your exact ControlNet settings with the Weight at 1.25 and Ending Control Step at 0.65 made very minor changes to the image.
As a quick check I also tried "Reference_adain+attn" with Style Fidelity 1 and it gave another result pretty close to the source image. Better prompting would likely make it even closer, or a more suitable model.
As for Tile, I've generally had better results with Img2Img. But with Txt2Img I will give an example of what I get.
1girl, picture frame, standing, mountains, clouds, sword, blonde hair, blue eyes, dress, thighhighs, sneakers, metal, planet, painting
Negative prompt: loli, claws, (Worst quality, bad quality:1.4)
Steps: 35, Sampler: DPM++ SDE Karras, CFG scale: 5, Seed: 572577993, Size: 512x512, Model hash: 61e23e57ea, Model: mixProV4_v4, Denoising strength: 0.3, Clip skip: 2, Version: v1.2.1, ControlNet 0: "preprocessor: none, model: control_v11f1e_sd15_tile [a371b31b], weight: 1, starting/ending: (0, 1), resize mode: Crop and Resize, pixel perfect: True, control mode: Balanced, preprocessor params: (64, 1, 64)", Hires upscale: 2, Hires steps: 15, Hires upscaler: 4x-UltraSharp
It seems that my expectations for "Reference Only" are different. I thought it could achieve a more accurate "img2img", but at present, it obviously does not.
ControlNet v1.1.178 Has face lock been applied since the upgraded version? If anyone knows, please explain. When using a partial image, the face is broken. Just the prompt works fine.
