progrockdiffusion
progrockdiffusion copied to clipboard
lowfuel
Symmetry causes intense pink or green coloring
Hi,
when symmetry_loss_v is set to true and the symm_loss_scale is high (10k - 20k) the whole picture (and any intermediate save) will be intensely colored (pink or green). This happens even with grey perlin noise and cut_icgray_p set to 1.
Not sure how to combat this even with lots of saturation scale and low or zero no cut_icgray the color does not bounce back. Lower symm_loss_scale values seem to nullify the symmetry setup.
By the way, I love progrockdiffusion! Thanks so much for it!.
Hi there! Yes, someone else reported this over the weekend. I will look into it today. If you could post your full settings it would be useful.
Try lowering the symmetry to 2000. I found 20000 doing the same thing. Also longer prompts and certain models have a greater impact. I haven't figured it out completely, once they start pink I rewind a bunch of steps and haven't got the exact cause down.
@KnoBuddy in this specific case the setting is even 500, i tried higher and lower settings, as long it is not much lower (50) it produces these colorings.
The only weird thing I see is your cut_ic_grey, which has 1 for the first two sections (which means 100% of innercuts will be gray), and then after that the values have commas instead of decimal points (I have no idea how that will get parsed).
Maybe try changing it to:
cut_icgray_p: "[0.7]*100+[0.7]*100+[0.6]*100+[0.2]*100+[0.0]*600",
Not sure if that will help, but it stood out to me as a possible problem.
Yes, this was one idea I had to combat it - I hoped together with grey perlin noise that it could push back the coloring. Here is an example that is even more intense that does not have this:
Default_183_settings.txt
Well I just tried your prompt but with my basic settings (+ symmetry), here's what I got:
And the settings if you want to compare to yours:
{
batch_name: "Default",
text_prompts: {
"0": [
"Matte painting portrait of a germanic man by Bastien Lecouffe-Deharme:3",
"slender, symmetrical face:2",
"artstation:1.5",
"blue eyes, a pixie cut:1",
],
},
n_batches: 1,
steps: 142,
display_rate: 10,
width_height_scale: 1,
width: 512,
height: 512,
set_seed: 1162862161,
image_prompts: {},
clip_guidance_scale: "[16166]*500+[8166]*500",
tv_scale: 0,
range_scale: 150,
sat_scale: 0,
cutn_batches: "[1]*400+[2]*600",
cutn_batches_final: null,
max_frames: 1000,
interp_spline: "Linear",
init_image: null,
init_masked: null,
render_mask: null,
init_scale: 1000,
skip_steps: 0,
frames_scale: 1500,
frames_skip_steps: "60%",
perlin_init: false,
perlin_mode: "mixed",
skip_augs: false,
randomize_class: true,
clip_denoised: false,
clamp_grad: true,
clamp_max: "[0.05]*600+[0.15]*400",
fuzzy_prompt: false,
rand_mag: 0.05,
eta: 0.35,
diffusion_model: "portrait_generator",
use_secondary_model: false,
diffusion_steps: 994,
sampling_mode: "ddim",
ViTB32: 1.0,
ViTB16: 1.0,
ViTL14: 1.0,
ViTL14_336: 0.0,
RN101: 0.0,
RN50: 1.0,
RN50x4: 0.0,
RN50x16: 0.0,
RN50x64: 0.0,
ViTB32_laion2b_e16: 0.0,
ViTB32_laion400m_e31: 0.0,
ViTB32_laion400m_32: 0.0,
ViTB32quickgelu_laion400m_e31: 0.0,
ViTB32quickgelu_laion400m_e32: 0.0,
ViTB16_laion400m_e31: 0.0,
ViTB16_laion400m_e32: 0.0,
RN50_yffcc15m: 0.0,
RN50_cc12m: 0.0,
RN50_quickgelu_yfcc15m: 0.0,
RN50_quickgelu_cc12m: 0.0,
RN101_yfcc15m: 0.0,
RN101_quickgelu_yfcc15m: 0.0,
cut_overview: "[12]*400+[4]*600",
cut_innercut: "[4]*400+[12]*600",
cut_ic_pow: "[1]*500+[10]*500",
cut_ic_pow_final: null,
cut_icgray_p: "[0.2]*400+[0]*600",
smooth_schedules: true,
animation_mode: "None",
key_frames: true,
angle: "0:(0)",
zoom: "0: (1), 10: (1.05)",
translation_x: "0: (0)",
translation_y: "0: (0)",
video_init_path: "/content/training.mp4",
extract_nth_frame: 2,
stop_early: 0,
fix_brightness_contrast: true,
adjustment_interval: 10,
high_contrast_threshold: 80,
high_contrast_adjust_amount: 0.85,
high_contrast_start: 20,
high_contrast_adjust: true,
low_contrast_threshold: 20,
low_contrast_adjust_amount: 2,
low_contrast_start: 20,
low_contrast_adjust: true,
high_brightness_threshold: 180,
high_brightness_adjust_amount: 0.85,
high_brightness_start: 0,
high_brightness_adjust: true,
low_brightness_threshold: 40,
low_brightness_adjust_amount: 1.15,
low_brightness_start: 0,
low_brightness_adjust: true,
sharpen_preset: "Off",
keep_unsharp: false,
gobig_scale: 2,
gobig_skip_ratio: 0.6,
gobig_overlap: 64,
symmetry_loss_v: true,
symmetry_loss_h: false,
sloss_scale: 2400,
symm_switch: 50,
perlin_brightness: 1.0,
perlin_contrast: 1.0,
use_jpg: false,
}
I could not render your setup, changes:
"cutn_batches": 1, "cut_overview": "[8]*400+[1]*600", "cut_innercut": "[1]*400+[8]*600",
everything else is the same, this is the outcome:
could it be that low amounts of cuts cause this?
Okay, no worries. Changing mine to what you used there, will see how it goes
If we're using identical settings the only difference should be speed. Images should look almost identical. So there must be something else different. You're using the latest PRD files I assume?
One lead could be i had a problem during the installation... the solution was: pip install --upgrade pywin32==228
following this thread: https://github.com/conda/conda/issues/11503
Hmm I haven't heard of that before but I can't imagine how that would effect the output of PRD in any way.
I tried it on my 3070 (with the smaller settings), and got the same pink result.
I also had to change
diffusion_model: "portrait_generator"
to
portrait_generator_v001
to get it to work at all.
I also tried it with different portrait generator versions, the 1.5 version does this:
Well, I'm stumped. I don't know how to troubleshoot it if I can't reproduce it. :/
Two questions: Does it happen with Symmetry off? And does it happen with the regular 512x512 finetune model (symmetry on and off)? I guess what I'm getting at is whether or not this is a symmetry problem, a model problem, or a combination. Or maybe it's something else entirely.
Are you using a different model with:
diffusion_model: "portrait_generator"
That just gives an error for me.
portrait_generator_v001 + symmetry off:
512x512_diffusion_uncond_finetune_008100 + symmetry:
needed to reduce the cuts a lot to: cut_overview: "[5]*400+[2]*600", cut_innercut: "[2]*400+[5]*600",
Confirm that reducing the cuts as @iceandstorm above means don't need to use secondary model, and then get same pink result.
batch_name: "Default",
text_prompts: {
"0": [
"Matte painting portrait of a germanic man by Bastien Lecouffe-Deharme:3",
"slender, symmetrical face:2",
"artstation:1.5",
"blue eyes, a pixie cut:1",
],
},
n_batches: 1,
steps: 142,
display_rate: 10,
width_height_scale: 1,
width: 512,
height: 512,
set_seed: 1162862161,
image_prompts: {},
clip_guidance_scale: "[16166]*500+[8166]*500",
tv_scale: 0,
range_scale: 150,
sat_scale: 0,
cutn_batches: "[1]*400+[2]*600",
cutn_batches_final: null,
max_frames: 1000,
interp_spline: "Linear",
init_image: null,
init_masked: null,
render_mask: null,
init_scale: 1000,
skip_steps: 0,
frames_scale: 1500,
frames_skip_steps: "60%",
perlin_init: false,
perlin_mode: "mixed",
skip_augs: false,
randomize_class: true,
clip_denoised: false,
clamp_grad: true,
clamp_max: "[0.05]*600+[0.15]*400",
fuzzy_prompt: false,
rand_mag: 0.05,
eta: 0.35,
diffusion_model: "portrait_generator_v001",
use_secondary_model: false,
diffusion_steps: 994,
sampling_mode: "ddim",
ViTB32: 1.0,
ViTB16: 1.0,
ViTL14: 0.0,
ViTL14_336: 0.0,
RN101: 0.0,
RN50: 1.0,
RN50x4: 0.0,
RN50x16: 0.0,
RN50x64: 0.0,
ViTB32_laion2b_e16: 0.0,
ViTB32_laion400m_e31: 0.0,
ViTB32_laion400m_32: 0.0,
ViTB32quickgelu_laion400m_e31: 0.0,
ViTB32quickgelu_laion400m_e32: 0.0,
ViTB16_laion400m_e31: 0.0,
ViTB16_laion400m_e32: 0.0,
RN50_yffcc15m: 0.0,
RN50_cc12m: 0.0,
RN50_quickgelu_yfcc15m: 0.0,
RN50_quickgelu_cc12m: 0.0,
RN101_yfcc15m: 0.0,
RN101_quickgelu_yfcc15m: 0.0,
cut_overview: "[12]*400+[4]*600",
cut_innercut: "[4]*400+[12]*600",
cut_ic_pow: "[1]*500+[10]*500",
cut_ic_pow_final: null,
cut_icgray_p: "[0.2]*400+[0]*600",
smooth_schedules: true,
animation_mode: "None",
key_frames: true,
angle: "0:(0)",
zoom: "0: (1), 10: (1.05)",
translation_x: "0: (0)",
translation_y: "0: (0)",
video_init_path: "/content/training.mp4",
extract_nth_frame: 2,
stop_early: 0,
fix_brightness_contrast: true,
adjustment_interval: 10,
high_contrast_threshold: 80,
high_contrast_adjust_amount: 0.85,
high_contrast_start: 20,
high_contrast_adjust: true,
low_contrast_threshold: 20,
low_contrast_adjust_amount: 2,
low_contrast_start: 20,
low_contrast_adjust: true,
high_brightness_threshold: 180,
high_brightness_adjust_amount: 0.85,
high_brightness_start: 0,
high_brightness_adjust: true,
low_brightness_threshold: 40,
low_brightness_adjust_amount: 1.15,
low_brightness_start: 0,
low_brightness_adjust: true,
sharpen_preset: "Off",
keep_unsharp: false,
gobig_scale: 2,
gobig_skip_ratio: 0.6,
gobig_overlap: 64,
symmetry_loss_v: true,
symmetry_loss_h: false,
sloss_scale: 20000,
symm_switch: 50,
perlin_brightness: 1.0,
perlin_contrast: 1.0,
use_jpg: false,
}```
I had to turn off ViTL14 to run it on my desktop at home but that shouldn't change the color that dramatically.
Are you using a different model with:
diffusion_model: "portrait_generator"
That just gives an error for me.
Sorry, that was just how it was named in my current local version, was working on taking the version # out of the name so we don't have to change it every time it gets updated.
I had to turn off ViTL14 to run it on my desktop at home but that shouldn't change the color that dramatically.
Copy-paste of those settings and run, it looks like this for me:
I am trying to think what else might be different. Can you check your pytorch version for me? Just run this:
pip show torch
and
pip show torchvision
And paste the results here.