Noisy image generations

[machineminded]

Hello, I am getting noisy generations and the likeness is not there. Here is my code:

import torch
import os
import sys
sys.path.append('../photomaker')

from diffusers import EulerDiscreteScheduler,  DPMSolverMultistepScheduler, AutoencoderKL
from photomaker import PhotoMakerStableDiffusionXLPipeline
from huggingface_hub import hf_hub_download

import modules.default_pipeline as pipeline

base_model_path = pipeline.model_base.filename
photomaker_ckpt = hf_hub_download(repo_id="TencentARC/PhotoMaker", filename="photomaker-v1.bin", repo_type="model")

def generate_photomaker(prompt, input_id_images, negative_prompt, steps, seed, width, height, guidance_scale):
    print(f"Using base model: {base_model_path} for PhotoMaker")
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    pipe = PhotoMakerStableDiffusionXLPipeline.from_single_file(
        base_model_path,
        torch_dtype=torch.bfloat16,
        use_safetensors=True,                
        variant="fp16"
    ).to(device)

    pipe.load_photomaker_adapter(
        os.path.dirname(photomaker_ckpt),
        subfolder="",
        weight_name=os.path.basename(photomaker_ckpt),
        trigger_word="img"
    )

    pipe.id_encoder.to(device)
    
    pipe.scheduler =  DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)

    pipe.fuse_lora()

    generator = torch.Generator(device=device).manual_seed(seed)

    images = pipe(
        prompt=prompt,
        input_id_images=input_id_images,
        negative_prompt=negative_prompt,
        num_images_per_prompt=1,
        num_inference_steps=steps,
        width=width,
        height=height,
        start_merge_step=10,
        generator=generator,
        guidance_scale=guidance_scale
    ).images

    return images

Any idea what might be going on?

[Paper99]

I found that you are using dpmsolver. Could you increase the number of steps and try again?

[Paper99]

Maybe you can also choose to use the quality option

[machineminded]

Hello. I did more experimenting today, and it appears to be due to my simple prompt and lack of a negative prompt. Apply a prompt from the style seems to resolve it. I will post my findings in the morning. Thank you.

[Paper99]

Cool! Thanks again for your efforts!

[machineminded]

At this point I'm pretty sure it's just the prompting. I am working on integrating the Fooocus prompt system into the diffusers pipeline. I have combined it with inswapper_128 for some pretty great results 👍👍👍.

[Paper99]

It's really cool!

[machineminded]

It seems like the euler scheduler generates better images than DPM++. I even tried adding euler_at_final but the generations with DPM++ still are noisy. Even with 60+ steps.

DPMSolverMultistepScheduler(algorithm_type="sde-dpmsolver++", euler_at_final=True)

[akshatbisht22]

DPMSolverMultistepScheduler(algorithm_type="sde-dpmsolver++", euler_at_final=True); DPM_hash($sove)=>solution;

[CHELSEA234]

@machineminded Nice works! Could you please elaborate a bit more on how you combining photomaker with inswapper_128 for these nice photos?

[machineminded]

@machineminded Nice works! Could you please elaborate a bit more on how you combining photomaker with inswapper_128 for these nice photos?

@CHELSEA234 I am using my fork of Fooocus with Inswapper and Photomaker! Soon instantID will be integrated as well.

To elaborate, I use PhotoMaker to create a likeness "substrate" and then inswapper lays the face on top.