dreambooth Lora training failure

[yayueLaurazhou]

I am using the image dataset used in dreambooth paper here https://github.com/google/dreambooth/tree/main/dataset/can

on model Efficient-Large-Model/Sana_1600M_1024px_BF16_diffusers

accelerate launch /mnt/sdc/zhouyayue/projects/Sana/train_scripts/train_dreambooth_lora_sana.py \
  --pretrained_model_name_or_path="Efficient-Large-Model/Sana_1600M_1024px_BF16_diffusers" \
  --instance_data_dir="/mnt/sdc/zhouyayue/projects/Sana/dreambooth_data" \
  --output_dir="/mnt/sdc/zhouyayue/projects/Sana/output_dreambooth_lora" \
  --mixed_precision="bf16" \
  --instance_prompt="a photo of a <edfe> can" \
  --resolution=1024 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --use_8bit_adam \
  --learning_rate=1e-6 \
  --report_to="wandb" \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=800 \
  --validation_prompt="ground level view, low camera angle, a realistic photo of a <edfe> can on terrazo tile floor" \
  --validation_epochs=50 \
  --seed="0" \
  --push_to_hub 

I changed the learning rate from 1e-4 to 1e-6, the result is below

at 100 epochs, this are the validation images

after 700 epochs, it gives me photos that doesn't even look like a can, not to mention subject specific features,

what is wrong with my training?

[bghira]

it's the model, not your training.

[nanaj96]

can you told to me can we test our diffuser sana lora dreambooth checkpoints model? do you have any workflow?