SD3 FineTuning Setup Guide

[vikas784]

🛠️ SD3 FineTuning Setup (Complete Guide)

1. ✅ Training Command

accelerate launch --mixed_precision bf16 --num_cpu_threads_per_process 1 sd3_train.py \
  --save_model_as safetensors \
  --train_data_dir "/workspace/sdxl/sd3/sd-scripts/exp_1_sd_3_training_Data" \
  --in_json "/workspace/sdxl/sd3/sd-scripts/exp_1_sd_3_training_Data/meta_lat.json" \
  --cache_latents_to_disk \
  --cache_text_encoder_outputs_to_disk \
  --pretrained_model_name_or_path /workspace/sdxl/sd3/sd3_weights/sd3.5_large.safetensors \
  --clip_l /workspace/sdxl/sd3/sd3_weights/text_encoders/clip_l.safetensors \
  --clip_g /workspace/sdxl/sd3/sd3_weights/text_encoders/clip_g.safetensors \
  --t5xxl /workspace/sdxl/sd3/sd3_weights/text_encoders/t5xxl_fp16.safetensors \
  --t5xxl_max_token_length 512 \
  --fp8_base \
  --highvram \
  --seed 42 \
  --gradient_checkpointing \
  --mixed_precision bf16 \
  --save_precision bf16 \
  --output_dir /workspace/sdxl/sd3/exp_1_weights \
  --output_name exp_1_sd3 \
  --persistent_data_loader_workers \
  --max_data_loader_n_workers 2 \
  --optimizer_type adafactor \
  --optimizer_args "relative_step=False" "scale_parameter=False" "warmup_init=False" \
  --lr_scheduler constant_with_warmup \
  --max_grad_norm 0.0 \
  --max_train_epochs 4 \
  --save_every_n_steps 1000 \
  --learning_rate 5e-5 \
  --sdpa \
  --fused_backward_pass \
  --full_bf16

2. 📁 Dataset Directory Structure

/workspace/sdxl/sd3/sd-scripts/exp_1_sd_3_training_Data/
├── Image_1.JPG
├── Image_1.txt
├── Image_2.JPG
├── Image_2.txt
└── meta_lat.json

3. 🧾 JSON Metadata Format (meta_lat.json)

{
  "Image_1": {
    "caption": "prompt text",
    "train_resolution": [1216, 832]
  },
  "Image_2": {
    "caption": "prompt text",
    "train_resolution": [1216, 832]
  }
}