Issue in using training AnimateLCM SVD

[habibian]

Thanks for the great work, also for releasing the training script train_svd_lcm.py.

I am trying to reproduce the results using the provided train_svd_lcm.py, but after half of the training (20,000 / 50,000 itrs) don't see any improvement neither in loss value nor generation qualities (training on a single A100 on WebVid2M).

Could you please confirm if Ishould set the hyper-params as follows?

accelerate launch train_svd_lcm.py \ --pretrained_model_name_or_path=stabilityai/stable-video-diffusion-img2vid-xt \ --per_gpu_batch_size=1 --gradient_accumulation_steps=1 \ --max_train_steps=50000 \ --width=576 \ --height=320 \ --checkpointing_steps=1000 --checkpoints_total_limit=1 \ --learning_rate=1e-6 --lr_warmup_steps=1000 \ --seed=123 \ --adam_weight_decay=1e-3 \ --mixed_precision="fp16" \ --N=40 \ --validation_steps=500 \ --enable_xformers_memory_efficient_attention \ --gradient_checkpointing \ --output_dir="outputs" \

In the current train_svd_lcm.py, the model is being trained on 576x320 resolutions, which is much lower than the standard SVD, i.e., 1024x572. Would not this cause a problem as normal (non LCM) SVD suffer from generating lower resolution videos?

Any input is much appreciated :)