AnimateDiff

~~I may stop develop this repo right now,~~ AnimateDiff is not designed to do I2V mission at first, I spent lots of time to read diffusers source codes, this route maybe not the best compared to webui(ldm injection) at the end.

Though it has potential, I believe new motion model trained on bigger-datasets/specific-motion will be released soon.

Still under development.

TODO

[x] update diffusers to 0.20.1
[x] support IP-Adapter
[x] reconstruction codes and make animatediff a diffusers plugin like sd-webui-animatediff
[x] controlnet from TDS4874
[x] solve/locate color degrade problem, check TDS_ solution, It seems that any color problems came from DDIM params.
[x] controlnet reference mode
[x] controlnet multi module mode
[x] ddim inversion from Tune-A-Video
[x] support AnimateDiff v2
[x] support AnimateDiff MotionLoRA
[x] support FreeU
[x] keyframe controlnet apply
[x] controlnet inpainting mode
[x] support AnimateDiff v3 wo SparseCtrl
[ ] keyframe prompts apply

Experience

Multi Controlnet

inpainting + canny

inpainting + canny
tail + tail

MotionLoRA I2V results:

Zoom In / Zoom Out results from this old branch

Ablation experiment (controlnet/ipadapter)

results from this old branch

all / without denoise strength / without ipadapter / without controlnet(first frame)

Origin SD1.5 I2V attempt

Below is old results from this ols branch

First image from pikalabs, second was generated from sd1.5

First used IPAdapter+init-image-denoise, second used only IPAdapter

Training

23.8.22: Drop local training scripts, using authors repo to do training experiences(I2V). First, make image injection refer IP-Adapter. Already test in AI_power.