[Request] Deforum plug-in for Auto

[ProducerMatt]

Deforum's scripts provide video-generation functionality with camera movements. They can be installed as a plug-in with this repo: https://github.com/deforum-art/deforum-for-automatic1111-webui

I'm trying to get it working in your repo. With these changes I can get the Deforum plug-in loaded in the UI, but running it produces the Python error "No module named 'py3d_tools'". I get conflicting info on whether the package for py3d_tools is open3d-python or pytorch3d, though I tried both.

[AbdBarho]

@ProducerMatt I am not sure how I can help you, I haven't used the script myself so it is hard to tell.

That being said, your changes seem correct, the problem is something else.

[StableInquest]

@ProducerMatt Did you solve this. I'm having the same issue. Python error "No module named 'py3d_tools'"

[ProducerMatt]

@StableInquest Nope. I tried with the new version and I still get the same problem. https://github.com/AbdBarho/stable-diffusion-webui-docker/compare/master...ProducerMatt:stable-diffusion-webui-docker:auto-ext

The only bread crumb I've found, someone had this problem because their deforum folder was not in (your auto repo)/extensions/deforum. Apparently the library locations are hard coded.

[AbdBarho]

here is how you setup Deforum:

1. checkout the branch #176 auto-extensions
2. mkdir -p data/config/auto/extensions
3. git clone https://github.com/deforum-art/deforum-for-automatic1111-webui/ data/config/auto/extensions/deforum
4. create a file data/config/auto/startup.sh and put the following in it:

set -e

pip install -q -r extensions/deforum/requirements.txt

PY3D_PATH="${PWD}/extensions/deforum/py3d"
ADA_PATH="${PWD}/extensions/deforum/AdaBins"
MIDAS_PATH="${PWD}/extensions/deforum/midas"

if [ ! -d $PY3D_PATH ]; then
  git clone https://github.com/MSFTserver/pytorch3d-lite.git $PY3D_PATH
fi
if [ ! -d $ADA_PATH ]; then
  git clone https://github.com/shariqfarooq123/AdaBins.git $ADA_PATH
fi
if [ ! -d $MIDAS_PATH ]; then
  git clone https://github.com/isl-org/MiDaS.git $MIDAS_PATH
fi

apt-get install wget -yq
export PYTHONPATH="$PY3D_PATH:$ADA_PATH:$MIDAS_PATH"

6. run docker compose --profile auto up --build
7. you should see a new tab called deform, click generate, it should work but would probably take hours

This is only 2D, I haven't tried 3D.

Example, default prompt, 384px 17steps 30 frames:

Getting this plugin to work was extremely unpleasant, the setup process is not well documented (or I have missed something)

this branch is still a work in progress, so the setup might change, however, it is obvious to me that maintaining the extensions would be a nightmare, therefore I would leave it up to the user.