Can you share the machine configuration for running this project?

[pr-agati]

I have run this project with 32 GB GPU Configuration machine, so it is taking 7 min to produce output. I am looking forward to reduce this time. How much GPU will reduce the time from 7 min to 2 min?

[byixi2023]

i also feel the speed is too slow. a video of 13s length need 520s to produce! my hardware is 4090 24G

[KeystoneScience]

@byixi2023 I am having trouble getting it running on my 4090, by any chance, could you share the installation steps you took?

[BryanHarrisScripts]

Setting Up Rerender_A_Video on Windows: A Comprehensive Guide

---

from pytorch_lightning.utilities.distributed import rank_zero_only

C:\Users\bryan\Rerender_A_Video\deps\ControlNet\ldm\models\diffusion\ddpm.py

---

https://github.com/williamyang1991/Rerender_A_Video

1. Pre-requisites:

Ensure you have a Windows machine with an NVIDIA GPU that supports CUDA.

2. Initial Setup:

• Python:

  • Download and install Python 3.11.5 (64-bit version).
  • Ensure the Python and Pip paths are correctly set in your environment variables.
  • Confirm your Python version and architecture with:

    python --version
    python -c "import platform; print(platform.architecture()[0])"
    

• CUDA:

  • Download and install CUDA 12.2. This is necessary for GPU support with PyTorch.

• Git:

  • Download and install Git for Windows.

• Visual Studio:

  • Download and install Visual Studio 2022. Ensure you include the "Desktop development with C++" workload during installation to get the necessary build tools and Windows SDK.

• GitHub SSH Setup:

  • Set up an SSH key and link it to your GitHub account using this guide.

3. Cloning the Repository:

Navigate to the directory where you want to clone the repository and run:

git clone [email protected]:williamyang1991/Rerender_A_Video.git --recursive
cd Rerender_A_Video

4. Setting Up the Python Environment:

• Install the necessary Python libraries listed in the requirements.txt file:

pip install -r requirements.txt

5. Compiling Dependencies:

Run the installation script to build and compile necessary components:

python install.py

This step will download the required models into the ./models directory.

6. Downloading Additional Models:

If the project requires additional .safetensors models, download them as per the project's instructions and place them in the models directory.

7. Configuring the Model:

Update the JSON config file (e.g., real2sculpture.json) to point to the correct model checkpoint in the models directory.

8. Running the Rerender Script:

Before you proceed, ensure that you have the right dependencies. There are known compatibility issues:

Note: Numba requires NumPy version 1.24 or less. If you encounter an error related to this, resolve it by downgrading NumPy:

pip install numpy==1.24

Once all dependencies are set:

1. Navigate to the project's directory.
2. Execute the following command:

python rerender.py --cfg config/real2sculpture.json

This will start the rerendering process using the configuration provided in config/real2sculpture.json.

9. Troubleshooting:

• Python Architecture Issues:

  • Ensure you're using a 64-bit version of Python, as 32-bit versions can cause compatibility issues.

• Rust and Tokenizers Library Issue:

  • If encountering an error related to Rust and mismatched architectures, set up the correct Rust target with rustup:

    rustup target add x86_64-pc-windows-msvc
    

• Visual Studio Build Tools:

  • If you encounter errors related to missing link.exe, ensure your PATH environment variable includes the path to the Visual Studio build tools.

• CUDA-Related Issues:

  • Make sure your NVIDIA drivers are up-to-date.
  • Ensure the PyTorch version is compatible with your installed CUDA version.

• PIP and Python Environment:

  • If pip is not recognized, you might have to reinstall it or adjust your PATH.
  • Use ensurepip if pip isn't installed:

    python -m ensurepip --default-pip
    

Summary:

1. Initial Setup: Cloning the repository and getting the basic tools ready.
2. Python Environment: Ensured the right Python version (64-bit) and architecture.
3. Python Libraries: Installed required libraries using pip.
4. Rust & Tokenizers Issue: Addressed architecture mismatch.
5. Visual Studio Build Tools: Addressed missing link.exe and ensured the necessary tools were in the PATH.
6. Compiling Dependencies: Ran the provided installation script.
7. PyTorch & CUDA: Addressed CUDA availability in PyTorch.
8. Downloading Additional Models: Downloaded and placed additional model checkpoints in the appropriate folder.
9. Configuring & Running the Model: Updated the JSON config and ran the rerender script.
10. Troubleshooting:
    • Addressed Python architecture issues.
    • Fixed Rust and tokenizers library issue.
    • Ensured the correct PATH for Visual Studio Build Tools.
    • Handled pip-related issues.

Using Additional Models with Rerender_A_Video:

1. Download Additional Models:

• Acquire the required model checkpoints. For our journey, the model realisticVisionV51_v20Novae.safetensors was essential.
• You can typically find these models on the project's GitHub page or associated links. Sometimes, they're hosted on third-party websites like CivitAI.

2. Place Models in the Correct Directory:

• Once downloaded, move the model checkpoints to the ./models/ directory of the Rerender_A_Video project.

3. Configure the Model for Use:

• Navigate to the configuration files (typically JSON files) in the project.
• Update the JSON config to point to the new model. For instance:

  {
    "sd_model": "models/realisticVisionV51_v20Novae.safetensors",
    ...
  }
  

• Adjust any other relevant parameters in the config as needed.

4. Run the Rerender Script with the New Model:

• Once everything is set up, navigate to the Rerender_A_Video project directory in your terminal.
• Use the rerender.py script with the updated config to run

the model: bash python rerender.py --cfg config/your_updated_config.json

5. Review the Output:

• After the script completes, review the output video or images.
• Adjust parameters in the config and re-run if necessary to get the desired results.

Remember, using deep learning models can be an iterative process. Sometimes, you may need to adjust parameters, prompts, or even use different models to achieve the desired output.

Notes & Troubleshooting:

Essential Software Components:

• Python: Use Python 3.11.5 (64-bit version). Ensure the Python and Pip paths are correctly set in your environment variables.
• CUDA: Download and install CUDA 12.2 for GPU support with PyTorch.
• Git: Download and install Git for Windows.
• Visual Studio: Download and install Visual Studio 2022. Ensure you include the "Desktop development with C++" workload during installation to get the necessary build tools and Windows SDK.

SSH Setup for GitHub:

• Check for existing SSH keys:

  dir %USERPROFILE%\.ssh\
  

  If files named id_rsa.pub or id_ed25519.pub are present, you already have an SSH key.
• If you don't see the SSH key files, generate an SSH key and link it to your GitHub account.

Python Environment Setup:

• If pip isn't recognized, you might need to reinstall or adjust your PATH. Alternatively, use ensurepip:

  python -m ensurepip --default-pip
  

PyTorch & CUDA Compatibility:

• PyTorch and CUDA must have compatible versions.
• Install a PyTorch version without specifying the CUDA version:

  pip install torch torchvision torchaudio
  

• Check PyTorch's CUDA version:

  python -c "import torch; print(torch.version.cuda)"
  

Visual Studio Build Tools:

• Install Visual Studio Build Tools. During installation, select the "Desktop development with C++" workload and the "Windows 10/11 SDK" component.
• Ensure the Visual Studio Build Tools' path is included in your system's PATH.

Rust Setup:

• Some Python packages have components written in Rust. If there are issues with Rust, ensure you're using the stable Rust toolchain:

  rustup default stable
  

Compiling Issues & Solutions:

• If facing issues related to link.exe, ensure your PATH environment variable includes the path to the Visual Studio build tools.
• In case of issues with Rust and mismatched architectures, set up the correct Rust target:

  rustup target add x86_64-pc-windows-msvc
  

Checking GPU Status:

• Use NVIDIA's System Management Interface (SMI) to check the status of your GPU:

  nvidia-smi
  

General Tips:

• Always ensure you're using the appropriate software architecture (64-bit is recommended for this setup).
• Backup essential data before starting the setup, especially when making significant changes.
• Check for library updates regularly and ensure all dependencies are up-to-date.
• If encountering issues, consult the project's official documentation and community forums for assistance.

System Configuration and Versions

1. Python Environment:

   • Python Version: Python 3.11.5

     Note: Ensure you're using a 64-bit version of Python. You can confirm this with:

     python --version
     python -c "import platform; print(platform.architecture()[0])"
     

2. CUDA & GPU Configuration:

   • CUDA Version: 11.8 (As detected by nvcc)

   • PyTorch's CUDA Version: 2.0.1+cu118

   • NVIDIA Driver Version: 537.34

   • GPU Model: NVIDIA GeForce GTX 1080

     Note: It's crucial that the CUDA version in PyTorch (torch.version.cuda) matches the system's CUDA version. To check GPU status, use nvidia-smi.

3. Git Configuration:

   • Git Version: git version 2.42.0.windows.2

     Note: Ensure you've set up an SSH key and linked it to your GitHub account for secure and password-less interactions with GitHub repositories.

4. Rust Configuration:

   • Rust Version: rustc 1.72.0

     Note: Some Python packages might be written in Rust, requiring Rust tools for compilation. Ensure you're using the stable Rust toolchain (rustup default stable).

5. Python Libraries:

   • PyTorch Version: 2.0.1+cu118

   • Tokenizers Version: 0.12.1

   • Transformers Version: 4.19.2

   • (And many others as listed in versions.txt)

     Note: Ensure that library versions are compatible with your project. PyTorch's CUDA version should match the system's CUDA version.

6. Additional Notes & Troubleshooting:

   • Visual Studio & Build Tools: Ensure you've installed Visual Studio with the "Desktop development with C++" workload. The absence or misconfiguration of this toolchain can hinder the compilation of various projects.
   • PATH Considerations: Some errors might arise if certain executables (like link.exe from MSVC) aren't in the system's PATH. Ensure directories for tools are added to your system's PATH variable.
   • Python Package Managers: Always ensure pip is up to date. If a particular Python package isn't installing correctly, it's helpful to check for updates or try installing it separately.
   • SSH Key Authentication: If cloning repositories using SSH URLs, it's essential to have the correct SSH key added to your GitHub account.

---

C:\Users\bryan\Rerender_A_Video>python -m xformers.info output

• A matching Triton is not available, some optimizations will not be enabled.
• Error caught was: No module named 'triton'
• xFormers 0.0.16rc425
• memory_efficient_attention.cutlassF: available
• memory_efficient_attention.cutlassB: available
• memory_efficient_attention.flshattF: available
• memory_efficient_attention.flshattB: available
• memory_efficient_attention.smallkF: available
• memory_efficient_attention.smallkB: available
• memory_efficient_attention.tritonflashattF: unavailable
• memory_efficient_attention.tritonflashattB: unavailable
• swiglu.fused.p.cpp: available
• is_triton_available: False
• is_functorch_available: False
• pytorch.version: 2.0.1+cu118
• pytorch.cuda: available
• gpu.compute_capability: 6.1
• gpu.name: NVIDIA GeForce GTX 1080
• build.info: available
• build.cuda_version: 1108
• build.python_version: 3.11.5
• build.torch_version: 2.0.1+cu118
• build.env.TORCH_CUDA_ARCH_LIST: None
• build.env.XFORMERS_BUILD_TYPE: None
• build.env.XFORMERS_ENABLE_DEBUG_ASSERTIONS: None
• build.env.NVCC_FLAGS: None
• build.env.XFORMERS_PACKAGE_FROM: None
• source.privacy: open source

[AIhasArrived]

Hello @BryanHarrisScripts I dont understand your comment, how is this related to video retalking whereas its installed withing a SEPARATE conda environement no??? What did you mean by all these explanations?

[AIhasArrived]

i also feel the speed is too slow. a video of 13s length need 520s to produce! my hardware is 4090 24G

@KeystoneScience Hey keystone, please please take the time to answer me, did you find how to make it work faster on a 4090? Please tell me.

[AIhasArrived]

**byixi2023 ** c

Hello @pr-agati please please telle me, did you find how to make it work faster on your card?? How to make it use GPU ? or @byixi2023 ??

[KeystoneScience]

@AIhasArrived I was able to get it a bit faster by trimming out some things I thought weren't needed. though it is still pretty slow, requiring at least 5x the duration of the video. You can make it use the GPU by installing the correct version of PyTorch, I recommend using a conda environment and gettin the specifics for your system from PyTorch's website, you can click through it and it will tell you the command to run to install everything... I may have installed condatoolkit in addition though.

[AIhasArrived]

Glad to see you, I was just on your profile then youtube channel earlier and was like: he did not make a video since a 3 months (or weeks dont remmeber) and he should not stop haha. Ok my question is how much faster? I just did a 10 minutes video in 21 minutes (step6) + the others smaller steps. Although I am using a powerful card. But I think it's related to RAM and not to GPU look: (I have another question under the image please look at it)

Question 2: If you installa new version of cuda and torch etc inside your conda env, does that require to RE RUN install requiremen line? Or is it just : update pytorch -> you can immediately start using retalker again? Thanks I am really frustrated with this. Also, did you use others tools?

[KeystoneScience]

@AIhasArrived I totally feel you, this stuff can be pretty frustrating. If I remember right, I commented out any lines in the requirements.txt relating to torch so that I could just install it with conda commands. It was certainly a ton of try something, make a new environment, try again, ect. Honestly, a 10 minute video in 21 minutes may be nearly as good as it can get without intensive improvements. I am looking to speed it up myself, but got sidetracked on other ML projects I am working on, so I will take a deep dive into it tonight and see what I can find! I do have some other tools I am using to get this output better... I was more concerned with the results of it than the time it takes to generate. Also, do you have a breakdown of your GPU usage from computational vs graphical, as well as the GPU memory being used? I know on linux you can use nvtop to view it, but I am not sure on windows... that may help to see what areas we can boost up on your system to get faster inferences.

[AIhasArrived]

Not sure how to see that But I am sure there is on windows a tool that would do that, but how could it help me? I mean the GPU is like 0 usage .. also.. I noticed that amd cards were very slow for some AI types, maybe linux is also slowing you down! Yeah please report any update you find.

Here is the requirement file:

addict
future
librosa==0.7.0
lmdb
numba==0.48
numpy
opencv-contrib-python>=4.2.0.34
opencv-python
Pillow
pyyaml
requests
scikit-image
scipy
tb-nightly
torch>=1.3
torchvision
tqdm
yapf

What are all the lines you would recommand commenting and instead installing myself? Yeah it's so tiring the install try delere retry.. I have 0 energy for that. lastly, did you use other tools? I am trying lip2sync HD but Got stuck with installation (some problem about (ERROR: Failed building wheel for llvmlite)), wonder if you tried other stuff?

[AIhasArrived]

Ok @KeystoneScience I might had that wrong, I don't think it was 10 minutes video in 21 minutes, I might had read something wrong, I think it was a ONE minute video. In any case, today it was : 1 MINUTE video for 1H11 I believe Do you believe that? I might have a big problem somewhere, Because now I am geting out of memory for another repo but I hadnt have it yesterday, any advice? I am so lost, this is so maddening lol