How do i speed up ?

[emrytp]

python main.py data/May --workspace model/trial_may -O --test --test_train --asr_model ave --portrait --aud ./demo/test.wav

A 20-second video takes 1 minute 10 seconds, how can I speed it up ? Should I use another command to speed up ?

[jinqiupeter]

try to append --preload 1 to load data to main memory or --preload 2 to load data to GPU memory

[oguzhandoganoglu]

I tried --preload 2 First it loads quickly, then it slows down and I lose connection to the remote computer. When I connect again, it is Killed.

[jinqiupeter]

That's because you don't have enough memory. You can try to increase swap memory and use --preload 1, or increase swap memory and GPU memory and use --preload 2

[oguzhandoganoglu]

We have 20GB RTX4090. The data is 3GB. Why do you think this could happen?

[jinqiupeter]

20GB of memory is not enough. You can monitor your memory usage while loading the data, and notice when the killed error happens

[oguzhandoganoglu]

How many GB of memory do I need?

[oguzhandoganoglu]

You can see what happend https://github.com/ZiqiaoPeng/SyncTalk/assets/78206007/22dc9a2e-2844-426a-a73b-c63c6581b31f

[jinqiupeter]

FRY, I have 64GB memory and it still gets killed if the training video is longer than 4 mins. You cannot easily increase your memory on a remote server, instead, you can increase swap memory to whatever you want

[jinqiupeter]

You can see what happend https://github.com/ZiqiaoPeng/SyncTalk/assets/78206007/22dc9a2e-2844-426a-a73b-c63c6581b31f

Dont monitor GPU memory, monitor CPU memory (main memory) with top/htop

[oguzhandoganoglu]

However, we use --preload 2 argument. This argument load to GPU memory ?

[jinqiupeter]

The data first loads to memory, then processed, and then loads to GPU, so you still need to have enough CPU memory

[oguzhandoganoglu]

It's done. Got it, so I'll do my memory checks, and if we have enough memory, how long will it take me to create a 10-second video?

[jinqiupeter]

For RTX 4090 I'd guess around 6 seconds

[oguzhandoganoglu]

Wit this command python main.py data/May --workspace model/trial_may -O --test --test_train --asr_model ave --portrait --aud ./demo/test.wav ? True? It seems very good.

[jinqiupeter]

Just test it. then try --preload 1, then --preload 2 and compare

[kamimachi]

Wit this command python main.py data/May --workspace model/trial_may -O --test --test_train --asr_model ave --portrait --aud ./demo/test.wav ? True? It seems very good.

hi! Could you tell me how long will it take to create a 10-second video now.

[HardikJain02]

I tried --preload 2 First it loads quickly, then it slows down and I lose connection to the remote computer. When I connect again, it is Killed.

hey, were you able to generate your model upon training? After loading train data, it says Killed. Any workaround? @jinqiupeter @oguzhandoganoglu

[varunjain99]

@jinqiupeter how much swap memory and gpu memory is needed to successfully do preload 2?

[HardikJain02]

Successfully trained 7000 frames using A100SXM 125 GB RAM & 80GB VRAM.

[varunjain99]

@HardikJain02 what is your inference speed?

I've run

python main.py data/May --workspace model/trial_may -O --test --asr_model ave --aud ./demo/test.wav

with various preload options (0, 1, 2) and it doesn't get above ~16 it/sec

[xiao-keeplearning]

@varunjain99 same question, Have you solved?