SyncTalk
SyncTalk copied to clipboard
ZiqiaoPeng
Real-time inference
Is SyncTalk suitable for real-time inference? Are there any stats about latency anf performance? Are there any benchmarks or optimization tips for real-time use?
It definitely can. On my RTX 3090 I can have ~50 fps if everything is loaded to GPU. Here is my example of realtime chatting using SyncTalk:
https://github.com/ZiqiaoPeng/SyncTalk/assets/12045814/623e5309-7d71-4fab-ad41-2280566b422a
How do you send a request after training the model? Can you share the code? Is it in the form of a test in the documentation?
Lipsync is also very good:
https://github.com/ZiqiaoPeng/SyncTalk/assets/12045814/67bb8404-3f34-4363-b260-adcba5fefe3b
Video doesn’t work. Alsı could you please answer my question if possible
How do you send a request after training the model? Can you share the code? Is it in the form of a test in the documentation? @jinqiupeter
@jinqiupeter which audio encoder are you using - ave? hubert? deepspeech?
@jinqiupeter which audio encoder are you using - ave? hubert? deepspeech?
both ave and hubert, they have very similar metrics for my video
both ave and hubert, they have very similar metrics for my video
@jinqiupeter Is your demo video using the GUI code path from synctalk repo? I believe this is the code path supporting real time streaming of audio, but I wasn't able to get it to work with ave
What code path does real time asr with ave or hubert?
@jinqiupeter Hello. Glad to know you. I'm working on Real-time for SyncTalker. I see you have solved this problem. Can you give me a little information? Thank you for your kindness
Does anyone figure out how to run it as realtime ? would be of great help to know about it
