Andrei Constantinescu

Can this replace syncnet in wav2lip and be used as the discriminator? would the core wav2lip architecture need to change?

Phonemizer as dependency

3

Is phonemicized dependency used? How is text turned into phonemes? Phonemizer has alot of phonemes that are grouped together and not standard but still IPA. for example the bpe_json file...

Training on own data

2

I'd like to train on my own languages. Using phonemizer as the phoneme set. Would I just add language ID embedding with no other changes? Is this setup for distributed...

Training speed

2

Hi how long did it take to train this model? I am currently training on my own implementation on a DGX 4 v100 cluster with deepspeed integrated. Gradient accumulation of...

Or Does the LM handle alignment during the self attention process? I read in the valle paper they use force alignment tools, but I dont see anything in the code.

Are the ar and nar models trained in parallel ( at the same time) or separately?

1

In addition, if I would like to train it on different languages (French) do I have to use another G2P tool?

I'm trying to train this on my custom dataset on faces. The core architecture of the model is roughly the same but I added a cross-attention block for conditioning on...

Higher Temporal Context window for Syncnet

1

I was wondering if there was a way to train Syncnet on a higher context window specifically 25 frames and 80 mel steps ( 80 corresponds to 1 second of...