VQMIVC icon indicating copy to clipboard operation
VQMIVC copied to clipboard

Published 20 hours ago • verified publisher icon Wendison

Training Loss Abnormal

Open Haoyanlong opened this issue 2 years ago • 3 comments

@andreasjansson @Wendison Hello, sorry to interrupt you! I'm a rookie of voice model. I have trained the model in VCTK-Corpus-0.92.zip dataset by "python3 train.py use_CSMI=True use_CPMI=True use_PSMI=True" in NVIDIA V100S. But after 65 epochs, the train loss are as follows: image Could you give me some advice? Thank you very much!

Haoyanlong avatar Jul 13 '22 03:07 Haoyanlong

Hi, I think the lld losses are normal, you could train for more epoches and listen to converted samples to verify whether your training is successful.

Wendison avatar Jul 13 '22 05:07 Wendison

@Wendison Hello! I have two questions:

  1. The default epoch of training is 500. I have achieved the process, training log as folllows(500 epochs). Whether each loss value meets expectations? image

  2. I want to see the training effect quickly.How many epochs do you suggest to train? Thank you!

Haoyanlong avatar Jul 13 '22 08:07 Haoyanlong

  1. I don't remember the exact value of each loss, but I think your losses should be normal according to the losses shown in https://github.com/Wendison/VQMIVC/issues/15#issue-1025051309
  2. Based on my experience, 500 epoches can obtain stable conversion results. After 500 epoches, I didn't see any noticable performance improvements. One direct way to select the suitable epoch is to listen to intermediate converted samples during training.

Wendison avatar Jul 13 '22 09:07 Wendison