Getting NaN in Mel Loss during the first few epochs for first training

[iAdityaVishnu]

I am noticing NAN in mel losses. The nan is coming in the first few epochs itself. Does anyone know how can this be solved?

This on the master brand code that I am training using two H100.

@yl4579

[iAdityaVishnu]

[kadirnar]

You should change the loss values in the config file. https://github.com/Respaired/Tsukasa-Speech/issues/6#issuecomment-2758477322

[iAdityaVishnu]

If I decrease the learning rate than it would lead to underfitted model

[iAdityaVishnu]

Here is tensorboard link

http://151.115.73.7/

[iAdityaVishnu]

@yl4579

[martinambrus]

A bit late to the discussion here but StyleTTS2 is currently incompatible with H100 hardware. Also, there are a few issues here mentioning NaN during training. Have a look around.