FastSpeech2 icon indicating copy to clipboard operation
FastSpeech2 copied to clipboard

M2voc training stuck?

Open CrazyCharles6 opened this issue 4 years ago • 2 comments

Why does the training of M2voc keep stuck and there is no information output?

CrazyCharles6 avatar Apr 12 '21 01:04 CrazyCharles6

You may need to check your system log.

ming024 avatar Apr 24 '21 01:04 ming024

print('total_step', total_step)
    for epoch in range(hp.epochs):
        print('epoch ', epoch)
        for i, batchs in enumerate(loader):
            print('batchs ', i)
            for j, data_of_batch in enumerate(batchs):
                start_time = time.perf_counter()
                print(f'batchs:{i}, samples:{j}')

The above code snippet is part of train.py, and print() is added for testing. The following is the log printed when nohup training:

total_step 27440000
epoch  0
Using cache found in /home/usrername/.cache/torch/hub/descriptinc_melgan-neurips_master

From the log, the training should be stuck in the position of the dataloader. Is this due to a network problem or a data problem?

CrazyCharles6 avatar May 13 '21 03:05 CrazyCharles6