Is it normal to wait a long time between each epoch？

[HuaHuaOfficial]

At the interval, the GPU does not seem to be working.

[Movelocity]

About 5 min/epoch.

dataset size: 10k batchsize: 20 gpu: A4000 model is the same as that in vctk_basse.json.

Here is the log: 2023-01-25 19:52:04,390 paimon INFO ====> Epoch: 560 2023-01-25 19:57:01,392 paimon INFO ====> Epoch: 561 2023-01-25 20:01:54,745 paimon INFO ====> Epoch: 562 2023-01-25 20:05:17,465 paimon INFO [220000, 1.7485471954399575e-05] 2023-01-25 20:05:22,223 paimon INFO Saving model and optimizer state at iteration 563 to ./logs/paimon/G_220000.pth 2023-01-25 20:05:23,070 paimon INFO Saving model and optimizer state at iteration 563 to ./logs/paimon/D_220000.pth 2023-01-25 20:07:00,388 paimon INFO ====> Epoch: 563

[NikitaKononov]

About 5 min/epoch.

dataset size: 10k batchsize: 20 gpu: A4000 model is the same as that in vctk_basse.json.

Here is the log: 2023-01-25 19:52:04,390 paimon INFO ====> Epoch: 560 2023-01-25 19:57:01,392 paimon INFO ====> Epoch: 561 2023-01-25 20:01:54,745 paimon INFO ====> Epoch: 562 2023-01-25 20:05:17,465 paimon INFO [220000, 1.7485471954399575e-05] 2023-01-25 20:05:22,223 paimon INFO Saving model and optimizer state at iteration 563 to ./logs/paimon/G_220000.pth 2023-01-25 20:05:23,070 paimon INFO Saving model and optimizer state at iteration 563 to ./logs/paimon/D_220000.pth 2023-01-25 20:07:00,388 paimon INFO ====> Epoch: 563

Batch sizes, that are not a degree of 2 (e.g. 8, 16, 32, 64, 128... are ok), are not recommended to use

[NikitaKononov]

At the interval, the GPU does not seem to be working.

Maybe you have bottleneck in CPU / num_workers / disk speed / RAM speed

[yijingshihenxiule]

Hello， @HuaHuaOfficial . I encountered the same question. Have you solved it?

At the interval, the GPU does not seem to be working.