PyTorch_Speaker_Verification icon indicating copy to clipboard operation
PyTorch_Speaker_Verification copied to clipboard

Published 20 hours ago • verified publisher icon HarryVolek

Change the code to use multi-gpu, but can not speed up the training.

Open Naminwang opened this issue 5 years ago • 2 comments

Use DataParallel model to start a multi-gpu training, change the config.yaml batch size, can not speed up the training.

Naminwang avatar Dec 04 '19 08:12 Naminwang

I've also tried multi-gpus and had same issue with you. Then I found the biggest overhead is in GE2ELoss part. Especially get cosine similarity matrix and calculate loss part.

https://github.com/HarryVolek/PyTorch_Speaker_Verification/blob/11b1d1932b0a226de9cabd8652c0c2ea1446611f/utils.py

Just copy and paste this codes to your utils.py. I don't know why author didn't have merged this codes yet, but it is much faster than original codes.

mindmapper15 avatar Dec 19 '19 02:12 mindmapper15

I've also tried multi-gpus and had same issue with you. Then I found the biggest overhead is in GE2ELoss part. Especially get cosine similarity matrix and calculate loss part.

https://github.com/HarryVolek/PyTorch_Speaker_Verification/blob/11b1d1932b0a226de9cabd8652c0c2ea1446611f/utils.py

Just copy and paste this codes to your utils.py. I don't know why author didn't have merged this codes yet, but it is much faster than original codes.

Thank you very much, I will try to use the code in the link.

Naminwang avatar Dec 19 '19 02:12 Naminwang