questions about model training

[forwiat]

hello, yuyq96, Thank you so much for the great work you've shared. I learned that D-TDNNSS mini-batch setting 128 from D-TDNN paper. But this model is too large to train on single gpu. Could you tell me how you train it? Using nn.Parallel or DDP? Looking forward to you reply

[yuyq96]

Actually, it should fit in a GPU with 12GB RAM when memory_efficient set to True, which is the default. If not, you can try:

• Using nn.DataParallel to train the model on two cards, and the mini-batch size on each card is 64, which should also be large enough for the batch normalization.
• Slightly decreasing the mini-batch size, e.g. to 100, then see if it fits in your card and how much RAM it consumes.

[forwiat]

Ok, I will try it. Thanks a lot

[shgidi]

@yuyq96 Hi, do you have a training script for this model?

[yuyq96]

@shgidi The original training script for D-TDNN or CAM are not open source due to the lack of company procedure. However, now you can use 3D-Speaker to train D-TDNN, CAM and CAM++. This project is lead by my former colleagues at Alibaba DAMO, and it is similar to the original training script.