train models with gpu memory error

[Logan-SUN]

General Question

I try to train a s2t model by paddlespeech on our owns machine with 7 gpus, but always report error about having no aviliable GPU memory, I have set batch_size ==8 and wav-id is sorted by name, our dataset contains 300+ million audios totally. Can anyone help me solve this gpu memory issue please? THX

[Zth9730]

What is the approximate duration of each audio in your dataset? If the input audios are very long, more GPU memory is required.

[Logan-SUN]

I have checked that the maximum duration of our audio is less then 10 seconds, thanks by the way. The error I have captured as followed:

[Zth9730]

Are you using the same model structure and configuration as our repository provides? Whether to add or delete modules or change model size?

[Logan-SUN]

I used the same model structure and configuration as your pretrained model named asr1_conformer_wenetspeech_ckpt_0.1.1.model which is published on websit.

[zh794390558]

Your GPU mem is not enough, please reduce batchsize to fit you GPU device.