Transformers-Tutorials icon indicating copy to clipboard operation
Transformers-Tutorials copied to clipboard
verified publisher icon NielsRogge

Donut training slowness

Open khadkechetan opened this issue 2 years ago • 3 comments

Hi, I am currently working on fine-tuning the Donut model on custom dataset. While I train on only 2500 samples, with 30 epoch it takes around 20 hrs on 24GB GPU.

How can we speed up this process?

@NielsRogge

khadkechetan avatar Jun 13 '23 05:06 khadkechetan

Hi, Any response would be highly appreciated.

khadkechetan avatar Aug 25 '23 08:08 khadkechetan

Which batch size are you using? Can you verify the GPU is used (by typing nvidia-smi in the terminal)?

I'd recommend this guide: https://huggingface.co/docs/transformers/perf_train_gpu_one

NielsRogge avatar Aug 25 '23 08:08 NielsRogge

Hi, Any response would be highly appreciated.

I'm guessing you were training with a RTX 4090? Did you manage to get your training times down?

poetaster avatar Oct 28 '23 10:10 poetaster