code-docstring-corpus icon indicating copy to clipboard operation
code-docstring-corpus copied to clipboard

Published 20 hours ago • verified publisher icon EdinburghNLP

How much memory does this require while training?

Open lukesalamone opened this issue 4 years ago • 3 comments

We are running this experiment on Google Colab Pro (16 GB machines) and are receiving out of memory errors using the same vocabulary size (89500). How much memory would you recommend for us to have to run this experiment?

lukesalamone avatar Apr 08 '21 21:04 lukesalamone

Strange, what framerwork are you using?

Avmb avatar Apr 08 '21 21:04 Avmb

We are training with Nematus. When we try to run with the defaults we get

Resource exhausted: OOM when allocating tensor with shape[5980,89500]

When we check our machine specs we get

Gen RAM Free: 26.2 GB  | Proc size: 168.6 MB
GPU RAM Free: 16280MB | Used: 0MB | Util   0% | Total 16280MB

lukesalamone avatar Apr 09 '21 18:04 lukesalamone

Nematus has been reimplemented from scratch in Tensorflow since then and I haven't trained with this dataset with it.

You can try to reduce the vocabulary size e.g. to 32000 or 16000.

Avmb avatar Apr 09 '21 19:04 Avmb