Hardware requirements to compile this project (training, evaluating)

[hamzaelouiaazzani]

I tried to run the evaluation script, but a notification of fully consumed memory appeared and then the process crushed immediately. Could you guys please tell me about the required hardware resources (RAM, CPU, GPU) to compile this project?

I run the project on a computer of 10 cores and with 16 GB RAM .

[boberle]

Hello ! It really depends on your input. A few GB will be enough to train a model with a few texts, like a demo. But if you have dozens of long texts, you will need far more than 16 GB of memory.

The requirements are really a factor of (1) the number of texts, (2) the length of the texts and (3) the length of the sentences in the texts (look especially for very long sentence). If you have memory issues, try to limit the length of the sentences (try to split the longest sentences) and the number of texts. You can make some trials by starting with a few texts and then adding more and more texts (and same for the sentence length).

Of course, this apply to the GPU memory has well if you use a GPU.

The speed of the CPU is not really important even if you don't use a GPU, provided you have enough time (of course, a few minutes of GPU will translate into hours or days of CPU, but this will not make the system crash, it will just take a very long time).

For the Democrat project, for example, I think we usually used at least 32 GB of memory (at least for the tests) but we had to upgrade to more memory to train the final model.

[hamzaelouiaazzani]

Hello, You explained everything Mr. Bruno thank you so much Best regards,

Le ven. 22 avr. 2022 à 6:42 a.m., Bruno Oberle @.***> a écrit :

Hello ! It really depends on your input. A few GB will be enough to train a model with a few texts, like a demo. But if you have dozens of long texts, you will need far more than 16 GB of memory.

The requirements are really a factor of (1) the number of texts, (2) the length of the texts and (3) the length of the sentences in the texts (look especially for very long sentence). If you have memory issues, try to limit the length of the sentences (try to split the longest sentences) and the number of texts. You can make some trials by starting with a few texts and then adding more and more texts (and same for the sentence length).

Of course, this apply to the GPU memory has well if you use a GPU.

The speed of the CPU is not really important even if you don't use a GPU, provided you have enough time (of course, a few minutes of GPU will translate into hours or days of CPU, but this will not make the system crash, it will just take a very long time).

For the Democrat project, for example, I think we usually used at least 32 GB of memory (at least for the tests) but we had to upgrade to more memory to train the final model.

— Reply to this email directly, view it on GitHub https://github.com/boberle/cofr/issues/3#issuecomment-1106065057, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASL5JZ6GD7QL72RXZZXLSILVGJC4NANCNFSM5T674GCQ . You are receiving this because you authored the thread.Message ID: @.***>