CLI tests of the cpu training benchmark with pytorch use the gpu if its available

[aliabdelkader]

I ran tests cpu_training_pytorch_bert_sweep and cpu_training_pytorch_gpt_sweep on a machine with cpu / gpu enabled. Those tests utilize gpu for training if it is available.

steps to reproduce:

• install optimum-benchmark along with pytorch-gpu version
• run: pytest -k "cpu and training and pytorch"
• monitor gpu utilization during the test ( using nividia-smi)

Expected:

• no increase in gpu utilization or memory consumption

Obtained:

• test process reserves memory on gpu
• gpu utilization or memory consumption increases during the test

[aliabdelkader]

Hi @IlyasMoutawwakil

I did some debugging into this issue. I think the bug is that those tests do not set benchmark.training.training_arguments.use_cpu to true. so, the Trainer class moves everything to the gpu for training.

[IlyasMoutawwakil]

thanks for spotting this ! it used to be automated but got removed during the last refactoring, my bad 😅