Running evaluation on Gemma-2 27B model

[zeynepgulhanuslu]

Hi,

I'm trying to run evaluation code with the Gemma-2 27B model. I have 2 GPUs, each with 80 GB of memory. However, I encountered an out-of-memory error when using the parallel command. Has anyone successfully used multiple GPUs for evaluation, and if so, how? Can you help me, please?

I have tried running the following command:

lm_eval --model hf --model_args pretrained=google/gemma-2-27b,parallelize=True --tasks mmlu,arc,gsm8k,hellaswag,truthfulqa,winogrande --output google-gemma-2-27b --batch_size 8

I also tried using the accelerate command and as Python code:

accelerate launch -m lm_eval

but that didn't work either.

Note: I have tried exporting CUDA_VISIBLE_DEVICES=0,1.

Thank you