torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 394.00 MiB

[shifu-learner]

Hello, I am trying to finetune GPT-j-6b. I followed the instructions provided in the documentation. But, I get this error.

I tried by changing batch size =1, gradient_accumulation_steps=4.

Any idea how can i solve this.

[mallorbc]

Need more information. How are you running it? What is your hardware?

[shifu-learner]

Sure I am trying to execute in a TESLA VM with Nvidia driver. I have followed the documentation to finetune GPT_J_6b.

[shavingtonpitsos]

is sounds like your GPU is not large enough, how much GPU ram is available to you when you run your script? It sounds like you likely have too little GPU ram, probably less than 16 GB